A lot of times when I'm in a party, I accidentally click on the wrong thing and it gives me the "Do you want to invite your party" prompt, but there's no way to back out of it besides chasing someone around with your finger, which could be problematic if you can't reach them (like falling out of the map). A "cancel" option already exists if some party members can't go into the room, it should be available for all party invites.