You’d be surprised how many women are scared to confront these men but not for the reason you think. They don’t want to offend them.
I kicked a dude out of the changing room/bathroom at a waterslide park and the women looked at me like I was the crazy one. No thank you, no, wow, wtf. There were little girls getting in and out of bathing suits and the guy was just standing there staring at them when I walked in. Nobody was saying anything to him or engaging with him. The whole episode was bizarre, and this was in 2014.