It's only happened to me a few times. I just remind them that they are wearing a mask and are thus protected from catching COVID, so please don't concern yourself with me. Just like vaccines, if you get a flu shot, then why are you concerned about me if I don't? You are protected.