not always.
sometimes if it's a group of friends and they are all good with whatever they all 'are', then the term 'the Jews' is used.
for example.
"on the friday we all went to the cafe after work except for the the Jews, because this was a holy day for them and they weren't with us."
just like you'd say "on that friday we went to the steak house and had steak and salad except for the Catholics, because it was lent they all ate fish"
so you're wrong. It's more often NOT a pejorative.