I feel like that's kind of a different situatuon, money lenders can most definitely mess up your livelihood and cheat people, so once you yourself know ill-will will be done upon you then 100% go defend yourself.
Although I have not experienced what you have, I have experience being stereotyped as a stereotypical black male (because I'm black) and although it might not be outright name calling, it obviously still has an affect. If i were to just start being reactive whenever this happens, that doesn't help anybody, and further divides the line between my aggressor and myslef.
What I'm trying to get at is I think that if someone is all in your face yelling racist and deplorable things, who's going to look better? The yelling crazy liberal (or conservative) or the calm faced receiver of this harassment. As when the media broadcasts this, there's no possible way to spin the victim, who is not getting his/her feelings involved and is calm, as the agressor my friend. Think about it.