Everyone with a lick of sense does what you do. And if you admit it, it's called prejudice. If a police officer does, it's called profiling and he's told he's not allowed to do it.
But if we get so brainwashed we stop ourselves from dealing with reality in the privacy of our own thoughts, then we're lost.