He means back when people used their fucking brains.
That general behavior extends beyond race. Any group or individual that is "elite" tries to suppress people below them to preserve their position. Anyone that could threaten that status is suppressed. It's dirty human nature.