Sad but true. ANTIFA have the media and activist judicials behind them who would love to document retaliation from right-leaning counter-protests as we've seen time and again. Not to mention the booby trap of organized events being honeypots started up by cointelpro types such as Richard Spencer/Charlottesville.
And as you said, unlike ANTIFA, who are societal failures with nothing to lose, most conservatives and middlegrounders actually HAVE families/careers/businesses that could immediately be harassed or sunk through doxxing and unwanted media attention. So all they really can do is sit things out and let ANTIFA have their little pissing contests in Portland, Seattle, San Francisco, etc. Those cities are already lost to democrat degeneracy anyway.