If you read this letter attached, you'll see that the concern wasn't then, nor is it NOW the Freemasons as a whole. It was - as your picture indicates - about the infiltration/corruption of a "rogue" group of Freemasons.
https://founders.archives.gov/documents/Washington/06-02-02-0435