Practically anyone can allege you are 'unfit' to own weapons.
Let's say your kid (Tommy) tells his friend (Joey) at school 'my dad has a lot of guns'. Next weekend Dad boo's kids taking a knee at a basketball game. Joey's mom is a BLM fanatic and is offended. She calls cops and says Tommy's Dad is a right winger anti Black racist and she know he has a lot of guns.
Your ex wife tells cops you threatened her (you didn't).