The problem with the estrogens in plants is not that they are exactly the same as human estrogen.
The plant estrogens are weaker overall than the estrogen made by the human body. They use the same receptors though. So while the human male body may not produce enough estrogen to max out your receptors, eating lots of plant estrogens will cause them to stick to the receptors and net you more estrogen than typical.
Eating one serving of soybeans is not going to cause you to grow tits and turn gay. Its when you ingest it for long periods of time that it has bad consequences.