Technically, a pedophile is someone who is attracted to a pre-pubescent on the basis that they are not yet mature.
These aren't people who had trouble waiting for someone to get old enough - they are people who are attracted to children because they are children.
There is also a huge difference between being attracted to minors yet to hit puberty, and being attracted to someone who us past puberty but still a legal minor. We still consider it inappropriate and unlawful because that person can't develop consent - a case which is even stronger among prepubescent minors.
Trying to portray pedophiles as "just people who have a strong affinity for children" is radically misconstruing the reality of how pedophiles view children, sexually, and move to exploit them for personal pleasure. These are not people who "love children" and somehow end up reciprocating puppy-love that grows into a lasting relationship from a taboo.