In Victorian era literature the term that was most commonly used was 'pederast'. This word comes from the Ancient Greek and refers to a relationship between a man and an adolescent youth.
Think cabin boys and all that nonsense. I get the impression that in the ancient world girls/women simply didn't count, so no-one thought to engage in sex with them. That appears to be a later invention.
https://en.wiktionary.org/wiki/pederast