It's a bit more complicated than that….
The concept servant was a bit different back then, as was the concept of "wife" versus a concubine.
Social status wasn't quite what we have today. Most people were nameless… We think of NPCs, today - but if you didn't know a trade, you were effectively a servant. Typically, only men with claims to a trade or political station held a wife. Servants had relations among each other at times, but concubines were also common. The servants were supported by the master, but almost like tenants of a house.
We… Don't exactly live that way, anymore - and arguably, this was more of a trend among the Arab and Med. Tribes. The Germanic/Slavic tribes trended toward what we understand as the western family structure, today. As I understand it. Although there was quite a bit of bandit raids and slave taking, a lot of that was to fund the ottoman slave trade.