What you're thinking of (pick up and listen to see if anyone is already using the line) is a specific kind of telephone line called a "party line".
The concept was that you could get from the telephone company a dedicated line (you're the only customer) or for a discount a shared line (multiple customers using the same wire.)
If you got a shared line ("party line"), since you were not the only customer, you might find when you pick up the telephone that one of the other customers was in the middle of a phone call (so you couldn't place a new call.)
I heard about "party lines" from my parents and grandparents, never used one myself. So I can't speak from experience. But my sense is that they were more common among telephone lines installed as far back as the 1950s and faded from public awareness as technology improved, and eventually everyone got dedicated lines. As those became ubiquitous, most people assumed that's how telephones always worked.