Vanderbilts owned a whole bunch of that area.
651 (caddycorner to 666): This entire block was the site of the Catholic Orphanage until 1900. The Vanderbilts bought up most of the land in 1902 to prevent a hotel from being built on this corner, which instead became the Morton F. Plant House (Robert W. Gibson, 1905). Plant sold it to Cartier in 1915 for $100 and a million-dollar pearl necklace.
The Aeolian Company then entered a 63-year lease with Charles A. Gould for a new building at 689 Fifth Avenue on the former site of William Rockefeller's brownstone mansion.
http:// www.nysonglines.com/5av.htm