William and Alva had three children; but by 1894 tensions between the pair were obvious. When William returned from Europe—alone—on December 12, 1894 The New York Times reported “Upon arriving in town, Mr. Vanderbilt did not go to his own house, 660 Fifth Avenue, but to the home of his mother, Mrs. William H. Vanderbilt, at 640 Fifth Avenue. He would see no reporters last night.”
Before long the drawing rooms of Fifth Avenue were shaken by Alva’s suit for divorce, citing William’s extramarital dalliances. She walked away with $10 million and much real estate, including the imperial Marble House in Newport. But William retained the Fifth Avenue mansion.
Eight years after Alva Vanderbilt walked out of No. 660 Fifth Avenue there was a new Mrs. Vanderbilt. William married Anne Harriman, the daughter of banker Oliver Harriman. Anne had been married and widowed twice–to Samuel Stevens Sands and to Lewis Morris Rutherfurd, Jr. The middle-aged couple (William was now 54 years old) would have no children. But Anne Vanderbilt successfully picked up the reins as hostess of the Petite Chateau.
In 1906 Vanderbilt’s son, William Jr., commissioned McKim, Mead & White to design a compatible mansion next door at No. 666 Fifth Avenue. Existing brownstone homes were demolished and the firm produced a mansion so harmonious with No. 660 that it was often confused as one building.
In July 1920 William Kissam Vanderbilt died while traveling in Paris. Within four months the mansion that had been his home for nearly four decades was on the market. The Times reported on November 10 “Negotiations are in progress for the sale of the mansion of the late William K. Vanderbilt at 660 Fifth Avenue…and reports among real estate men yesterday were that the deal would be concluded before the end of the week, with the purchase price between $2,500,000 and $3,000,000.”
Two weeks later the Real Estate Record & Builders’ Guide reported that the Empire Trust Co. had purchased the house. “The structure will be remodeled into a banking house.” The New York Times was less optimistic saying “The ornate graystone structure…will probably be replaced by a modern skyscraper.”
As it turned out, the Empire Trust never converted the house to a bank branch as intended. Instead it put the house back on the market. It sat empty for five years before being purchased by Benjamin Winter in May 1925. The Times reported on May 20 that “the real estate operator…will erect a twenty-story business building on the site.”
Manhattan being Manhattan, that structure did not last, either.