The building was constructed in 1884, designed by architect Joseph M. Wells of the architecture firm McKim, Mead and White. The chosen style was Renaissance revival. Among the artists who worked on the building's elaborate interior were artist John La Farge, sculptor Augustus Saint-Gaudens and painter Maitland Armstrong. The houses were commissioned by Henry Villard, then president of the Northern Pacific Railway, shortly before his railroad empire began to crumble. Ownership of the building changed through many hands throughout the century. In 1948, the Archdiocese of New York purchased the houses at 451 and 453 Madison Avenue, and in early 1949 it purchased 455 Madison Avenue and 24 and 30 East 51st Street. The houses themselves are now owned by the Sultan of Brunei, while the land remains the property of the Archdiocese of New York under a 99-year lease. In 1968, the New York City Landmarks Preservation Commission designated the complex an historical landmark. A combined restoration/new hotel complex construction project was proposed by real estate developer Harry B. Helmsley who constructed the 51-story New York Palace Hotel tower directly behind the original building. The project was designed by architects Emery Roth & Sons and Hardy Holzman Pfeiffer in 1977 and completed in 1980. Office space was furnished for city preservation group The Municipal Art Society, as part of an agreement to save the building from demolition. As a part of the renovation of the hotel, part of what is now known as the Villard Mansion is available as an event rental.. The exterior of the buildings were restored in 2003. In 2014, the residences were available for public viewing when they hosted the annual Kips Bay Decorator Show House program to raise money for the Kips Bay Boys & Girls Club.
Exterior shots of the Villard Houses were used on the TV seriesGossip Girl as the hotel where the van der Woodsen family lives. On the popular NBC sitcom, Will and Grace, the wealthy socialite Karen Walker lived in a suite at the Palace Hotel/Villard Houses.
