Felt was born to a Jewish family in Manhattan growing up around West 140th Street. His father was a lawyer. At the age of 19, he graduated from the Wharton School of the University of Pennsylvania with a degree in Finance and then took a job with the Wall Street concern, Hayden Stone. At the age of 28, he left the firm and helped to reorganize the investment firmGraham-Paige. In 1959, with Felt as its president, Graham-Page purchased a controlling interest in the old Madison Square Garden. Wanting to replace the old Garden with a new and modern facility that was more flexible, could handle larger crowds, and provided unobstructed views, in November 1960, he quietly purchased from the Pennsylvania Railroad the rights to build at Penn Station. In July 1961, Felt announced that he would demolish McKim, Mead, and White's Pennsylvania Station at West 33rd Street and 7th Avenue and build a new Madison Square Garden at the site. Prior to the hearing for the demolition, his brother, James Felt, the current chair of the New York City City Planning Commission, resigned. He formed a new company, Madison Square Garden, Inc. which was 75% owned by Graham-Page and 25% owned by the Pennsylvania Railroad. From 1960 to 1968, Felt oversaw the project from demolition to completion. The project was fully privately financed. Despite the controversy generated over the demolition, Felt stated that he "believed that the gain from the new buildings and sports center would more than offset any aesthetic loss" and that "Fifty years from now, when its time for to be torn down, there will be a new group of architects who will protest." Thereafter, he served as president of the Madison Square Garden Corporation when it purchased the New York Rangers hockey club, the New York Knicks basketball club, Roosevelt Raceway on Long Island, and the ice show production company, Holiday on Ice Productions. He is known for bringing the "Fight of the Century" between Muhammad Ali and Joe Frazier to Madison Square Garden on March 8, 1971. The theater within the new Garden was named Felt Forum in his honor; it is now known as Hulu Theater.
Felt was married twice. In 1945, he married Elaine Edelman who was also Jewish; they had two children, Constance Harriet Felt and Jonathan Seth Felt. They divorced and in 1973, he married Serene Leavenworth, sister to the Bihari brothers. He has a stepson with Bihari, Randolph Leavenworth III.