When the minor-league Buffaloes were succeeded by the MLB Houston Colt.45s in, Richardson joined the expansion team in its debut National League season as the Colt.45s' business manager. The team was renamed the Astros in. Richardson took over as Astro general manager after his two predecessors, Paul Richards and Tal Smith, had built one of the most talented farm systems in baseball. In its five-year history, the Houston franchise had produced star players Rusty Staub, Joe Morgan, Jimmy Wynn, Larry Dierker and Dave Giusti, with future standouts Doug Rader and Don Wilson ripening in its minor league system. But Paul Richards was fired by Astro owner Roy Hofheinz after the 1965 season. Houston spent with Smith heading up a three-man baseball operations management team that included Richardson and field managerGrady Hatton. Then, during the 1966–67 off-season, Hofheinz named Richardson the club's official general manager, a job he would hold for almost nine full seasons. Richardson traded 31-year-old Mike Cuellar to the Baltimore Orioles for Curt Blefary on December 4, 1968. Cuellar appeared in three consecutive World Series with the Orioles from 1969 to 1971 including a 1970 Championship run, was a co-recipient of the American LeagueCy Young Award in 1969, had at least 20 wins in each of four seasons and won 143 games in eight years in Baltimore. Blefary spent only one season with the Astros before being dealt to the New York Yankees. Richardson then traded Staub — a 24-year-old outfielder and first baseman who displayed remarkable ability to hit for high average and decent power within the confines of the cavernous Astrodome — to the Montreal Expos for Jesús Alou and Donn Clendenon. Clendenon refused to report, however, and Montreal instead sent Jack Billingham, Skip Guinn, and cash. The club may have felt that Staub's ankle problems would shorten his career, but Staub instead continued his exceptional hitting in Montreal, Detroit, and New York. John Mayberry – another young first base prospect – was traded to the Kansas City Royals for mediocre bullpen help. There he became an integral part of the Royals. However, Richardson's most memorable trade featured Morgan, the Astros' star second baseman, who was dealt to the Cincinnati Reds in exchange for first baseman Lee May, second baseman Tommy Helms and utility infielder Jimmy Stewart. Morgan went on to become the offensive heart of the Big Red Machine, who would later win back to back World Series championships in both 1975 and 1976 and is widely held by many to be the greatest second baseman of the second half of the 20th century. He was elected to the Baseball Hall of Fame in 1990. That trade is generally regarded as being one of the most lopsided in the history of Major League Baseball. To his credit, Richardson recognized César Cedeño as a potential superstar, but the Astros spent his tenure and several years after it mired in mediocrity. During Richardson's nine-year stewardship, Houston would post only four.500 or above records. On July 11, 1975, with Houston at 32–58 and lodged in last place in the National League West Division, Richardson was fired and soon replaced by Smith, who returned to the Astros after a stint as an executive with the New York Yankees. Sabermetric research shows that the Astros would likely have been the best team in the National League throughout much of the 1970s. Instead, the team would not appear in the playoffs until 1980.
San Francisco Giants
Richardson also served as general manager for the San Francisco Giants from December 1975 to July 8, 1981. He joined the Giants on the recommendation of other National League owners during a transitional period, while the team was sold by longtime owner Horace Stoneham, eventually to Bay Area businessman Bob Lurie. For four of the five full seasons during which Richardson helmed the Giants, the team scuffled on the field, averaging only 74 wins a year. The exception, however, was. Led by slugger Jack Clark and pitchers Vida Blue and Bob Knepper, the 1978 Giants posted an 89–73 record to finish a strong third in the NL West, only six games behind the eventual league champion Los Angeles Dodgers. Richardson was hailed as the Major League Executive of the Year by The Sporting News. But the Giants returned to mediocrity in, and Lurie fired Richardson in favor of farm system director and former Giants' catcherTom Haller in July 1981. Richardson, then 58, finished his career in a consultants' role with the organization. He died in Columbus, Georgia at the age of 93 on April 12, 2016.