After graduating in 1981, Bailiff's first coaching job was in the high school ranks in New Braunfels, Texas. After three years coaching the defensive line at New Braunfels High School, he left coaching and entered the private sector for four years. He returned to coaching in 1988 as a graduate assistant at Southwest Texas and was promoted the next year to a full-time position that he held until he was hired to coach the defensive line at the University of New Mexico by Lobos' head coach Dennis Franchione. In 1997, Bailiff returned to Southwest Texas as the defensive coordinator. In 2001, he was hired to serve in the same role at TCU by coach Gary Patterson. Following the 2002 season, in which the Horned Frogs' defense, led by All-American linebacker LaMarcus McDonald, ranked 2nd in the nation, Bailiff was named the nation's Top Assistant Coach by the All American Football Foundation.
Texas State
On February 5, 2004, Bailiff again returned to his alma mater, which by now was known as Texas State University, this time as head coach. In his first season as the Bobcats' head coach, he guided them to a 5–6 record. In 2005, they finished the regular season 9–2 and were Southland Conference Champions. They then won two games in the NCAA Division I-AA playoffs, eventually losing to Northern Iowa. In 2006, the Bobcats were again 5–6.
Rice
On January 19, 2007, Rice University hired Bailiff as head coach, replacing Todd Graham. In 2008, he led Rice to a 9–3 record and an appearance in the 2008 Texas Bowl—only their second bowl appearance in 47 years. The Owls won that game 38–14 for Rice's first 10-win season since 1949, and Bailiff was rewarded with a five-year contract extension. In 2012, Rice returned to bowl action with a win over Air Force in the Armed Forces Bowl. The following season, Bailiff became the first coach in Rice history to record two 10-win seasons as Rice won its first outright conference championship since 1957 by defeating Marshall 41–24 and earning a berth in the Liberty Bowl. In 2014, Bailiff led the Owls to a third consecutive bowl game as Rice defeated Fresno State 30–3 in the Hawai'i Bowl. In 2017, Bailiff was fired after the conclusion of a 1-11 season, his worst at Rice.
Texas A&M–Commerce
On December 9, 2018, Bailiff was hired as the head coach at Texas A&M University–Commerce. He replaced Colby Carthel, who had been hired at Division I Stephen F. Austin State University. Bailiff was hired mainly due to his familiarity with the Lone Star Conference as a player, his relationship with Texas high school coaches, and having had success at schools that had traditionally not been football powers. Unlike his previous tenures at Rice and Texas State, Bailiff was taking over an A&M-Commerce program that had deep tradition and that was considered by most Division II football observers to be the best Division II program in Texas. In his first season at Texas A&M-Commerce, he led the Lions to an 11-3 record, finishing 2nd in the Lone Star Conference. The Lions went to the NCAA Division II playoffs for the fifth straight season. In the first round, they avenged their only conference loss by defeating # 3 and undefeated Tarleton State University, 23-16. In the second round, they defeated Rocky Mountain Athletic Conference Champion and previously undefeated Colorado School of Mines 23-3 to send them back to the Region 4 Championship for the second time in 3 years. They were defeated by # 4 Minnesota State-Mankato 42-21 to end their season at 11-3.
