As a youngster, Barbet represented AS Mouliets, Olympique Saint-Pey and FC Vallée de Gamage, before joining the youth setup at Ligue 1 club Bordeaux in July 2007. He made his debut for the club's reserve team in a 3–0 win against Saint-Alban on 2 April 2011 and went on to make a further four appearances during the final month of the season, helping the team to promotion to the Championnat de France Amateur. Over the next three seasons, Barbet established himself as a regular in the Bordeaux reserve line-up, making 70 appearances and scoring 12 goals. He finally won his maiden call into the first team squad for a Europa Leaguegroup stage match versus Maccabi Tel Aviv on 12 December 2013, but remained an unused substitute during the 1–0 defeat. It was Barbet's only involvement with the first team and he departed the Stade Chaban-Delmas at the end of the 2013–14 season.
Chamois Niortais
Barbet joined Ligue 2 club in Chamois Niortais in July 2014, signing a two-year contract after a successful trial period. Initially signed as a back-up player, Barbet was given his chance at Niort after first-choice central defender Frédéric Bong suffered a long-term injury. He made the first professional appearance of his career in a 1–1 draw with Créteil on 22 August 2014, partnering Matthieu Sans in central defence. Barbet scored his first goal for the side in a 1–1 draw with Orléans the following month. He made 33 appearances and scored two goals during a mid-table season for Niort and departed the club on 25 June 2015.
On 25 June 2015, Barbet moved to England to sign a four-year contract with Championship club Brentford for an undisclosed fee. He made his debut with a start in a 4–0 League Cup first round drubbing at the hands of Oxford United on 11 August and possibly put through his own net for the Us' second goal, which was credited to Danny Hylton. With Jack O'Connell largely preferred as third-choice centre back behind Harlee Dean and James Tarkowski, Barbet spent long periods in and out of the matchday squad through to the end of 2015. The unavailability of Tarkowski saw Barbet make his first appearance in over four months against Preston North End on 23 January 2016 and he assistedAlan Judge for one of Brentford's three goals in the victory. Tarkowski's subsequent departure from the club saw Barbet become Harlee Dean's partner in central defence through to the end of the 2015–16 season. He was sent offfor the first time in his career after just five minutes of a match versus Sheffield Wednesday on 13 February, which Brentford eventually lost 4–0. On his return to the team in March, Barbet scored his first Bees goal with what proved to be a consolation in a 2–1 defeat to Charlton Athletic. He finished the 2015–16 season with 19 appearances. After suffering an ankle injury during the 2016–17pre-season, Barbet missed the first five weeks of the regular season. The form of centre backs Harlee Dean, John Egan, Andreas Bjelland and the emergence of teenage left back Tom Field relegated Barbet to a substitute role through the first half of the season, with occasional appearances at left back. After just three appearances in the opening months of 2017, Barbet broke into the team at the expense of centre back John Egan in mid-March and finished the season with 25 appearances. Barbet began the 2017–18 season as one of manager Dean Smith's two first-choice centre backs, before dropping to the bench in October 2017. Despite being in and out of the team, he scored three goals in five appearances between September and November. With Rico Henry injured, Barbet was deployed as a left back for much of the season and finished the campaign with 37 appearances and three goals. Entering the final year of his contract, Barbet continued as Dean Smith's first-choice left back during the early months of the 2018–19 season and after losing his place to fit-again Rico Henry, he returned to the team in as part of a central defensive trio under Smith's replacement Thomas Frank in December 2018. Barbet continued as a starter in the new formation until suffering a season-ending ankle injury in a match versus Wigan Athletic on 30 March 2019. Barbet rejected a new contract during the season and ended his Brentford career with 118 appearances and seven goals.
Queens Park Rangers
On 18 June 2019, Barbet joined Queens Park Rangers on a three-year deal.
