Born in Bangui, Youga began his career at French club Olympique Lyonnais. He transferred to Charlton Athletic on a free transfer in July 2005 after finishing his school exams. Before playing a game for Charlton, in October 2005 he was loaned to Bristol City, where he played four times for the first team. In 2007, he was loaned to Bradford City until the end of the 2006–07 season, playing 11 times for the club before his loan spell was cut short by injury, as they were relegated from League One. In summer 2007 he turned down a second loan to the club. He subsequently joined newly promoted Championship side Scunthorpe United on a six-month loan deal. He returned to Charlton in January 2008 after making 19 appearances and scoring once for the Iron against Barnsley. He made his senior debut for Charlton on 5 January 2008 in the side's 1–1 FA Cup draw with West Bromwich Albion, before making his league debut for the club a week later on 12 January 2008 against Blackpool in a 4–1 win. Furthermore, Kelly Youga cemented his place as the regular first choice LB at Charlton Athletic. He scored his first goal for Charlton Athletic in a 2–2 draw away at Plymouth Argyle with a header. 2008–09 season was Youga's most successful spell with Charlton Athletic, playing 36 games and scoring once. Also 2009–10 season started promisingly, Youga occupying a full back spot in the Charlton Athletic XI. However, Youga suffered a knee injury in November 2009 which limited his presence in competitive matches to 22 appearances in 2009–10. After suffering another knee injury in October 2010 Youga did not play any games for Charlton Athletic that season except for one outing with the Addicks reserve side. In May 2011 Youga's contract with Charlton expired leaving the left back without a club. Although Youga trained with Charlton Athletic in the pre-season of 2011, a new deal was not agreed.
On 19 January 2012, Youga signed for League One side Yeovil Town on a short-term contract until the end of the 2011–12 season. On 31 March 2012, Youga's contract was terminated by mutual consent having only made one 45-minute appearance for Yeovil Town.
On 13 November 2012, Youga signed for Championship side Ipswich Town on a short-term deal until January 2013. Youga's contract ran out in the end of January 2013.
On 11 March 2013, as a free agent, Youga signed for League Two club AFC Wimbledon. Dons manager Neil Ardley said of the signing that "Kelly can play anywhere across the back four," adding that he is "a player who was playing in the Championship at his peak so hopefully he can bring that experience for us." Youga made three appearances for AFC Wimbledon before being released from the club with nine other players on 14 May 2013.
On March 26, 2015 League One club Crawley Town announced they had made a short-term contract with Youga until the end of season 2014/2015 after a successful trial spell.
