Colley started out playing youth football with Wallidan FC who are a part of his home country's top flight GFA League First Division. After he started playing for the first team he trialed with Major League Soccer club Sporting Kansas City on two separate occasions in 2010 and 2011. The Americans wanted to sign Colley but refused to pay any form of transfer fee or training compensation to his Gambian club. Wallidans founder called their offer "out of touch with modern football" and "an insult to the local clubs" so Colley instead moved to Real de Banjul where he helped the team become Gambian champions in 2012.
KuPS
In January 2013, he signed a two-year contract with KuPS. During the time between his two seasons in Finland he trialed with a couple of 2. Bundesliga clubs and Arminia Bielefeld made an offer to buy Colley. An agreement was not reached however as they could not agree on the transfer fee. As his contract with the Finnish club was nearing the end Colley was described as "perhaps our best foreign signing ever" by the KuPS club president.
In August 2014, Colley signed a three-year contract with Swedish side Djurgårdens IF, starting the first of January 2015. He played his debut game for the club on 22 February 2015 against Ängelholms FF in the Swedish Cup. On 12 August 2016, Colley travelled to Belgium to complete a medical with top-tier club K.R.C. Genk and by doing that Colley missed the training session ahead of the match versus IF Elfsborg which was against the will of Djurgården. Later that day the Director of Sport of Djurgården Bo Andersson told the football site Fotbollskanalen.se that Djurgården were prepared to take the case to FIFA due to the fact that K.R.C Genk without the permission and knowledge of Djurgården booked a flight and hotel for Colley which violated the contract between Colley and Djurgården. Andersson stated that Djurgården would refuse to sell Colley to Genk after that incident. On 14 August, Colley started the match against IF Elfsborg on the bench due to the recent event. The following day Colley was sold to Genk despite the promise not to sell Colley to Genk. The transfer sum was reported by the Swedish newspaperExpressen to be €2 million and a resale clause which would make Colley the third most expensive defender ever sold by an Allsvenskan team after former Djurgården player Daniel Amartey and IF Elfsborg player Jon Jönsson.
