Passley started his football career at Fulham and joined them at age eleven. He was initially a winger but switched to right back at sixteen, as well as playing at centre-back. Passley signed his first professional contract in 2013. On 26 January 2014 he made his debut for Fulham in a 1–1 draw against Sheffield United in the FA Cup fourth round, playing the full ninety minutes. Passley joined Shrewsbury Town on a youth loan on 27 November 2014. He made his Football League debut two days later in a 1–0 win over Burton Albion at New Meadow. Despite being ever-present for the duration of his loan period, he returned to his parent club when his loan deal ended on 5 January 2015. On 29 January 2015, Passley joined Portsmouth on an initial one-month youth loan. He made his Portsmouth debut two days later, playing 90 minutes in a 0–0 draw at Wycombe Wanderers. During his run of first team games, he helped the club achieve three clean sheets in three games, playing in a wing-back role. This was the same position he played while at Shrewsbury Town. Having made six appearances, Passley's loan spell with Portsmouth was extended for another month and again until the end of the season. His loan spell with Pompey came to an end on 12 April 2015 when he was taken off injured in the 83rd minute after losing consciousness in a defeat at Morecambe. At the end of the 2014–15 season, Passley was among nine players to leave Fulham.
Dagenham & Redbridge
Passley joined League TwoDagenham & Redbridge on a one-year deal in July 2015. Despite being a virtual ever-present for The Daggers in his first season, Passley was among eleven players to leave following relegation to the National League.
Passley joined National League South club Whitehawk in February 2017, the team he scored against in an FA Cup second round replay on 16 December 2015 in a match televised live on BT Sport. After helping Whitehawk secure safety in the National League South Passley left the club at the end of the season.
Dover Athletic
On 19 July 2017, Passley signed for Dover Athletic on a one-year deal. Passley started on the opening day of the season in a 1–0 away victory to Hartlepool United. Passley signed a one-year contract extension on 21 April 2018 after making 33 appearances in his debut season for the club. Passley scored his first goal for the club in the last match of the 2018–19 season, winning and scoring a last minute penalty in a 3–0 victory over Sutton United.
