Downes was born in Brentwood, Essex. He joined Ipswich Town at the age of seven, signing a two-year scholarship in May 2015. Downes signed a one-year professional contract with the club on 29 June 2017, with the option of a further year. However, after impressing manager Mick McCarthy during pre-season, he signed a new three-year contract with Ipswich on 19 July to keep him at the club until the summer of 2020. He made his professional debut on the opening day of 2017–18, coming on as a substitute for the injured Andre Dozzell in first-half stoppage time in a 1–0 home victory over Birmingham City.
On 31 January 2018, Downes joined League Two leaders Luton Town on loan until the end of 2017–18. He made his debut three days later, starting in Luton's 1–0 home win over Exeter City. Following the match, manager Nathan Jones said "He was outstanding. It's as if he lives here in terms of the way he went about the pitch, the confidence he played with, the security and everything he had". Downes finished the loan with 10 appearances, as Luton were promoted to League One after finishing second in League Two.
Return to Ipswich
Following his return from loan, Downes played regularly for the first team during the 2018–19 season, making 30 appearances across all competitions - ending in the team being relegated to League One. He signed a new three-year contract with the club in March 2019, with the option of a further year, and scored his first professional goal in a 3–2 home victory over Leeds United on the final day of the season. Downes remained a regular player for Ipswich following relegation to League One during the 2019–20 season. On 1 December 2019, Downes captained Ipswich for the first time in a FA Cup second round tie against Coventry City, which ended as a 1–1 draw. Aged just 20 years old, this made him the youngest ever player to captain The Tractor Boys in their history.
International career
Downes was called up to the England under-19 team on 24 August 2017 for two friendly matches against Poland and Germany. He made his debut on 5 September against the latter, starting in a 3–1 defeat, in which he was substituted in the 60th minute. He was capped five times by England at under-19 level. Downes was named on standby for the England under-20 team on 9 November 2018 for a match against Germany. He was called into the squad three days later and earned his first cap as an 89th-minute substitute in a 2–0 victory the following week.