Born in Bacup, Lancashire, James began his football career as a youth player with local club Rossendale United, and spent time with Blackburn Rovers and Preston North End before signing for Manchester United in July 2012.
Manchester United
He signed for Carlisle on a half-season loan contract in July 2013, and made his professional debut on the opening day of the 2013–14 season in Carlisle's 5–1 home defeat to Leyton Orient. However, he suffered an injury shortly before half time in Carlisle's penalty shoot-out victory against Blackburn Rovers in the League Cup on 7 August, and returned to Manchester United on 24 September 2013. After returning to Manchester United, James was ever-present in the club's reserve team for the remainder of the season, and his performances led to a nomination for the Denzil Haroun Reserve Player of the Year award, ultimately losing to Saidy Janko. He made his first-team debut in the opening match of the club's pre-season tour of the United States on 23 July 2014, and scored twice from a left-wing-back role as Manchester United beat the LA Galaxy 7–0. On 26 November 2014, James was sent on loan to Rotherham United for two months. After eight appearances, including one in the FA Cup, he returned to Manchester United on 26 January 2015. On 26 March, he joined Huddersfield Town on loan until the end of the 2014–15 season. He made his debut in the 1–1 draw against Sheffield Wednesday at Hillsborough Stadium on 4 April 2015. His first goal for the club came in the 4–4 draw with Derby County at the John Smith's Stadium on 18 April 2015, when he scored directly from a corner to give Town a 3–1 lead.
Wigan Athletic
On 21 July 2015, James joined Wigan Athletic on a three-year deal for an undisclosed fee. After being largely ever present for the first half of the season an injury suffered in a 3–1 victory over Chesterfield, in which James scored his first goal for the club, led to him missing the remainder of the 2015/16 season and the majority of the following season. He was released by Wigan at the end of the 2017–18 season.
Sunderland
He subsequently signed a one-year deal with Sunderland on 2 July 2018.
Doncaster Rovers
James signed for Doncaster Rovers on a two-year deal on 19 June 2019 for an undisclosed fee. On 5 October, he scored his first goal for the club in a 1–2 defeat against Portsmouth.