Brown worked his way through the youth team of Queens Park Rangers, earning a professional contract in July 2008 having been part of the Football League Youth Alliance winning side the previous season. He spent the first half of the 2009–10 season on loan at administration-stricken Salisbury City of the Conference National, along with teammate Joe Oastler. He made fourteen appearances for the club, including twelve starts, before returning to Loftus Road before the end of the calendar year. On 24 April 2010 he made his debut in the Football League, replacing Tamás Priskin as a substitute in a 1–0 win over Barnsley at Oakwell. At the end of the season Neil Warnock offered him a new one-year deal, which he duly signed. "Lee Brown has emerged as an exciting prospect from the QPR youth system in recent years and was rewarded with a one-year deal at QPR in 2009. He, along with several youngsters were given the opportunity to train with the first team in the latter part of the 2008–09 season, after impressing in the reserves. The left-back played an important role in QPR's pre-season of 2009–10 and was subsequently given his first squad number in light of the lack of full-backs at the club. Brown, along with fellow defender Joe Oastler, was loaned out to Conference Premier side Salisbury City. His stay there was curtailed due to Salisbury being unable to renew their loan deals" In September 2010, Brown joined Hayes & Yeading United on loan, initially for one month before being extended for a second. Brown returned to Church Road on transfer deadline day in a loan deal until the end of the 2010–11 campaign, snubbing a similar move to League One side Leyton Orient.
Bristol Rovers
2011–2014
Brown was released by QPR following the 2010–11 season and went on to sign a 2-year contract with recently relegated League Two outfit Bristol Rovers. On 6 August 2011, Brown made his Bristol Rovers debut in the opening game of the season, live on Sky Sports, against AFC Wimbledon. Rovers won the game 3–2. In his first season at Rovers he appeared in 42 league games and scored seven goals as the club turned around a difficult first half to the season, which resulted in the sacking of manager Paul Buckle who was replaced by Mark McGhee, to finish in a comfortable, albeit disappointing, 13th place. On 18 July 2012, Brown signed an extension to his Bristol Rovers contract to keep him at the Memorial Stadium until the summer of 2014. He was again a regular for Rovers as he made 39 league appearances, scoring three times. For Rovers, the season was again a struggle in the first half, spending two different spells in the relegation zone leading to the dismissal of Mark McGhee. He was replaced by the returning John Ward who successfully guided Rovers to 14th in League Two. Brown's third season at Bristol Rovers started with the club among the pre-season favourites for promotion. The season proved to be a disaster for Rovers as the club was relegated out of the Football Leaguefor the first time since joining it in 1920. Brown made 41 appearances, scoring twice. Despite attracting interest from fellow Football League sides, Brown signed a new contract with Rovers in July 2014. Brown stated "I thought I had to give something back to " and that manager Darrell Clarke, who had replaced John Ward toward the end of the previous season and had played alongside Brown at Salisbury City in 2009, "has got to take a lot of credit for me staying". On 14 October 2014, he made his international debut, representing England C in a 2–0 defeat to Turkey B in Istanbul.
2015–2018
Brown achieved promotion to the Football league via a play-off victory in the 2015 Conference Premier play-off Final. The 2015–16 season was arguably his best season ever as a footballer as he played every minute of Rovers' games. He scored 6 goals over the course of the season after opening his tally in a 3–1 home win against Barnet. On 14 April 2016, Brown signed a new and improved deal with the club. His last goal of the season proved to be the most important as his 92nd-minute winner on the last game of the season against Dagenham & Redbridge made Rovers a League One side once again. Brown scored his first goal of the 2016–17 season in an FA Cup first round tie against Crawley Town, striking a superb shot from 35 yards into the bottom left corner. Brown started on the opening day of the 2017–18 season in a 1–0 defeat to Charlton Athletic. He also played the following Tuesday in a 4–1 Carabao Cup victory over Cambridge United in which he got two assists; a 40-yard pass to Billy Bodin for the first goal and a delightful cross to Ellis Harrison for the third goal. Following the 1–1 draw with Wigan Athletic, it was confirmed that Brown would leave Rovers at the end of the season, having made over 321 appearances for the club, in order to be closer to London. Manager Darrell Clarke confirmed his last game, the forthcoming home fixture with Gillingham, would see Brown take the captain's armband.
Portsmouth
On 11 June 2018, Portsmouth announced they had signed Lee Brown on a two-year deal from Bristol Rovers on a free transfer. He officially became a Pompey player on 1 July 2018 when his contract with Bristol Rovers expired. In March 2019, he won his first piece of silverware when Portsmouth won the 2018–19 EFL Trophy, defeating league rivals Sunderland5–4 on penalties following a 2–2 draw after extra time.