Spencer was born circa 1760, the second son of Conway Spencer of Tremary and his wife, Mary. His brother was politician Joshua Spencer, and their nephew was politician George Canning II.
The Dos de Mayo Uprising of Spain against Napoleon found Spencer in command at Gibraltar. On his own initiative he sailed for Portugal and arrived at Mondego Bay on 5 August 1808. The force that he brought with him were 946 men of the 6th Foot, 806 soldiers of the 29th Foot, 874 troops of the 32nd Foot, 948 men of the 50th Foot, 929 soldiers of the 82nd Foot, and 245 artillerists. All infantry units consisted of 1st Battalions, except the 29th which was not organized into battalions. At Mondego Bay he joined his force to the 8,123 troops that General Arthur Wellesley brought from Cork in Ireland. At the Battle of Vimeiro in 1808, Spencer was Wellesley's second-in-command. He was made a Knight Companion of the Order of the Bath on 26 April 1809. He commanded the 1st Division at the Battle of Bussaco and in the 1810-1811 French invasion of Portugal. At the Battle of Fuentes de Onoro in 1811, Spencer, now a lieutenant-general, temporarily led both the 1st and 3rd Divisions. After Fuentes, Wellington went south to participate in the Siege of Badajoz. He left Spencer and 28,000 Anglo-Portuguese to defend the province of Beira in the north. Later that year, he was replaced by Thomas Graham, 1st Baron Lynedoch because of pessimistic letters which he had sent back to England. In 1825, Spencer was promoted to full general. Wellington wrote of Spencer, "He was exceedingly puzzle headed. He would talk of the Thames for the Tagus."
Later career
After having been in succession Colonel of the 9th Garrison Battalion and the 2nd West India Regiment and Colonel-Commandant of the 2nd Battalion of the 95th Regiment of Foot and the 1st Battalion of the Rifle Brigade, in 1818 he was given the colonelcy of the 40th Regiment of Foot, which he held until his death in 1828.
Romance with Princess Augusta
Spencer met Princess Augusta Sophia, the second daughter of King George III and Charlotte of Mecklenburg-Strelitz, about 1800, and a romantic friendship had developed by about 1803. In 1805 Spencer was appointed as an equerry to the king. According to Augusta, the couple conducted their relationship with great discretion as they were not of equal rank; news of such an attachment was feared to disturb the King's precarious mental health. In 1812 Augusta sent a long letter to her brother the Prince Regent to ask for his consent to her marrying Spencer, but it is unknown whether she did so. It is believed the couple remained together until Spencer's death in 1828.