Andrew was born Andrew Wedderburn in 1779. His sister Jean married Thomas Douglas, 5th Earl of Selkirk, in 1807. His brother James Wedderburn was Solicitor General for Scotland until his death in 1822; his posthumous daughter Jemima Blackburn became a highly regarded artist, and played the role of an elder sister to James Clerk Maxwell, the physicist. Andrew's other legitimate brother was Peter Wedderburn Ogilvy, who became a sea captain; his sons went into the army. Andrew had a half-sibling whom he very publicly rejected: the son of his father by an enslaved woman in Jamaica. When the mixed-race Robert Wedderburn showed up at the family seat seeking to claim kinship, he was sent away with a flea in his ear. Following this rejection, Robert wrote The Horrors of Slavery, which was circulated by the abolitionist movement. Andrew furiously denied the claims, and in turn insulted Robert's mother.
Career
Andrew remained in Europe, inherited his father's estates and set up as a sugar broker. When his brother-in-law began buying into the Hudson's Bay Company, Andrew followed suit. By 1810, he was on the HBC board and worked to rationalize the company's administration. In 1820, he was largely responsible for sending out Sir George Simpson to take charge of HBC affairs in Canada. During Simpson's long administration the two worked closely together, one in London business circles and the other in the wilds of Canada.
Family
His grandfather, Sir John Wedderburn, 5th Baronet of Blackness, was involved with the Jacobite rising of 1745, and was convicted of treason. The punishment for this was threefold: the death penalty, the confiscation of all his estates, and the attainting of his family, including the baronetcy. At least two of his sons moved to Jamaica, including Andrew's uncle and father. The former, John Wedderburn of Ballendean, is notable for the civil case brought under Scots law by his former slave Joseph Knight. Andrew's father, James Wedderburn, set up as a doctor without academic qualification. After 28 years, James had become a wealthy slave-owner and sugar planter. In 1773 James moved back to Inveresk—the estate had been restored—and married. At some point James changed his name from Wedderburn to Wedderburn Colvile. His first wife was Elizabeth Susannah, daughter of John Wedderburn of Clapham, but she died in 1803, only a year after their marriage. They had no children, but he inherited the Wedderburn estate in Jamaica. In 1806, he married the Hon. Mary Louisa Eden, fifth daughter of William Eden, 1st Baron Auckland, and was thus connected to an influential family of politicians and diplomats. The couple had four sons and 12 daughters: