Duff was the eldest illegitimate son of James Duff, 2nd Earl Fife, and Margaret Adam, of Keith. As his mother was of humble status, her three children with Fife were placed in the guardianship of William Rose, Fife's factor. Fife liked Duff and took good care of him, sending him to Keith Academy and at King's College, Aberdeen, where he graduated MA in 1771, and after he entered the army paid for Duff's promotions. Duff entered the army as an ensign in the 1st Foot Guards on 18 April 1769. He was promoted lieutenant and captain on 26 April 1775, and made adjutant of his battalion in 1777, and on 30 April 1779 he was knighted as proxy for the celebrated diplomatist Sir James Harris, afterwards 1st Earl of Malmesbury, at his installation as a knight of the Bath. James Duff was promoted captain and lieutenant-colonel, on 18 July 1780, colonel on 18 November 1790. His regiment was part of the expeditionary force sent to Flanders to campaign against the French as part of the First Coalition. He fought at the Battle of Valenciennes commanding the guards' light infantry battalion in 1794. On 30 October he was promoted to major-general. In 1797, Duff received the command of the Limerick district. While there he rendered important services during the Irish Rebellion of 1798, and managed to keep his district quiet in spite of the state of affairs elsewhere. His name is associated with the Gibbet Rath executions. He was elected M.P. for Banffshire, and colonel of the 50th Regiment of Foot from 1798 to death. Duff was promoted lieutenant-general on 1 January 1801, and general on 25 October 1809, and at the time of his death, at Funtington, near Chichester, on 5 December 1839, he was the most senior general in the British army, and was one of the few officers who held a commission for over seventy years. He had as aides-de-camp during his Limerick command two famous officers, William Napier and James Dawes Douglas. There are numerous allusions to him in the Life of Sir William Napier. According to the Legacies of British Slave-Ownership at the University College London, Duff was awarded a payment as a slave trader in the aftermath of the Slavery Abolition Act 1833 with the Slave Compensation Act 1837. The British Government took out a £15 million loan with interest from Nathan Mayer Rothschild and Moses Montefiore which was subsequently paid off by the British taxpayers. Duff was associated with "Jamaica Hanover 455 ", he owned 202 slaves in Jamaica and received a £4,101 payment at the time. This is recorded by the Slave Compensation Commission and the records held at the National Archives in London.
Family
On 12 August 1785 Duff married Basilia, daughter and heir of James Dawes of Rockspring, Jamaica, through whom he gained control of a considerable fortune. They had one son and three daughters.
