In 1754 he became Member of Parliament for Banffshire, was re-elected in 1761, 1768, 1774, and 1780, and in 1784 elected to represent Elginshire until 1790. He gave the Banff town of Macduff its name, having changed it from Doune in 1783. He extended the town and built a harbour at a cost of £5,000 ensuring economic prosperity. Duff was one of the most powerful and influential men in Aberdeenshire, Banffshire and Morayshire due to his massive estate. Due to his political commitments in Westminster, the running of the Estate was largely left to his factors - in particular William Rose of Ballivant. For his service to Duff, Rose was awarded many political favours. First, Duff gave Rose a vote in three different constituencies, he appointed him the first Provost of Macduff, and latterly the Sheriff Clerk of Banffshire. He was created a British peer by the title of Baron Fife, 19 February 1790. He held the appointment of lord-lieutenant of county Banff.
Estates
He succeeded his father in the title and estates in September 1763, and devoted himself to the improvement of the property, which he largely increased by the purchase of land in the north of Scotland. He was twice awarded the gold medal of the Society for the Encouragement of Arts, Manufactures, and Commerce, for his plantations, with which he covered fourteen thousand acres. He offered the farmers on his estate every inducement to cultivate their land on the most approved principles, and himself set the example by instituting near each of his seats a model farm, where agriculture and cattle-breeding were carried on under his personal supervision. In 1782 and 1783, when all crops failed, he allowed his highland tenants a reduction of twenty per cent. on their rents, and disposed of grain to the poor considerably below the market price, importing several cargoes from England, which he sold at a loss of £3,000. In 1787 he was elected a Fellow of the Royal Society.
Family
James Duff married on 5 June 1759, Lady Dorothea Sinclair, only child of Alexander, ninth earl of Caithness, but he had no issue, and his British peerage became extinct on his death. He was succeeded in his Irish earldom by his next brother, Alexander. However he had three children by his mother's personal maid, Margaret Adam of Keith, all of whom were born before his marriage to Sinclair.
Death
He died at his home, Fife House, Whitehall, London, on 24 January 1809, and was buried in the mausoleum at Duff House, Banffshire.
