Lord Jersey served as a Lord-in-waiting between 1875 and 1877 in the Conservative administration of Benjamin Disraeli. He returned to the government in 1889 when Lord Salisbury made him Paymaster-General, which he remained until 1890. The latter year he was sworn of the Privy Council and made a Knight Grand Cross of the Order of St Michael and St George. In August 1890 Jersey was appointed Governor of New South Wales. He arrived in Australia to take up his position in January 1891. According to the Australian Dictionary of Biography there were no major political difficulties during his term. He was described by Sir Henry Parkes as "amiable and well-intentioned", but "very much occupied with his own family". He "did not excel as a public speaker". He was the official host at the 1891 Australasian National Convention in Sydney. Jersey tendered his resignation already in November 1892 citing pressing business affairs. This did not go down well with the Colonial Office in London. Lord Salisbury thought that Jersey had found that there was "less individual power to his office than he imagined". Jersey himself wrote to the Colonial Secretary: "the duties and responsibilities of a governor can hardly be called serious nowadays being chiefly of a social character". He left Australia in March 1893. Lord Jersey represented the United Kingdom at the 1894 Colonial Conference in Ottawa, Ontario, Canada. He also acted as New South Wales agent-general in London between 1903 and 1905 and through his ties with the banking institutions helped the state's loan negotiations. He revisited Australia in 1905 and Prime Minister Alfred Deakin considered appointing him Australia's first High Commissioner to London, although nothing came out of this. One of Lord Jersey's godparents was Queen Victoria. The Queen accepted her role as a token of friendship to Robert Peel, Prime Minister, who was Lord Jersey's grandfather.
A Freemason, he was initiated to the craft on 25 October 1865 in the Apollo University Lodge No. 357 at the age of 20. In December 1865 he was passed in the Churchill Lodge No. 478 and in February 1866 he was raised in his mother Lodge. In 1870 he was appointed Senior Grand Warden of the United Grand Lodge of England and served for a year. In 1885 he was appointed Provincial Grand Master of Oxfordshire. When he became Governor of New South Wales, he became a member of the Lodge Ionic No. 65. On 11 June 1891 he was installed Grand Master of the Grand Lodge of New South Wales.
Having suffered a stroke in 1909, Lord Jersey died at Osterley Park, Middlesex, in May 1915, aged 70. He was succeeded in the earldom by his eldest son, George. The Countess of Jersey survived her husband by 30 years and died at Middleton Park, Oxfordshire, in May 1945, aged 95.