He was one of only four peers who continued to support James II after the Prince of Orange embarked for England. On 18 December 1688 he accompanied King James to Rochester when he fled London. Elgin himself chose to remain in England; he was prepared in the short term to offer his support to the new regime, although his loyalty to it was always deeply suspect. In May 1695, Lord Elgin was accused, almost certainly with good reason, of having conspired to plan the restoration of King James II and in February 1696 he was imprisoned in the Tower of London, but admitted to bail a year later and allowed to leave England for Brussels. After more than 40 years in exile, he died in Brussels and was buried there. Some historians have accused him of double-dealing in swearing allegiance to William III while plotting the restoration of James; others argue that his true loyalty was to the institution of the monarchy, and that he supported whichever monarch seemed best fitted to rule at any given time. William III clearly did not regard him as a dangerous character, as shown by the fact that he was left in peace once he fled from England; he was fortunate in having a great many friends and very few enemies. It seems that from about 1710 he was free to return to England, but he was by then happily settled in Brussels, where he had made a second marriage for love to Charlotte, comtesse d'Esneux, and, since he was able to draw at least part of the revenue from his English estates, he had no pressing need and no apparent desire to return home.
Character
Ailesbury seems to have been almost universally liked even by his political opponents, having a reputation for honesty, decency and fair dealing. Charles II was clearly fond of him and confided in him to a degree unusual for such a secretive man; James II also liked him, and Louis XIV regarded him as almost the only British nobleman who was not motivated purely by self-interest. Though he changed allegiance himself he had no patience with time-servers: he detested Sunderland and in 1689 told his cousin Danby that for his treachery to James II he deserved to "be knocked on the head".
Memoirs
Ailesbury devoted many years to writing his Memoirs, which were not published until 1890. Historians have praised them highly, particularly for the vivid portraits of the leading figures in British life, including James II, William III, Danby, Sunderland, Lauderdale and Halifax. Perhaps the most striking feature of the memoirs is the author's absolute devotion to Charles II : "my good and gracious master, the best that ever reigned over us".