Von Armansperg was born in Kötzting, Lower Bavaria, nowadays part of Upper Palatinate, in 1787. His father was Count Joseph Felix von Armansperg and his mother was the Baroness Ludovica Verger von Moosdorf. His grandparents were Count Franz Xaver Ignaz Joseph von Armansperg, imperial Chamberlain and Maria Anna Elisabeth von Sainte-Marie Eglisine. In the Napoleonic Wars, in 1813–1814, he was Commissioner of Bavaria in the allied army, and belonged to the board which governed the conquered regions on the Rhine River. He participated in the Congress of Vienna in 1815, and was one of the plenipotentiaries with the allied army during the occupation of France, and administered a large district of that country. In 1825 he was chosen President of the Bavarian Chamber of Deputies, and became leader of the moderate opposition. King Ludwig of Bavaria made him Finance Minister and Minister of Foreign Affairs. He was one of the founders of the German Zollverein, a tariff union instrumental in advancing German unification. By his opposition to the Catholic ultramontanes, he forfeited the confidence of the king, and retired into private life.
In Greece
When King Ludwig's son Otto was offered the Greek throne in 1832, King Ludwig made von Armansperg the President of the Privy Council and the 1st representative of the new government. The other members of the were Karl von Abel and Georg Ludwig von Maurer with whom von Armansperg clashed often. After the King reached maturity in 1835, von Armansperg was made Arch-Secretary but was called Arch-Chancellor by the Greek press. Von Armansperg became an almost independent political actor in Greek politics as time went on; finding himself increasingly at odds with the king he was sent to support and advise. The situation came to a head when Otto was in Bavaria for his wedding to Queen Amalia and the king discovered that his physician had been sending dispatches to the king's father describing young Otto as deranged. In 1837, von Armansperg was dismissed from his post and he returned to the von Armansperg residence of Schloss Egg near Deggendorf in Lower Bavaria. He died in 1853 in Munich. Josef Ludwig had 4 daughters: Countess Louise von Armansperg, wife of Michael Kantakuzenos, Countess Carolina Antonia von Armansperg, married to Prince Demetrius Kantakuzenos, Countess Karolina von Armansperg who married two times, and Countess Maria Catharina von Armansperg who married Baron Julius Bernhard von Eichthal. Although Count Josef Ludwig's branch died out in his 4 daughters, the descendants of his uncle, Count Franz Seraphin Felix von Armansperg still exist nowadays.
