Fontaine joined the United States Navy and served in the Mexican–American War of 1846–1848, including the Siege of Veracruz. Fontaine first worked as a surveyor for the Yazoo and Mississippi Valley Railroad. He explored the Amazon River with William Lewis Herndon as well as Israel, where he helped draw maps. He drew maps of Russia and sketches of the Great Wall of China. He traveled to Japan with Matthew C. Perry, and he "was instrumental in ridding the seas of the Malay pirates." His autobiography states he participated in the Crimean War. Fontaine joined the Confederate States Army during the American Civil War of 1861–1865. He served under generals Stonewall Jackson and Robert E. Lee. He became a Major on July 28, 1863. He took part in the Battle of Mine Run and the Battle of Bloody Angle. He was a dispatch bearer at the Siege of Vicksburg. His autobiography states that he was responsible for sinking the famed cannon "Whistling Dick" in the Mississippi River to prevent it falling into Union hands. He said that he was wounded 67 times during the war. In an 1892 letter, he said that he had been with the 2nd Virginia Cavalry, Company I. An article in America's Civil War says that he was a spy with the Mississippi infantry. During the Reconstruction era, he joined the Ku Klux Klan chapter in Hinds County and Madison County. An essay he wrote about the Klan was also published in The Ku Klux Klan or Invisible Empire by Mrs S. E. F. Rose. He is quoted as writing, "In the courts of this invisible, silent, and mighty empire, there were no hung juries, no laws delayed, no reversals, on senseless technicalities by any Supreme Court, because from its Court there was no appeal, and punishment was sure and swift, because there was no executive to pardon." Fontaine authored several books. He composed poems, like Oenore, Only a Soldier or Dying Prisoner in Camp Chase, and claimed to have written All Quiet Along the Potomac Tonight. He published his memoir, My Life and My Lectures. Another book was about the Ku Klux Klan: The Cause and the Effect of the Ku Klux Klan in the South.
Fontaine married Miss Brickell; they had three sons and three daughters. They resided in Lyon, Mississippi. Fontaine died on October 1, 1921 in Lyon, Mississippi, at 92.
