Smth resigned from the army to enter politics and was elected a Democratic delegate to the Confederate Congress until hostilities ceased in 1865. He established a very successful law partnership in Columbus, Georgia, and was elected to the Georgia Legislature in 1870 as an outspoken opponent of Radical Reconstruction. The following year, he became Speaker of the Georgia House of Representatives. Running unopposed, Smith was elected Governor in 1872, succeeding acting Governor Benjamin F. Conley. To many, Smith's inauguration on January 12, 1872, symbolized the end of Reconstruction and the "redemption" of the Democratic Party in Georgia. Smith was reelected in 1874, serving until 1877. During his second term, he was a delegate to the Democratic National Convention of 1876. Major accomplishments included restoring the state's credit rating by voiding fraudulent bonds and reducing overall expenditures, retiring the debt and leaving office with a surplus in the state treasury. He was a supporter of creating a state department of agriculture, and was noted for appointing the most qualified candidates to fill openings in his administration, a contrast to the patronage system that was popular at the time. In an 1876 interview with the Atlanta Journal Constitution, Smith made racist remarks about African-Americans, calling them "idle, thriftless" and "always depending on the whites for everything". The Equal Justice Initiative, a non-profit organization based in Montgomery, Alabama, noted posthumously that African-Americans made up 46 percent of his constituents. Smith was defeated in his bid for the U.S. Senate in 1877. He was named the first chairman of the new Georgia Railroad Commission, serving a 6-year term. Returning to his legal career, his former Civil War commander, John B. Gordon appointed him as Judge of the Chattahoochee Circuit of the Superior Court from 1888 until 1890.
Smith was married twice, first to Sally Marshall Welborn, then after her death to Hester Ann R. Brown. Smith died on November 25, 1890. He was buried in Alta Vista Cemetery in Gainesville, Georgia. The Atlanta Constitution eulogized James Milton Smith as "one of the boldest and most fearless men in the history of Georgia." The James M. Smith Science Hall at the State Normal School in Athens was named for him.
