Brown was born September 8, 1817 in Odessa, Delaware. His grandfather was a Methodist minister. At the age of ten he moved to Wilmington, Delaware where he lived with William Seals, a Quaker. Even as a student, he was frustrated by segregation in education, and moved from a Presbyterian Sunday school to a Roman Catholic school to avoid the segregation. His mother was Methodist, though, so he did not convert to Catholicism. After two years in Wilmington he moved to Philadelphia where his sister lived. He lived with Emerson and Henry Chester, a doctor and a lawyer, where he worked for them in exchange for secular and religious lessons. He attended St. Thomas' Colored Protestant Episcopal Church until January 1836 he joined Bethel AME church. In 1837 began to apprentice to be a barber with Frederick H. Hinton. He also attended an evening school taught by James N. Glouster. He also took time off from his apprenticeship to attend a manual labor school in Amherst, Massachusetts. Among his classmates from Philadelphia were Edward H. Ferris and A. G. Crippen. Shortly later he returned to Philadelphia, but did not remain, instead moving to Poughkeepsie, New York where he attended a school led by Rev. Nathaniel Blount and working as a barber with Uriah Boston. In the summer he worked in New York City. In the fall of 1838 he enrolled at Wesleyan Academy in Wilbraham, Massachusetts to prepare for college. In 1840, he returned to Philadelphia where he studied Latin and Greek. He moved west and in the fall of 1844 he opened a school in Detroit, Michigan, and soon after became acting pastor of an AME church in Detroit, a position he held until 1847. In the fall of 1846 he enrolled at Oberlin College in Ohio.
In September 1846 he became a deacon in the AME church and moved from Detroit to Columbus, Ohio. He also became principal at the Union Seminary, which became Wilberforce University. In August 1852 he moved to Pittsburgh to preach, and later that year or early in 1853 to New Orleans. In New Orleans he helped raise the money for the Morris Brown chapel, but his work was generally opposed by local whites and he was imprisoned at least once for each of the five years he was in that city for allowing slaves to attend his sermons. In 1857 he asked that he be relieved of his position, and Bishop Daniel Payne assigned him to Louisville, Kentucky in April, and then to Bethel church in Baltimore in May 1858. In Baltimore, he also began editing a church periodical. In December 1863 he was asked to superintend the organization of AME churches in Virginia and North Carolina, and in 1864 he became editor of the Christian Recorder. He was also elected corresponding secretary of the Parent Home and Foreign Missionary Society of the AME Church. Along with James F. Sisson, William B. Derrick, and William E. Matthews, he was very important in the work of creating new schools and churches throughout the South for the AME church. In May 1868 he was ordained bishop of the AME church, first serving a district consisting of South Carolina, Georgia, Florida, and Alabama. He organized the Alabama Conference of the AME church in Selma, Alabama on July 25, 1868, and organized the Payne Institute in South Carolina which became Allen University in Columbia, South Carolina. In 1872 he changed district, then becoming bishop of Texas, Louisiana, Arkansas, and Tennessee. In this role he helped establish the Paul Quinn College at Waco, Texas, and organized West Texas, South Arkansas, West Tennessee, and Columbia conferences. He also helped Bishop T. M. D. Ward establish the North Georgia Conference in 1872. In 1876 he again moved districts, then overseeing Baltimore, Virginia, North and South Carolina, serving there until 1880. He then moved to a district consisting of Philadelphia, New Jersey, New York, and New England. In 1884 he moved to a district including Missouri, Kansas, Illinois, Iowa, North Missouri, South Kansas, and California. He organized the Colorado conference in September 1887. From 1888-1892 his district included Indiana, Illinois, Michigan, and Iowa. He held numerous positions in the national AME church as well. He was president of the financial board of the church. He was a leader in the movement of the AME church to send missionaries to Africa and Latin America, particularly in a speech at a general conference of the church in May 1872. He worked with Frederick Douglass and others to push for the enforcement of the Civil Rights Act of 1875, pointing particularly to an 1882 incident when Bishop Payne was removed from a railroad car and forced to walk to the next station while travelling in Florida. Brown was important in the move to include women in the AME ministry, licensing Emily Calkins Stevens to preach in the New Jersey conference in 1883. He was an outspoken and radical opponent of lynchings, especially in the 1890s. He was a delegate to the first World Methodist Ecumenical Conference in London in 1881 and to the second conference in Washington, DC in 1891.
Personal life
On February 13, 1852 he married Mary L. Lewis in Louisville, Kentucky. They had nine children, including John M. Brown and Mary L Brown, medical doctors; William L Brown and Martha L Brown, educators; Daniel L Brown and George A Brown, ministers. Brown died on March 16, 1893 in Washington, DC. His funeral was at Metropolitan AME Church and was presided over by Bishop A. W. Wayman. He was buried at Graceland Cemetery.
