Pell "had a passion for agricultural pursuits, and was a familiar figure in agricultural clubs and conventions, as well as an author of many valuable papers on fruit-culture and kindred topics. His information on agricultural questions, and his knowledge of agriculture as a science, were thorough and profound; and inheriting the parental bias for landhold, his possessions at Hyde Park were of vast extent. Besides being the owner of Pelham farm, an estate of 600 acres, beautifully laid out, and situated four miles above Hyde Park, he was the proprietor of another estate, of large acreage, named the Cedars." In 1838, Pell established a 1,200 acre farm just north of Esopus. He was one of the first farmers to grow apples and export them to Europe. Also on his farm, they had apple orchards, 50 acres of vineyards, and ten artificial lakes used for breeding fish. On his estate, he built a brick mansion, in the Romanesque style that was painted "a beautiful straw color and white." In June 1904, the Redemptorist Fathers purchased 235 acres from the Pell's estate and built Mount St. Alphonsus Seminary, which they owned until 2012, when it was sold to a Bruderhof Community for $21.5 million. Pell owned an island known as Pell Island, over a mile in length, in the middle of the Hudson River situated across his estate, and Overlook Mountain, where he had built the Overlook Mountain House, which became a fashionable resort for summer tourists. Pell also owned four million acres of Yellow Pine timber lands in Southern Georgia. Reportedly, during the U.S. Civil War, Pell purchased "seventeen million acres of wild land, for nominal prices, in Georgia, Virginia, Florida, Texas, Arkansas and Missouri; and sold to the Rothschilds in Europe three millions of acres for seven dollars an acre, and they resold the same for fourteen dollars an acre."
Personal life
On July 8, 1837, Pell was married to Maria Louisa Brinckerhoff. Maria was the daughter of James Lefferts Brinckerhoff and Charlotte Brinckerhoff. Her maternal grandparents were Robert Troup and Jannetje Troup. Together, Maria and Robert were the parents of two sons and one daughter, of whom only his daughter survived him:
James Brinckerhoff Pell, an 1863 Columbia University graduate, who died unmarried.
His wife died on November 10, 1866. On February 11, 1880, Pell died at 218 Fifth Avenue, the home of his ninety year old aunt in New York City where he usually spent his winters. After a funeral at Trinity Chapel in New York, he was buried at Green-Wood Cemetery in Brooklyn.
Descendants
Through his daughter Adelia, he was a grandfather to John DeCourcy Ireland, who married Elizabeth Gallatin in 1895; Robert Livingston Ireland, who married heiress Kate Benedict Harvey and was the father of Robert Livingston Ireland Jr.; Maria Louise Ireland, who married Easton Earl Madeira; and Adelia Ireland, who married Dr. Montgomery H. Sicard.