On May 31, 1967, Frey was appointed the eleventh Bishop of Savannah, Georgia, by Pope Paul VI. He received his episcopal consecration on the following August 8 from Archbishop Philip Hannan, with Bishops Charles Pasquale Greco and Robert Emmet Tracy serving as co-consecrators. He selected as his episcopal motto: Serviam. During his tenure, he launched the Social Apostolate, a social service agency designed "to put people in the pews in touch with the poor." He also encouraged every church in the diocese to establish a parish council.
Bishop of Lafayette
On November 7, 1972, Frey was appointed the third Bishop of Lafayette, in his native Louisiana. During his tenure, he initiated reorganization plans that increased and expanded participation by clergy, religious, and laity in diocesan affairs. He also named the first woman to serve as chancellor of a Catholic diocese in the United States. In 1987 he opened a diocesan synod. Frey was the Bishop of Lafayette when the Archdiocese and the Catholic Church faced the first wave of civil suits seeking compensation and treatment for abused children. In a legal deposition, Frey admitted to confronting Father Gilbert Gauthe in 1974. Gauthe was a Catholic priest who later confessed to sexually abusing 37 children entrusted to his care, though he pleaded not guilty to criminal charges by reason of insanity. According to Frey, Gauthe admitted at the time that he was guilty of "imprudent touches" with a boy and promised that it was an isolated instance that would not recur. The following year, 1975, Frey appointed Father Gauthe as chaplain of the diocesan Boy Scouts. Gauthe was ultimately convicted in the first sex-abuse case against the Catholic Church. He registered as a sex-offender and, as of 2013, resides in Texas.
Retirement and death
Upon reaching the mandatory retirement age of 75, Frey resigned as Bishop of Lafayette on May 13, 1989. He was succeeded by his coadjutor bishop, Harry Flynn. Frey retired to a family compound in Bay St. Louis, Mississippi, which was heavily damaged by Hurricane Katrina in 2005. His brother Jerome drove to Bay St. Louis to rescue him, returning him to Louisiana. Frey spent the remainder of his life first at Consolata Nursing Home in New Iberia and later in a private home in Lafayette provided by the diocese. He died after a lengthy illness at age 93. He is buried in the crypt of the Cathedral of Saint John the Evangelist.
