Withers was born in California's San Fernando Valley. He had seven siblings and his father was a mechanical engineer. Neither of his parents were practicing Catholics during his childhood. His father had been raised Catholic but then "lapsed". Instead, he was raised culturally Jewish according to the faith of his mother. When Withers was eight, his family moved to Cherry Hill, New Jersey. In 1974, he had an encounter with a group of Catholic activists at the bicycle repair shop where he worked, which caused him to convert to Catholicism and be baptized. The same year, Withers left home for nearby Camden, New Jersey, to live in and repair abandoned row houses.
Religious life
After his baptism, Withers lived in a loosely affiliated religious community. While there, Withers almost got married but decided that he wanted to live a religious life instead. In 1984, he took private vows of "poverty, chastity, and obedience" and became a hermit. Before taking his vows, Withers had looked into several religious orders, but he could not find one that he felt compatible with. Passionate about the spirituality of the Desert Fathers, he and Sister Margaret McKenna debated about "Where is the desert today?" Deciding it was in the abandoned inner cities, in 1989 they moved into an abandoned row house that they began restoring, continuing their work even when their tools were regularly stolen by drug addicts. There, they founded New Jerusalem Laura, a treatment center for drug addicts. Two years later, in 1991, Withers bought a derelict rowhouse from the city of Philadelphia for $1 and parted ways with McKenna. This would become his hermitage. He fixed the building, which lacked doors, windows, or a working roof and then went on to build furniture for it by himself. After the 1983 revision of Canon 603, the option of being a consecrated hermit while independent of the system of religious orders had been permitted. In 1995, Withers attempted to be consecrated by the Archdiocese of Philadelphia but was rejected. In 2001, Withers was finally approved and consecrated as a hermit by Cardinal Anthony Bevilacqua and took his public vows. Withers was the first hermit to be consecrated by the Archdiocese.
Life as a hermit
As a hermit, Withers lives in solitude. He does not own a car or television; he gets his news from the people around him and gets around on a bicycle that he found, broken, on the side of the road. Though he has an open-door policy for people that come to visit him, he himself visits family only twice a year. He does have a computer, which he uses to keep in touch with other hermits via email. To earn his food and clothing allowance, which amounts to less than $5,500 per year, Withers makes pottery to sell and works one day a week for a scientific-instrument company. At the end of each year, he donates any funds that remain to charity. Withers wakes up at 5 AM and then fills his day with prayer and chores, following his rule of life. He prays for 4½ hours each day. Withers claims that "it's in the solitude that I hear God best".