The priory was founded in 1160 by Gervase Paganel, Lord of Dudley, in memory of his father. It was established as a dependency of the Cluniac Priory of Much Wenlock and was dedicated to Saint James. The priory was built from local limestone, quarried from Wren's Nest. The first known prior, mentioned in Gervase Paganell's charter, was named Osbert. A bull was issued by Pope Lucius III and the possessions included the Church of St. Edmund, Church of St. Thomas and the churches of Sedgley and Northfield with the chapel of Cofton Hackett. In obedience to a papal mandate in 1238, the bishop of Worcester and the bishop of Coventry and Lichfield came to an agreement as to the bounds of their respective dioceses whereby it was decreed that the town of Dudley, with its churches and appurtenances, should belong to Worcester, while the castle and priory should remain under the jurisdiction of the bishop of Coventry and Lichfield. The priory was dissolved in 1395, but refounded as a denizen priory soon after. The priory was enlarged after being founded. A surviving piece of one of these enlargements is an archway to the Lady chapel area of the Priory, built in the 14th century, dedicated to the Virgin Mary. It is believed that the Lady chapel was constructed by the Sutton family. The priory was closed by King Henry VIII in the 1530s as part of the nationwide Dissolution of the Monasteries. Following the dissolution, the priory was granted to Sir John Dudley, 1st Duke of Northumberland in 1540. Following his execution, the buildings fell into disrepair and became ruinous. In the 18th century, part of the ruins of the church had been used by a tanner, a thread manufacturer, and for grinding glass and polishing steel. The surrounding area became industrialised. The nearby pools were drained and in 1825, Priory Hall was constructed. In 1926, Dudley County Borough council purchased Dudley Priory and the surrounding rural land as part of plans to develop a new residential area. Dudley Priory was incorporated into the new Priory Park, to serve the new Priory Estate that was developed to rehouse people from town centre slums. In 1939, archaeologist Rayleigh Radford put stone lines in the grass to mark out the walls of rooms and cloisters in the priory. Rayleigh Radford also excavated medieval tiles which are now exposed at the surface.
The Cluniac Prayer
"O God, by whose grace thy servants, the Holy Abbots of Cluny, enkindled with the fire of thy love, became burning and shining lights in thy Church: Grant that we also may be aflame with the spirit of love and discipline, and may ever walk before thee as children of light; through Jesus Christ our Lord, who with thee, in the unity of the Holy Spirit, liveth and reigneth, one God, now and for ever."