According to tradition, the monastery, dedicated to the Virgin Mary, was founded in 1182 by Margrave Heinrich von Ursin-Ronsberg, to house a community that had grown up around a local hermit. The monastery was first established at the long-abandoned Burg Ursin, the margrave's ancestral castle, where St. Stephen Church's cemetery is now located. A few years later, the monks headed by their first abbot Cuomo, decided to build a new monastery in the valley below where water was more readily available. The original name Ursin or Ursinium was eventually changed to Irsee. The small abbey's community averaged 6 monks during its first century of existence, but this number was reduced to a single monk at one time during the troubled 14th century when Irsee came close to collapse due to poor harvest, famine, war and excessive expenses by pleasure-loving abbots. It was saved only by the intervention in 1373 of Anna von Ellerbach, the second founder, sister of the Bishop of Augsburg, and her appointee, abbot Conrad III, known for his extreme frugality. Prosperity was restored within 20 years and during the late Middle Ages, Irsee Abbey had become one of the major abbeys in the diocese of Augsburg. The abbey was nearly obliterated during the German Peasants' War and again during the Thirty Years' War. It was ravaged no less than five times by Swedish troops and then devastated by Imperial Croat troops and French troops. Its library as well as its archives were destroyed. For many years the monastery was so destitute that it could not accommodate even half a dozen monks. The abbey was finally able to put itself back on a stable footing in the later 17th century.
Imperial Abbey
Irsee recuperated quickly but in 1662 the powerful prince-abbot of Kempten purchased the right of advocacy, which limited the autonomy of the abbey. In 1694, following the election of the energetic Romanus Köpfle as the new abbot in 1692, Irsee succeeded in obtaining the status of an Imperial abbey, which it will keep until it was dissolved in 1802. As an imperial abbey Irsee enjoyed Imperial immediacy. While its small territory covered 118 square kilometersand 22 villages and hamlets, its abbot ruled over 3200 to 4200 subjects. The abbot had seat and voice on the Bench of the Swabian prelates at the Imperial Diet. In case of a declared war, the abbot was required by the Swabian Circle to supply a contingent of one officer and 61 infantrymen. In 1699 the dilapidated church tower collapsed and damaged the choir of the old Romanesque church built in 1194, which prompted Abbot Romanus to undertake the revamping of the church and monastery buildings. The construction project was greatly expanded by its successor, abbot Willibald Grindl and the monastery buildings were completely rebuilt. The plans have been attributed to Magnus Remy, an Irsee monk, who also created many of the paintings in the church. During the long rule of Abbot Bernard Beck, Irsee became a center of intellectual and scientific life in Swabia and beyond. Irsee's natural history collection with its instruments of mathematics and physics were famous. The abbey celebrated the 600th anniversary of its foundation in 1782 but its prosperity soon came to a brutal end in the aftermath of the French Revolution and Napoleonic campaigns. War refugees seek refuge and accommodation in the abbey, which also suffered heavily from military marches, billeting and heavy war contributions. Finally, in the course of the German mediatisation of 1802-1803, Irsee, like all the other Imperial abbeys, lost its independence and was dissolved. On 3 September 1802, soldiers of the Electorate of Bavaria launched a "provisional military occupation" of Irsee and in 1803 the territory of the former abbey, including the villages of Irsee, Romatsried, Eggenthal, Baisweil, Lauchdorf, Ingenried, Schlingen, Ketterschwang, Rieden, Pforzen, Leinau and Mauerstetten, was absorbed into the Electorate, as were its 3221 inhabitants. The monastery's inventory was auctioned off. In 1833, the greater part of the library was moved to Metten Abbey.
Hospital
For many years the Bavarian authorities found no use for the former monastery complex. After 1849, the buildings served as an asylum and hospital for the mentally ill. Between 1939 and 1945 more than 2,000 patients, both adults and children, were transported by the then regime from Irsee and Kaufbeuren to death camps.
Conference centre
In 1972 the hospital was wound up. The local authority of the district of Schwaben began the restoration of the buildings in 1974, which opened as the Schwäbische Tagungs- und Bildungszentrum Kloster Irsee in 1984.