After graduating from college, Swem worked at several libraries in Chicago. Around 1903 he moved to Washington, D.C. and began working at the Library of Congress, becoming chief of the cataloging division of the Copyright Office. In 1907, Swem moved to Richmond, Virginia and became assistant state librarian of Virginia. During his 12 years in Richmond, Swem compiled numerous catalogs of indexes, finding lists and bibliographies in Virginia archives and at the Virginia State Library. He also preserved and increased the state library's collection of books, manuscript materials and historical records. In 1920, Swem moved to Williamsburg, Virginia and accepted a position as librarian at the College of William and Mary, and also helped edit the William and Mary Quarterly. By the time Swem retired in 1944, the William and Mary library collection grew from 25,000 books and 20,000 manuscripts to more than 240,000 books and approximately 400,000 manuscripts. Swem also made the library more accessible to its patrons by offering classes on library use to students and library assistants and opening the stacks to students and the public. In 1936 Swem completed the Virginia Historical Index, also known as "Swem's Index" or simply "Swem," which contains a million entries from seven sources. Reprinted in 1965 and 2003, it continues to be an important resource for genealogists and historians, especially of colonial Virginia. Sixteen members of the Virginia Historical Society and the Rockefeller Foundation helped finance the original effort. After his retirement from William and Mary in 1944, Swem continued to write short articles, as well as edit books and manuscripts on Virginia history and serve as librarian emeritus.
Death and legacy
In their final years, the Swems moved to Louisville, Kentucky to be near their son and their grandchildren. Earl Swem Sr. died aged ninety-four in 1965, two years after he learned the college's new library would be named in his honor and wrote a note for inclusion in the library's cornerstone, but a year before its completion. In recent years, the Earl Gregg Swem Library has celebrated its namesake by hosting events at which friends can be photographed with a life-size portrait of Swem dressed for a walk outdoors. In 2008 it acquired some of the family papers, which it holds in the Special Collections division.