Olson formed the Jayhawks in 1985 with singer and guitarist Gary Louris and was originally the principal singer-songwriter in the group. Along with Marc Perlman and Thad Spencer, they released their eponymous debut album in 1986 through the independent label Bunkhouse Records. The band were signed by Minneapolis label Twin/Tone in 1987 and released Blue Earth, which saw Louris sharing more songwriting and singing duties with Olson. It was this album that led the band to be signed by major labelDef American; A&R representative and producer George Drakoulias is said to have heard Blue Earth playing in the background while on a phone call to Twin Tone in 1991 and signed the band later that year. Their first album for Def American was the Drakoulias-produced Hollywood Town Hall in 1992. After a successful single, "Waiting for the Sun", and extensive touring the band went back in the studio and released the follow-up, Tomorrow the Green Grass in 1995, which yielded the radio hit "Blue". The same year Olson quit the Jayhawks to look after his wife, Victoria Williams, after she was diagnosed with multiple sclerosis, and the band continued without him, releasing three more albums before going on hiatus in 2005. For his post-Jayhawks career, Olson returned to his folk and country roots and with Williams and multi-instrumentalist Mike "Razz" Russell released the well-received album The Original Harmony Ridge Creek Dippers in 1997. The trio teamed up again for the 1998 album Pacific Coast Rambler and 1999's Zola and the Tulip Tree. By 2000, Olson releasing the autobiographical My Own Jo Ellen as Mark Olson and the Original Harmony Ridge Creekdippers. Two more albums recorded with the Creekdippers followed, December's Child in 2002 and Mystic Theatre in 2004. The same year saw the release of another Creekdippers album, Political Manifest. After splitting from Williams in 2005, Olson paired up once again with Louris for two short tours in the winter of 2005 and the spring of 2006. He released his acclaimed solo album The Salvation Blues in June 2007. The album, written while he was staying with Cardiff-based folk singer and writer Charlotte Greig and her husband, novelist John Williams, was inspired by his divorce. After the release of The Salvation Blues, Olson toured the United States and Europe with Italian violinist Michele Gazich and Norwegian multi-instrumentalist Ingunn Ringvold playing djembe, percussion and piano and singing harmony vocals. They played more than 300 shows together. An album with Louris, Ready for the Flood, produced by Chris Robinson of the Black Crowes, was released in November 2008. In 2011 the Jayhawks reunited and recorded a new album. The lineup consisted of Olson, Louris, Perlman, Karen Grotberg, and Tim O'Reagan, and, as Louris says, "Our goal is to make the best Jayhawks album that's ever been done". The album, Mockingbird Time, was released September 20. The band toured in support. In 2012 Olson again left the band.
