Olsen's parents are Danish and he used to hold a Danish passport, but grew up in Sweden. He began his career playing for Malmö youth teams before moving to the academy of BK Olympic. At the age of 16 he joined IF Limhamn Bunkeflo and made his debut in the Swedish second tier of a year later. Following a knee injury he went on to play for Bunkeflo FF and IFK Klagshamn.
In 2011, Olsen re-joined Malmö FF. He made his Allsvenskan debut on 1 October 2012 in an away fixture against Syrianska FC, when first choice goalkeeper Johan Dahlin was suspended, and kept a clean sheet as the match ended 2–0 in Malmö's favour. In the match on 5 April 2013 against Åtvidabergs FF Dahlin had to be substituted at half time for an injury and Olsen once again took his place. Olsen then managed to hold a clean sheet in the remaining part of the match as well as a further three matches before conceding a goal in stoppage time in his fourth match of the season in the away fixture against IFK Göteborg. Olsen kept Dahlin out of the starting eleven for some further matches before he was given the place on the bench once again. In total Olsen played 10 matches for Malmö FF during the league title winning 2013 season, an unusual feat for a second choice goalkeeper. Prior to the start of the 2014 season the club sold Johan Dahlin, making Olsen the first choice goalkeeper. In the 2014 season, Olsen made 29 out of 30 league appearances and was a vital piece of the team that defended the clubs league title and qualified for the group stage of the 2014–15 UEFA Champions League. For his performances in the league Olsen was awarded Allsvenskan goalkeeper of the year. He was also nominated for Swedish goalkeeper of the year at Fotbollsgalan.
PAOK
On 1 July 2015, it was announced that Olsen had signed a four-year contract with PAOK FC, with a transfer fee of nearly €650,000 being paid to Malmö.
Copenhagen
On 26 January 2016, it was announced that Olsen would join F.C. Copenhagen on loan from PAOK for six months. According to F.C. Copenhagen's manager Ståle Solbakken, the Danish team was in search for a goalkeeper due to the injury of Stephan Andersen. The performances of 26-year-old international in the Superleague Greece and Europa League until January were below average, but he retained his good form since he joined the Danish club and increased his chances to be included in Sweden national team's squad ahead of Euro 2016. As a consequence of his performance, F.C. Copenhagen wanted to make the loan move permanent. On 24 May 2016, it was announced that Olsen would remain with F.C. Copenhagen signing a four-year contract. F.C. Copenhagen paid a transfer fee of almost €600,000 to PAOK.
Roma
In 2018, Olsen helped Sweden reach their first World Cup quarterfinal since the 1994 FIFA World Cup and gained international interest from some of Europe's top clubs. On 24 July 2018, Olsen signed a five-year contract with the Italian club A.S. Roma with a reported fee of around €12,000,000. Olsen's arrival to the club was announced just five days after the departure of Roma's starting goalkeeper Alisson. He marked his debut for the Italian capital side in 2018–19 Serie A with a clean sheet in the 1-0 away victory against Torino F.C. on 19 August 2018.
Cagliari (loan)
On 30 August 2019, Olsen joined Cagliari Calcio on a season-long loan deal from Roma. On 25 November 2019, Olsen was sent off in an away game against U.S. Lecce for pushing Gianluca Lapadula, who in turn had pushed Olsen for kicking the ball away. Both players were sent off and received a fine of 10,000 Euros. Olsen was also suspended for the next four Serie A matches.
International career
Olsen was eligible for international play for both Denmark and Sweden. When he came to Malmö FF he said that he would prefer to play for Denmark but in 2014 he opted to play for Sweden according to Danish international coach Morten Olsen. On 15 January 2015, he made his debut for Sweden in a friendly fixture against the Ivory Coast. In May 2016, Olsen was called up for Sweden's squad for UEFA Euro 2016, as back-up keeper for Andreas Isaksson. After Sweden's early exit from the tournament, Isaksson retired and Olsen became Sweden's first-choice goalkeeper. In May 2018 he was named in Sweden's 23-man squad for the 2018 FIFA World Cup in Russia. At the 2018 World Cup, Olsen started in all five games for Sweden as they were eliminated by England in the quarter final.
