Elm played 32 matches scoring 8 goals in 28 games in the 2005 season.
Kalmar FF
Elm joined Kalmar FF from Falkenbergs FF before the 2006 season, where his younger brother Rasmus was a player and at the end of the season his older brother David followed from Falkenbergs FF. On 9 April 2007 they became only the second trio of brothers to be fielded at the same time for one team in Allsvenskan, a feat the five Nordahl brothers never managed. Viktor and his brothers played a big part in the second part of the 2007 season which led to Kalmar equalling their best result in Allsvenskan from 1985 at second place and winning their third Swedish Cup, the first time since 1987. In the 2008 season Elm was even more instrumental in Kalmars first league title, scoring 15 goals including four in the last home game to effectively seal the title and winning the award "Swedish Midfielder of the Year". In Kalmars UEFA Cup run Elm scored the winning goal for against Feyenoord in the first leg of the 2008–09 UEFA Cup First round. At the end of the 2008 season Elm's contract was expired and was a free agent after 24 goals in 78 appearances for the Röda bröder.
He signed with Dutch side SC Heerenveen on a bosman during the fall of 2008 and joined the team when the transfer window opened in January 2009. He scored his first two goals for the club against Feyenoord in the Fourth Round of the 2008–09 KNVB Cup. His first Eredivisie goal came on 8 February 2009, in a match against NAC Breda. After the 2012 season Elm was out of a contract at sc Heerenveen and after 102 appearances and 15 goals for De Superfriezen decided it was time for a new challenge.
In June 2012 Elm signed a four-year deal at AZ Alkmaar, teaming up with his brother Rasmus for the first time since their stint at Kalmar FF. He made his official debut in a 2–2 draw against Ajax on 12 August 2012. He scored his first hattrick on 27 September in a KNVB Cup 4–1 win against SC Veendam. On 30 September he scored his first league goal in a 3–3 draw against RKC Waalwijk. He captained AZ for the first time against VVV Venlo on 25 January 2013 because Maarten Martens was injured and Nick Viergever was suspended. Elm scored 5 goals in the KNVB Cup, as AZ won the cup for the 4th time in club history and scored 3 in the League. For the 2013-14 season Elm appeared 49 times out of 58 competitive games, including 11 games en route to the Europa League semi-final. Elm scored 3 goals in the League.
Return to Kalmar FF
On 30 March 2015, it was announced that Elm returned to his former club Kalmar FF, where he reunited with his brothers David and Rasmus.
