Born in Brechin in 1915, Milne played for local club Brechin Vics, until signing for United in 1934. Remarkably, he scored four goals on his senior debut, which no other player in the club's history has done. Milne continued his prolific rate of scoring, netting 85 goals in 81 appearances for United. No other player in the club's history has achieved a ratio better than one goal per game. Milne had a pay dispute with United in 1937, which led to him going on trial with Liverpool. This trial, however, created a registration mix-up which led to Milne being declared a free agent. Hibernian took advantage of the opportunity to sign Milne for no fee. Milne also scored goals at a good rate for Hibs, finishing as their top scorer in the next two seasons. His career, however, was interrupted by the Second World War, which curtailed competitive football in the United Kingdom. The outbreak of war meant that players were allowed to play for other clubs with their team's permission. It was under these circumstances that in November 1939 Milne returned to play for Dundee United who had joined the Eastern Division of the recently set up emergency league. United were also competing in the hastily set up Scottish War Emergency Cup. Milne showed flashes of the form he had shown in his previous spell at Tannadice, as United reached the cup final; he scored a hat-trick in the second round of the competition in a 7-1 win over Third Lanark and was in the starting line-up in the final which his side narrowly lost to Rangers. Milne was United's top scorer for the season with 24 goals. Shortly afterwards he was called up for military service. During the war he appeared for Aberdeen in unofficial competitions. When league football resumed in 1946, Milne's career was nearing its end. He only played in three league matches, scoring one goal, for Hibs after the war before being sold to St Mirren for £1650. Milne played for the Paisley club for four seasons, before moving to Northern Irelandin the summer of 1950 to become a player/coach of Coleraine. Milne represented Scotland once, in a wartime international against England on 14 October 1944; Milne scored one of the Scottish goals in a 6–2 defeat.
