The son of Thomas Caldwell and Agnes Caldwell, née Smith, he was born at Williamstown, Victoria, on 11 August 1888. He married Clarissa Ann Hayes, the daughter of Richard and Elizabeth Hayes on 7 August 1912. They had two children, Edna May Caldwell, and John Arthur Caldwell. He was also the brother-in-law of former Williamstown footballer "Jack" May, who married Clarissa's sister in 1917.
In his debut season, South Melbourne reached the Grand Final but Caldwell missed playing due to a nine-week suspension for striking Carlton's George Bruce in the Preliminary Final. Almost a decade later, in 1918, as captain-coach, he led the club to victory in the 1918. By then, Caldwell was playing as a rover and had become a regular Victorianinterstate representative.
Footscray
Cleared from South Melbourne on 12 May 1920, he played three matches for Footscray in the VFA — on 15 May, 22 May, and 29 May — before resigning as a player and moving on to coach "Footscray Diggers", in the Victorian Junior Association; and, at least, by 31 July 1920, he was serving as captain-coach of Camperdown in the Corangamite Football Association.
Williamstown
In 1921, Caldwell captain-coached Williamstown to a premiership.
Perth
Cleared from Williamstown on 25 April 1923, he captain-coached the Perth team in 1923 and 1924 "without much success"; and, in the second half of the 1923 season, "was disqualified for the the season for abusing a field-umpire".
On the resignation of Carlton's captain-coach Paddy O'Brien — who went to play with Footscray in its first VFL season — Ray Brew acted as coach for four matches. Caldwell was appointed coached of Carlton for the remainder of the 1925 VFL season.
Rutherglen
Although offered another year at Carlton in 1926, Caldwell took up a more lucrative offer, and was appointed captain-coach of the Rutherglen Football Club in the Ovens and Murray Football League. Caldwell resigned as coach of Rutherglen in July, 1926. Due to financial considerations, the Rutherglen club not only decided not to appoint a playing coach for the 1927 season, but also decided not to appoint a paid coach at all.
South Melbourne
Caldwell was appointed South Melbourne coach for the 1929 VFL season; yet, despite the team's victory against Richmond on 1 June 1929, much dissatisfaction was expressed about Caldwell's coaching,and Caldwell was invited to resign, which he did, on the following Tuesday.
Death
Admitted to the Melbourne Hospital with "internal trouble", Jim Caldwell died of peritonitis some three weeks later, on 20 August 1929.
