The traditional owners of Broadford are the Taungurong people, a part of the Kulin nation that inhabited a large portion of central Victoria including Port Phillip Bay and its surrounds. The first European occupier of the townsite of Broadford was Lt. Col. Henry White who was reported as being "of Mount Piper" in October 1838. Mount Piper is 4 km west of Broadford. White was the first recorded land holder in the vicinity. The first survey map of the region in 1842 shows Colonel White’s Station on Sunday Creek only 4 km downstream of its junction with Dry Creek. That junction became the future townsite of Broadford. This map also shows Gideon Stewart’s sheep station exactly on the future Broadford townsite, 9 chains East of the junction of Dry Creek with Sunday Creek. Stewart arrived from Hobart and took out a licence for a sheep station from 1 July 1840. Stewart was an associate of the well known Hobart publican Reay Clarke. Clarke left Hobart and set up an inn at the location of Stewart’s sheep station in 1843. The location was referred to as "Clarke’s Ford" in October, 1843. Clarke was granted a General Licence for the "Broadfoot" Inn, Sunday Creek on 27 April 1844. By April 1845 it was recorded as "Broadford" Inn. An area traditionally connected with Broadford has been Sugarloaf Creek, Victoria. The Sugarloaf Creek Station near Broadford was the first inland settlement in Victoria when it was set up by Charles Ebden and Charles Bonney on about 14 March 1837. Gold was discovered in 1858 in the nearby Reedy Creek, and later at Strath Creek and Sunday Creek but was not long-lived. The Post Office opened on 1 July 1852. The North East railway line was built through the town in 1872. The Broadford Magistrates' Court ceased operation in July 1980 and was formally closed in November 1981. The former courthouse was subsequently used by a local religious group.
The Town today
Industry in Broadford includes a paper mill. It has a primary school, a secondary college and two kindergartens. Broadford is also home to the Victorian State Motorcycle Complex, which has both road and off-road tracks, and hosts National riding events, as well as amateur track days. In 2014 Calibre Sports Inc. leased some land on the complex and in 2015 the complex hosted Tough Mudder Australia event. Popular sports in Broadford include Australian rules football and the town has a team, the Kangaroos, competing in the Riddell District Football League. The Broadford juniors play in the Seymour Junior Football Netball League. Lawn bowls is also enjoyed with the club located next door to the Lions Park on the old caravan park grounds. Broadford cricket club has four seniors teams playing across A, B and C grade, the club also has three junior teams u12, u14 and u16. Golfers play at the Broadford Golf Club on Davidson Street.
On the night of 9 January 2008 firefighters were called to a blaze at the Broadford RSL, but there was little crews could do to save the weatherboard hall. The Country Fire Authority's Scott Hamilton says the blaze caused about $300,000 damage to the building and destroyed its contents.