The Oneida people were the original inhabitants of the area. The town was settled around 1799 near Brownville village. It was one of the first towns in the county. Its name is derived from the founder and first settler, Jacob Brown, who was a major-general in the United States Army and is considered to be a heroic figure in the War of 1812. The town was formed in 1802 while still part of Herkimer County. Many of the northern towns of the county were established, or partly established, from parts of Brownville, including Le Ray, Lyme, Pamelia, Orleans, and Alexandria. In 1828, the community of Brownville set itself off from the town by incorporating as a village. In 1855, the community of Dexter set itself off from the town by incorporating as a village. The William Archer House, Gen. Jacob Brown Mansion, Brownville Hotel, St. Paul's Church, Vogt House, and Arthur Walrath House are listed on the National Register of Historic Places.
As of the census of 2000, there were 5,843 people, 2,184 households, and 1,633 families residing in the town. The population density was 98.5 people per square mile. There were 2,857 housing units at an average density of 48.2 per square mile. The racial makeup of the town was 97.84% White, 0.27% Black or African American, 0.55% Native American, 0.27% Asian, 0.02% Pacific Islander, 0.27% from other races, and 0.77% from two or more races. Hispanic or Latino of any race were 0.58% of the population. There were 2,184 households out of which 37.3% had children under the age of 18 living with them, 60.3% were married couples living together, 10.5% had a female householder with no husband present, and 25.2% were non-families. 20.5% of all households were made up of individuals and 9.1% had someone living alone who was 65 years of age or older. The average household size was 2.66 and the average family size was 3.07. In the town, the population was spread out with 27.3% under the age of 18, 6.8% from 18 to 24, 29.2% from 25 to 44, 23.8% from 45 to 64, and 12.8% who were 65 years of age or older. The median age was 38 years. For every 100 females, there were 93.1 males. For every 100 females age 18 and over, there were 91.9 males. The median income for a household in the town was $38,277, and the median income for a family was $43,866. Males had a median income of $33,889 versus $21,904 for females. The per capita income for the town was $16,303. About 6.0% of families and 8.2% of the population were below the poverty line, including 5.5% of those under age 18 and 11.3% of those age 65 or over.
Brownville - A village in the south part of the town on NY-12E by the Black River.
Bull Rock Point - A projection into Lake Ontario at the west end of Pillar Point peninsula.
Dexter - A village in the south part of the town on NY-180 west of Brownville village. It was formerly known as "Fish Island".
Everleigh Point - A projection into Lake Ontario at the west end of Pillar Point peninsula.
Glen Park - A village primarily in the town, on its south border along NY-12E.
Guffin Bay - A bay of Lake Ontario in the northwest part of Brownville.
Limerick - A hamlet at the junction of Routes NY-12E and NY-180 by the Perch River, north of Dexter village.
Natural Bridge - A location on the Perch River, southwest of Limerick.
Perch Lake - A lake in the northeast corner of Brownville.
Perch River - A hamlet on NY-180 in the north part of the town near the river of the same name. The community was once called "Moffatsville".
Perch River State Wildlife Area - A conservation area located in the northeast part of the town by Perch Lake.
Pillar Point - A peninsula into Lake Ontario in the western part of the town, as well as a western projection into the lake on the end of the peninsula.
Pillar Point Village - A hamlet on the south coast of Pillar Point peninsula, located on County Route 59.
Reynolds Corner - A location near the north town line on County Road 125.
