Berman was living in Hoboken, New Jersey with two University of Virginia friends, Stephen Malkmus and Bob Nastanovich and recording discordant tapes in their living room. Around this time Berman worked as a security guard at New York's Whitney Museum of American Art, which influenced their music. "We were working at the Whitney with all this conceptual art, and we were learning about it … and so I thought, "Well let’s just make this record that looks like a record, and has song titles and everything, but the songs would be the ones we make at home that sound terrible." said Berman. Although Silver Jews is often called a Pavementside project, it was formed at about the same time as Pavement. Walnut Falcons was tossed around as a potential band name before settling on Silver Jews. Berman has misrepresented the origin of the band's name as coming from the transcription of a billboard that he saw reading "Silver Jewelry", but he has stated in an interview that the name was actually intended to be a pseudonym for a conceptual art piece and later evolved into the actual band. Others claim, however, that the name pays homage to Silver Apples, The Silver Beatles, and slang for blonde-haired Jews. In an interview with Vish Khanna on the June 12, 2019 episode of Kreative Kontrol, Berman stated, "Silver Jews was a very burdensome band name. It confused people: I never made it clear what I meant by it. It wasn’t until the end until I realized what a proper definition of Silver Jew would be outside of the context of my music, which would be a Jew that was Jewish down through patrilineal descent, which would be a Jew that’s not a Jew, really. It’s the outsiders to the outsiders."
Touring
Though Berman had pointedly avoided playing live throughout much of the band's existence, then Silver Jews surprised fans by embarking on their first-ever tour following the release of Tanglewood Numbers in 2005. They toured North America, Europe, and Israel in 2006. A documentary, titled Silver Jew, was filmed during the band's time in Israel.
Break up
On January 22, 2009, Berman announced via the official Drag Citymessageboard that he planned to retire from music and that the Silver Jews would play their final show on Bluegrass Underground at Cumberland Caverns in McMinnville, Tennessee at 3 pm on January 31, 2009. The caverns are located 333 feet underground, and only 300 general admission tickets were made available. Berman stated that he would play his 15 favorite Silver Jews songs. He also wrote that his intentions are to move to "screenwriting or muckraking". He closed the entry by saying, "I always said we would stop before we got bad. If I continue to record I might accidentally write the answer song to 'Shiny Happy People'." Berman originally planned for the concert to be recorded and aired sometime in February 2009 on Nashville's famed WSM AMradio station; however, the day before the show, Berman posted on the Drag City message board to say that he had "negotiated out of the WSM part out of the deal." The concert was documented by several 16 mm film cameras, and Berman has hinted at a possible DVD release of the show at some point in the future. The last song that the Silver Jews performed live was "Smith and Jones Forever".
Post-breakup releases
On June 19, 2012, Early Times was released by Drag City. Early Times is a compilation of EPs and singles that were released before the debut LP Starlite Walker.
Post Silver Jews projects
Brian Kotzur is the frontman/drummer for Tim Chad and Sherry and also plays drums for Super Bowl Rocket Ship. William Tyler plays guitar for Tim Chad and Sherry.
Possible reunion
In 2015, Bob Nastanovich posted a photo to his Facebook page featuring the band rehearsing. He said that the band is working on two new songs, "The Veranda Over The Toy Shoppe", and "Wacky Package Eyes". David Berman later said the photo was a prank. Berman instead returned with new project Purple Mountains in July 2019, but died the following month.