The palace was initially constructed in 1520–1522, by the bishop of Bitetto, Lorenzo Serristori. The Serristori family is originally from the small Tuscan town of Figline, and the last name derives from Ser Ristoro, a local magistrate who is commemorated by a statue in thia palace's entrance stairwell. The Serristori coat of arms of the lineage above the portal facing San Niccolò and was modified by Pope Leo X, who added three lilies to the three existing stars. The original plan of the palace had a classic U shape of clear Roman derivation, with a quadrangular inside courtyard that looked over an important garden through a loggia with three arched windows. Historians suppose, by certain analogies, that the architect was Giuliano da San Gallo, active at that time in Florence. Lorenzo's nephew Averardo, a diplomat for Cosimo I de' Medici, enlarged the palace and gardens facing the Arno. During the 16th-century, the palace is further enlarged under the patronage of Antonio Serristori, governor of the Livorno port. He enlarged and transformed the "casa dell'orto" in a palace endowed with an elaborately decorated ballroom. This transformation was designed by the architect Gherardo Silvani, who was well-paid according to documents. The architect Felice Gamberai helped complete the ballroom which is flanked by three large windows surmounted by other smaller windows, and housing Murano glass chandeliers. The vaults and walls are decorated by Pier Paolo Lippi, Agnolo Gori, and Cosimo Ulivelli. In 1803, the architect Giuseppe Manetti was commissioned by then Senator Averardo Serristori to refurbish the palace. The size of the palace and gardens was reduced in size during the enlargement of the Lungarno along the river. The architect Mariano Falciani created a new river facade. In 1822 housed briefly the Count Demidoff, who then moved to the nearby Palazzo Amici-Demidoff. Joseph Bonaparte, the brother of Napoleon, lived here in exile till his death in 1844. The palace returned in the 19th century to the property of the Serristori family. The interior has a helical staircase by Gherardo Silvani.
