In 1857 Lord Taunton engaged Henry Clutton to construct a house for the estate, as a summer residence and to house Lady Taunton's extensive art collection. This required the demolition of the hamlet of Aley Green, and was built in stages in the 1860s, and ultimately came to comprise the main house, stable-block, gatehouse and lodges. The main house has been described, perhaps unkindly, by Sir Nikolaus Pevsner as "a large rather dull Tudor house... Gothic Stables, a specially crazy Gothic Dovecote and a big Gothic Lodge on the Aisholt Road"; it comprised a library, great hall, billiards room and several other major rooms. The Cockercombe tuff necessary for construction was quarried on site. In 1869 on the death of Lord Taunton the house and estate passed to his eldest daughter, the Hon. Mary Labouchere, who married Edward Stanley in 1872. Edward Stanley became an MP for West Somerset at a by-election in 1882, and represented the seat, and its successor, as a Conservative until 1906, making three contributions in parliamentary debates, typically on local issues. Edward Stanley had two sons, Henry Thomas Stanley and Edward Arthur Vesey Stanley. Henry was the elder and expected to inherit the estate. Henry was a keen cricketer, representing Somerset County Cricket Club in fifty first-class matches between 1894 and 1899. He had joined the West Somerset Yeomanry as an officer, but died in 1900 during the Second Boer War. As a memorial, Edward Stanley planted trees in the house's grounds in the formation of a cricketing eleven. After Henry's death, Edward Arthur Vesey Stanley, for whom the Quantock Staghounds had been founded in 1902, became heir and inherited the 8,000 acre estate after his father's death in 1907. In 1919 the estate, house and contents were sold at auction over eleven days to clear Edward's debts.
The Sanatorium
At the 1919 auction, the estate was divided, but the main house was bought by Somerset County Council under the Public Health Act 1913 to become the Quantock Sanatorium, along with 2,045 acres. The sanatorium lasted until 1961, when the rise of antibiotics rendered such treatment ineffectual.
The School
In the 1962 the house was purchased by David T. Peaster and made into a school in 1964, which he initially headed alongside Cotham School, Bristol, before the latter closed in 1966. The school preserved various traditions of the house, preserving Lord Taunton's arms as the school's crest, and adopting the phrase passibus citis sed novis as their motto. The school was run on traditional lines, and proved especially popular with diplomats and armed services personnel, not least since it was in the exclusive list of schools recommended by the Ministry of Defence. Within a few years of opening, it was described by the Gordonstoun Record as 'the Gordonstoun of the West' and the Daily Express termed it 'Britain's newest Public School'. The estate was gradually developed for educational purposes in this period, with the Patio Block and Sports Hall Block being constructed in the 1970s, and the Stable Block in the following decade. Similarly, to sustain the school's co-curricular activities, an all-weather pitch, tennis court and new heated swimming pool complex were built in the 1980s. In 1986 Quantock School became co-educational, but soon after the turn of the 1990s was in gradual decline, due in the main to the end of the Cold War and the subsequent closure of a number of overseas service bases, which in turn led to a drying up of new pupils. The hand-over of Hong Kong to China in 1997 also had a serious impact on the school population. In 1998, Quantock School closed.
Present Day
Following the death of David Peaster in 2000, the school reverted to its former name of Quantock Lodge, and was remarketed by Peaster's widow Jane as a centre for recreation and banqueting. It is also a youth summer camp centre.