The Fermors’ house at Tusmore was large, for in 1665 it was assessed as having 19 hearths for the hearth tax. It was built of local stone and had a Roman Catholic chapel as well as ornamental gardens and a fish pond. After 1758 William Fermor had the house demolished except for the chapel and commissioned the Scottish architect Robert Mylne to design a new house. The exterior was completed by 1770 and the interior by 1779. Mylne’s house was of seven bays and built of local stone, most of it from Fritwell. Mylne laid out the gardens and landscaped the park, the latter with a lake and an ornamental Temple of Peace dedicated to the late poet Alexander Pope (1688–1744). Pope had been a friend of the Fermor family and in 1712 had written The Rape of the Lock about an incident in which Robert Petre, 7th Baron Petre had offended Arabella Fermor.
The chapel burned down in 1837. In 1857 the new owner, the 2nd Earl of Effingham, commissioned the Scottish architect William Burn to build an office wing on the site of the chapel. This was completed along with alterations to the house in 1858. After 1929 Vivian Smith had the house restored, most of the Victorian office wing demolished and some other alterations made. Despite the 1858 and 1929 alterations, a number of the principal rooms retained their original 18th-century ceilings in the style of Robert Adam. Smith greatly changed the gardens and park, retaining little of Milne’s landscape design except the Temple of Peace and the lake.
In 1960 Randal Smith, 2nd Baron Bicester had the house demolished, which Sherwood and Pevsner condemned as “a great loss to a county in which important houses of the last quarter of the C18 are few”. The stables were retained and are now a private house. In 1964–65 the architect Claud Phillimore had a new Tusmore House built on the site. It was neo-Georgian but smaller than the house by Mylne. In 1970 the 2nd Baron died in a car accident and was succeeded by his nephew Angus Smith, 3rd Baron Bicester.
In 2000 Wafic Saïd replaced Phillimore’s house with a large new neo-Georgian one designed by Sir William Whitfield (born 1920) of Whitfield Lockwood Architects. In 2004 the Georgian Group gave the completed house its award for the “best new building in the Classical tradition”, although other architectural critics have been more sceptical.