The Romans were amongst the first recorded visitors to what is Yaxley Hall. They left behind a mound, close to the A140 Roman Road, which is now a scheduled ancient monument within the grounds. The name "Yaxley" means Cuckoo Woods in Old English, so we can be sure the Anglo-Saxons were later here too. The first mention of a building on the site is in the Domesday Book. For a long time there was a priory on the site, although this was replaced by a Hall before the Reformation. The current building dates back to the 1500s, although it has been built on to, developed and changed over the years since then.

For some 300 years Yaxley Hall was in the hands of just two families. The first of these were the Yaxleys themselves, who were a Catholic family aligned with other notable Norfolk Catholics such as the Howards and Bedingfields. The Yaxleys appeared on the list of recusants, but despite being persecuted for their faith, remained true to it. There are rumours of a priest hole at the Hall, although none has been found in modern times.

The Hall was acquired by the Fourth Earl of Scarsdale in the 18th century for his mistress and their illegitimate children. It therefore went at one moment from being a pious Catholic household to what would by some have been considered a house of ill repute! Miss Henrietta Nelson, the reputed ghost of Yaxley Hall, was a member of this family.

During the 18th century the owners carried out a particularly unusual and interesting set of alterations in the "Gothick" style (also known as the Strawberry Hill style of architecture). This fanciful and playful style of architecture was considered quite cutting edge at the time, but few examples of it have survived anywhere in the country. The remodelling led to the addition of the castellations on the front façade and the insertion of a splendid and intricate Gothick window.

The Hall remained in the hands of the Earl's descendants up until the 20th century, when Lord Henniker acquired the property and added it to his substantial Suffolk land holdings. It was remodelled by Lord Henniker to serve as a base for the typical Edwardian country weekend for guests and shooting parties. The fanciful Gothick window was removed in 1905 as part of these works. Disaster struck in 1922 when two of the five wings of the Hall were destroyed in a terrible fire which made the national press. A wayward Italian butler was blamed, but the exact circumstances remain unclear.

The Hall, now reduced in size, passed into the hands of Blanche Broadwood (of the piano family), who lived here for forty years. The modernist architect Sir Basil Spence (of Coventry Cathedral fame) also lived here for a brief period in the 1970s and is buried nearby in Thornham Magna. A gazebo, called the "Teahouse of the August Moon", is one of his last works and stands in the kitchen garden.

The current owner, Dominic Richards, acquired the Hall after viewing it in the winter of 2001. He immediately began his sensitive redesign and refurbishment of the building, which has brought luxury and contemporary style to Yaxley Hall, whilst fully respecting the history of the ancient building. In 2006, Dominic reinstated the lost Gothick window. Hand made by local craftsman Simon Pyke, and glazed in leaded panels by Devlin Plummer, the return of a Gothick window to Yaxley marks an important milestone in returning the Hall to its former glory. It was unveiled by the chairman of Suffolk County Council, Charles Michell, after a lunch held at the Hall.

In 2006 Yaxley Hall was described by The Daily Telegraph as "prepared for another Golden Age". Dominic's on-going plans include the restoration of the entire two lost wings of the Hall at some point in the future.