The main building of the George in the 18th century appears to have been the house which now stands end-on to the street on the eastern side of the present block and is of at least 17th century origin. At that time the site of the western house and the courtyard next to the Crown was occupied by other buildings set back from the road. The stables of the George stood at the back of the building, apparently reached through an alleyway on the east side.

The two inns were separated and the present western house and courtyard built in 1832, the courtyard presumably being provided for better access to the stables. The gateway at that time however was set further back, on the site of the present steps up to the hall. The alleyway on the east side (now a garden footpath) was retained for pedestrian access to the rear. On the west face of the building overlooking the courtyard is a datestone marked ‘REBUILT 1832’ set around an older stone, its armorial lettering almost completely weathered away, which was evidently taken from part of the earlier building. The inscription, still legible in the 1930s, read “A.B.S. 1681,” the central “B” standing above the “A” and “S.” This almost certainly refers to Abel and Sarah Broad whose marriage is recorded in the parish registers in 1671. The new house was presumably built by a ‘John Smith’, who acquired the property in 1830.  However, in 1878 the inn was purchased for the parish by the owners of the Cloth mills, Isaac and William Carr, to provide premises for the Twerton Club and Institute. A library and reading rooms (previously located near the lower mill) was provided in the western wing, accessed through a door at the front facing Mill Lane. A clock was later installed above the front windows. The present hall at the rear was also built, presumably on the site of the stables, and the ornamental iron gates and spur stones installed at the front. The eastern wing, which was separated off as a private dwelling, was later converted to a small shop in the early 1900s. The Institute continued in use until about 1959 when the hall was given over for the formation of the St.Michael’s and All Angels’ Youth Club and the west wing for a private dwelling. In 1965 the shop in the east wing was also closed and converted back to a dwelling.

Next to the alleyway on the east side of the building was a walled garden which is clearly shown on the early maps in line with the frontage of the George. In the mid 19th century however it encroached forward, in line with the White Hart Inn on the east side, and for this reason there was insufficient room for a foot-walk between the two inns. It was not until the 1970s, when the present house was erected on the site, that the wall was removed and the boundary, now marked by a beech hedge, set back to its original alignment, providing room for a foot-walk.