On the east side of the old Rectory grounds, running down to the river, was large pasture ground attached to the back of Twerton Farm known as Court Orchard, named perhaps after a manorial courthouse that formerly existed in or near the farm. In 1788 a two acre section on the west side of the orchard was purchased by Rev.Thos.Broughton (then Rector), and added to the Rectory grounds. However it was not until after Oriel College became rectors, in the early 1830s, that Rose Cottage was built on the east of the ground. Initially this building, which consists of two dwellings, served as a residence for churchwardens and parish officers, but since WWI has been mainly occupied by curates and pastoral staff. The low garden wall in front of Rose Cottage was originally topped with iron railings until WWII, when they were removed for the war effort. There were also two matching front gates until recent times when the one on the west side was blocked in, leaving only its Pennant sill still visible at the base of the garden wall. When the access road into Clyde Gardens was driven through the wall of the garden on the west side of Rose Cottage in the early 1970s, a portion next to the house, which included the turnpike milestone and benchmark, was retained.

Formerly there were a number of small cottages that lay behind Rose Cottage. In 1815 a narrow strip along the eastern edge of the Rectory ground was purchased by the owner of Lower Mill to build a long rank of ‘blind-back’ weaver’s cottages known as Rackfield Place which ran down to the river. Access from the High Street to these cottages ran along the eastern edge of the ground, next to Rose Cottage through a narrow walled pathway known as ‘Cuckoo Alley’. When the GWR was built through the middle of these grounds in the late 1830s, most of Rackfield Place was cut off next to the river (as can still be seen today), leaving the cottages mentioned above isolated on the south side of the railway line. They were demolished in the 1980s, having remained unoccupied for many years, and Cuckoo Alley has subsequently been incorporated into the adjoining gardens on the east side of Rose Cottage.