The building, if not the oldest in Didsbury, may have been built at least in part around 1650 'for the use of the minister'. Various people occupied the building, and the Rev W J Kidd may be the best known. He left it complaining of "ghosts and troubles" – this was later found to be rats.

In 1865, Fletcher Moss, then aged 22, moved in with his parents and subsequently bought the house in 1885. He died in 1919 after an active life of public service and was renowned for his in depth writing on local history, flora and fauna and local people.

Alderman Moss bequeathed the house and gardens to the City of Manchester on his death in 1919 because he wanted the house and its contents to remain, as far as possible, intact "to show what a comfortable house of the olden times was like". Unfortunately, the house became difficult to maintain and in 1922 many features, including stained glass and fireplaces were removed.

In due course the house became an art gallery, containing much of the Fletcher Moss Collection, which included several Turner paintings (now in the National Collection). The emphasis of the displays was on items made in or associated with Manchester.

As an 'economic measure', the City of Manchester closed the gallery in 1978. In the fine gardens can be seen the graves of several of Fletcher Moss's dogs, under the yew tree in the shrubbery fronting the house.