The house is recorded at No. 3, Minamiyamate. Nagasaki opened to foreign trade under the treaty of 1858, and settlements were laid out at Higashiyamate and Minamiyamate on the slopes above the harbour. The ground was divided by numbered lots rather than by street names, so buildings here are identified by their lot number — the former Glover House was built at No. 3, Minamiyamate, in 1863, and that is still how it is referred to.

Glover Garden opened in September 1974 and gathers nine historic buildings within its grounds. The former Glover House was inscribed as World Heritage in 2015. The surrounding streets are stone-paved slopes and flights of steps, with the harbour below, so almost every view here includes water at an angle. Trams handle movement across the city. This remains a residential district of steep lanes and stairways, which means a good deal of it is still reached on foot because no car can get there.

A treaty settlement is a piece of ground laid out for people arriving from outside. Numbering the lots instead of naming the streets, and cutting the slope into flights of steps, were both procedures for building a district quickly and efficiently, and neither took account of what would be convenient for the people who ended up living in it. A hundred and fifty years later, those two decisions are what the neighbourhood is known for and what people climb the hill to see. The inconvenience became the character, which is the usual fate of an efficient plan once the efficiency it served has stopped mattering to anyone.