I thought of the white-haired old lady in the big house, and that vivid wicked little face that had just smiled into ours, and a vague chill of foreboding crept over me. 
I thought of the old lady in the house.
In this situation 'upstairs' refers to the 9th floor and that's where his pad was.