The House was built before Confederation, it is irrelevant that it only became the PM's residence in 1951 as it was already 85 years old then.  Jimbo5 didn't say that the PM is head of state, that bills are signed there or that key meetings with foreign leaders are held there although I'll bet that the latter has happened on occasion.