@Jag1990
The building has only been the residence of the PM since the 1950s - 60 or so years.  There is no glorious history unlike 10 Downing or the White House.  The address - 24 Sussex - is associated in the minds of many Canadians as where the PM lives.  I doubt 1 in 10 (before this article anyway) could identify it in a picture.  The building itself holds no special value.  Tear it down.  Build a newer, (but please not in modernist/brutalist style) mansion that will be good as a residence and to host small functions for the next 100 years.  Keep the location and the address, that is what Canadians know.  Build something worthy of the location and its use.