Not sure I follow.  The residence is used to host official state functions and foreign leaders - so you can't get rid of you you can only replace it.

Realize too you're looking at 5+ years from date of go-ahead before its done, so even if they do it, it's the next PM who will benefit