Essential security for Linux servers (2013)
Do not lock down to your IP address. Home IP's change all the time, and in proper security there should be no other way to access the server other than proper ssh authentication or physical access. There is no good reason to be doing it in this context. If SSH turns out to have a massive vulnerability that bypasses keyauth then every service on the net will be torn down.