When you log in to an ssh host using a public/private key pair, the server only sees the public key.  So by breaking and entering the attacker can disable your ability to log in in this fashion (by deleting the key) or can enable you to log in somewhere else (by copying the key, probably useless) but they cannot use that key to log in somewhere else absent modification of the SSH server.  And even then I doubt they could get the private key, although they could probably run a MITM attack.