Yes, if you have a computer with a private key that is available to the internet, and it gets knocked over, you're dead.  Just as you would be if he knocked it over and installed a keylogger and you were using passwords.  Cracked is cracked.<p>Assuming you are <i>not</i> running services on your main workstation, which is not that unreasonable (Ubuntu workstations are installed with 0 services available to the local network; my Mac has 0 services available by default), then you can get knocked over but it will probably happen because of an browser bug or something like that, i.e. not an active attack.<p>The whole point of keys is that knocking over one of the intermediate nodes in the network <i>no longer gives you control over everything</i>.  Just because you rooted my server doesn't mean you can automatically log in to all my other machines, even if I'm using keys.  Keys are better than passwords.