A Yubikey-like device is actually more secure than what the parent comment describes, because it prevents the private key from being viewed or copied by the host computer. You can think of it as a dedicated piece of hardware that runs something like ssh-agent, except over USB instead of a socket.