Restricting SSH access to key-based authentication only So I've successfully authenticated myself with my private key, but now I want to disable passwords all together, so I can only access it with my private key.

I edit /etc/ssh/ssh_config and I add the following:

PasswordAuthentication no
PermitRootLogin no
ChallengeResponseAuthentication no
UsePAM no


at the bottom of the file. I then restart SSH with:

sudo launchctl unload /System/Library/LaunchDaemons/ssh.plist 
sudo launchctl load -w /System/Library/LaunchDaemons/ssh.plist


and I try to authenticate with a device that doesn't know my private key at all. I can log in fine just by logging in with my regular user (username@123.123.123.123).

How do I do this? I'm running El Capitan 10.11.1 on my machine.
You need to change your settings in /etc/ssh/sshd_config, not ssh_config. ssh_config is the global configuration file for the local ssh client software, whereas /etc/ssh/sshd_config is the configuration file for the sshd server.