Using security keys similar to the use of TouchID for removing need of passwords With the new MacBooks getting TouchIDs, password authentication is becoming an old school way to do things. Since TouchIDs work in the same way as Security keys (both having public-private keypair), and As written here, we can use touchID without passwords as long as we're logged in: Is there a way to use security keys for the same purpose in older Macbooks as the touchIDs in new Macs?
It is not currently possible to get the system dialogs for TouchID to also work with a security key, such as for example the popular Yubikey product. Third party applications can however provide support for products such as the Yubikey through similar popups windows.

One method of using security keys in a manner that resembles TouchID is to set your system up to them as a kind of "virtual smart card". This allows you to use for example a Yubikey as a means of logging in to your Mac, and for authenticating with Keychain, etc.

The specific method of achieving that depends on the type of security key you have. In general you need to configure the system and/or install drivers to let the security key function as a "smart card".

For the Yubikey product you can follow this guide from the manufacturer:

https://support.yubico.com/support/solutions/articles/15000006468-using-your-yubikey-as-a-smart-card-in-macos