AirPods connect to correct device I own AirPods and connect to by my iMac (for video conference) and iPhone (for phone calls).

How do I get the AirPods to switch to the device that I want to use?

I can't turn bluetooth off on the iMac because then the mouse stops working. If I select AirPods on the iPhone when I am on a call nothing happens. But if I restart the computer then I can use it on the iPhone while the computer is not on bluetooth.
Under the Listen with your AirPods section of this Apple support document:

Follow these steps to use your AirPods to play audio from a different
device or to switch the audio to different headphones or speakers.
On your iOS device
Open Control Center. Press deeply or touch and hold
the audio card in the upper-right corner of Control Center to control
audio. Tap the airplay icon, then choose the device you want from the list.*
On your Mac
Click the sound icon in the menu bar and choose your AirPods* or other speakers
under Output Device.
*AirPods appear only when they're nearby and ready to use.