Keep the current audio output device, when attaching a new one I have my TV connected to my iMac via HDMI and speakers hooked up to the iMac via the headphone port.

Whenever I turn the TV on, the Mac starts playing audio through the TV instead of the speakers and I have to manually switch back.

Is there any way to make my Mac keep playing through the headphonejack?
I use a free app from Rogue Amoeba called SoundSource that lets you quickly pick the source for input, output and system sounds from the task bar. One of the side effects of running SoundSource is your source doesn't change when you connect something new to your Mac. You have to select the new source explicitly via SoundSource to have audio routed to or from it.

The source will switch automatically if you remove a source and that source happened to be the currently selected source for any of the channels.