How does Apple Pay on OS X/Safari know to use the iPhone or Apple Watch? When I purchase something using Safari and the Apple Pay option, I either get to pay with my iPhone or Apple Watch.

The selection isn't consistent. 

How does Apple Pay on iOS choose to send the verification (to a given device)?
If you're wearing an authenticated Apple Watch, you'll be asked to verify the payment with your Apple Watch. If you're not wearing it, then this won't be an available option, in which case you'd need to use your iPhone instead.

If this doesn't answer the situation in your experience, then you may need to provide more details/examples of what is happening for you.