This sounds like a bug. It’s a little hard to help without seeing code, but here are a few things you can try:

Trimming down your example. You can try deleting parts of your model until you arrive at a more minimal example that triggers the error.
Figure out what function actually throws that error. In the backend, checking for arguments on the same GPU gets performed. If you figure out which function it’s in, and which arguments aren’t on the same GPU, you might be able to track down where in your code this happens.