I ended up changing my architecture specifically to work around this issue.
For inference mode, I simply use volatile=True and this solves everything.
Otherwise, you can move your model using .cpu() and .cuda() calls.
E.g. if seq is a class module, then you can do:
seq.cpu()
output = seq(input)
seq.cuda()

Furthermore, you can call pin_memory() on the input tensors to facilitate transfers back and forth.
It’s very hacky, and in the end, I found an alternate way of training my model.