Ahh, thanks (and drats).
Following this thread, I had decided to not make use of LSTMCell's, but I guess I need to go back on that decision.
Before I go too far down this track: is it possible or even supported to swap paramters of an LSTM in and out of LSTMCell's on the fly?
Thanks for your time.
Edit: I just realized probably an even simpler solution to this whole problem is to transfer my model back and forth to the CPU using pinned memory. Way better solution than to rearchitect everything.