Unless there is a better way to fix this, I implemented a hacky fix by copying the class member data that I need every time the forward pass is called. This guarantees that the class data will match the device and type of the input.