Don’t think you missed anything. Has given me a way forward to solving my problem. Thanks alot for the help.
So the reassignment of inputs acts like a pointer to a device memory space containing the tensors?
model.to(‘cuda:0’) will override explicit statements within the model itself?
Also the reason I am passing my labels to device 1 is because in theory (if I had reassigned) the logits should be output on device 1. I did this because I thought it would reduce memory use on device 0 allowing more room for the optimizer.
The model is for genomic sequence data and can be found at https://www.biorxiv.org/content/10.1101/694851v1 1. if your interested.
E: I think I pinpointed the error to when the optimizer attempts to step over the embedding layer it exceeds memory because of the size of my vocabulary. When I move it to device 1 I get a OOM on device 1 while the optimizer params stay on device 0. If I move the embedding to cpu. The model trains.