I assume “overloading” means your GPU is running our of memory?
If so, you could try to wrap this special forward pass in a with torch.no_grad() block so avoid storing the intermediate activations (which are not needed, if you don’t plan on calling backward).
If that doesn’t help and you are still running out of memory, you could of course chunk the data in smaller batches, which should still be faster than passing each sample one by one.



 spacemeerkat:

o temporarily map the network to the CPU without saving and reloading the model


You don’t have to save and reload the model in order to push it to the CPU.
Just call model.to('cpu') to move all parameters to the CPU.