Solved by eqy in post #8 
              
              
                Is the OOM happening at the very first batch of the very first epoch or is it happening after a while? 
There shouldn’t be any issues with GPU caching provided there isn’t some sort of unintentional “leak” somewhere due to tensors being kept around when they aren’t used anymore. From the dataloader …