Hi,
The training/validation engine code I’m using is here 2, I have editited it slightly to include some torch.cuda.empty_cache() statements but it is functionally the same code.
All of the images are of identical size from the same camera, there are differences in the number of objects per image but not by a huge amount, also which images are train and are test are chosen randomly each time at runtime so I don’t believe it is one sample causing issues. Also the test and train dataloaders are now both using shuffle=True and the OOM always occurs on the first time evaluation is called.