I’ve been playing around with restarting the kernel before every run and it seems to have fixed the majority of the problems! By that I mean the models that were successfully run with batch size 64 earlier are now being successfully run again; and for those that weren’t, decreasing the batch size works in removing the OOM. However, this leads me a final point of inquiry I hope you won’t mind addressing!
All the models successfuly trained with batch size 64 achieved accuracies of ~60% on my dataset, after training for 20 epochs; however, if I run these models again, this time with batch size 32, the testing and training accuracies drop to ~30%, with all other parameters kept the same. I would be happy to run the networks that can’t run with batch size 64 using a lower batch size, but this seems to be having a large effect on my results.
Do you have any idea why this might be happening?