Solved by alex.veuthey in post #2 
              
              
                That is quite impossible, even for datasets with fewer targets/classes (like 1000 for ImageNet). Also, there are a few people advocating for not-too-large batch sizes like 32 and 64 like you said. Not only it fits in the memory of most consumer GPUs now, but it is also better for… reasons. I’ll let …