I used the code CRNN and add horovod  to do distributed training.
when the dataset images about 1000w, the batchsize is 128, and used 6 V100, everything is ok.
But when I add dataset, and the images about 1900w, the batchszie is also 128, this error happens when the network forwards the conv layer.
I noticed somebody said decrease the batchsize, so I changed the batchsize 128 -> 64, and the error is cuda OOM. So I continue to reduce bathsize to 32, OOM error also exists.
I’m confused. Can someone help me? Thanks