When I was training my model on single GPU(cuda:0), it just worked with batch_size==4. However, I keep the batch_size == 4 and train my model on 4 GPUs, it raise warning : RuntimeWarning: RNN module weights are not part of single contiguous chunk of memory. This means they need to be compacted at every call, possibly greatly increasing memory usage. To compact weights again call flatten_parameters().
and error:RuntimeError: CUDA out of memory. Tried to allocate 98.38 MiB (GPU 1; 15.89 GiB total capacity; 14.56 GiB already allocated; 33.25 MiB free; 150.76 MiB cached
Hope somebody helps me. It’s confusing。