In the model I am training, I can only submit batch sizes of one through the model during training due to memory constraints. Each of my input tensors are (1, 1, 192-65536). I could normalize to (1, 1, 65536) so that I could do batching, but that means that in my Conv1d(512, 768, 3) layer, that the memory required to house all of those dimensions with a batch size of 128 would be 128 * 65536 * 4 (bytes per float) * 768 (channels) = 24GB of RAM, but my GTX 1080 only has 8GB of RAM.
Bottom line, I can’t push more than one sample per batch. I currently call loss.backward() on each loss for 128 samples which aggregates the gradients. I then call optimizer.step() after all 128 samples have been processed.
I want to include batch normalization. Is there someway to use BatchNorm1d in this scenario, or would it be possible to implement BatchNorm1d with just the gradients produced by calling loss.backward() 128 times?