The error messages might point to different allocation failures, since increasing the batch size might also change the point of failure.
E.g. batch_size=4 might fail in the backward pass when the gradient of layer17 is calculated, which would need 38MiB, while batch_size=32 might fail in the forward pass when layer3 is trying to allocate memory of 296MiB for the forward activation.
Reducing the batch size or model is one possible way to avoid the OOM, using torch.utils.checkpoint could be used to trade compute for memory.