One potential reason this could be happening is poisson sampling. If you’re using default setup with Opacus 1.0, then your dataloader is using a non-standard sampler (UniformWithReplacementSampler ). (it was available before, but had to be manually enabled).
With this sampler batches are of variable size: the average size is still the same, but some batches are larger than average. This leads to an increased memory requirement, as your memory is limited by the peak batch size, not average.
We have a utility to handle this: BatchMemoryManager (see example here 2)
If it’s not that, then it’s interesting and we’ll need to double check and see what could’ve caused it.