The intermediate forward activations should have been already freed during the backward() call, however the gradients would be set to zero in optimizer.zero_grad() and not removed. To save more memory, pass set_to_none=True to zero_grad, which would lower the memory footprint a bit more.
Also, based on your description it seems that the OOM issue is raised in a specific iteration regardless of the batch size, which could indicate a larger memory requirement for this iteration. Are you using variable input shapes and could this particular batch be larger?