I assume you’ve implemented the VectorQuantizerEMA module?
Are you seeing the OOM directly in the first iteration or are you seeing an increased memory usage?
In the former case, you might need to reduce the batch size or if that’s not possible you could try to use torch.utils.checkpoint to trade compute for memory.