Sounds good! Let me know if you can create a reproducible colab with the issue.
By the way, inspecting the current code, I wonder how do you create data loaders. The way it is done it is unlikely that the training would do a DDP. Data will be the same for all processes and this is probably undesired.