Hey @khalil
I am trying to identify which component hogs memory. Could you please share some more details about the sentence below.

I have trained this model in one node and it consumed 30GB on the cache

By “trained this model in one node”, do you still use DDP and the same data loader? It will be helpful to know the memory footprint of the following cases:

Train a local model without data loader and without DDP. Just feed the model with some randomly generated tensors.
Train a local model with data loader but without DDP.
Wrap the model with DDP, and train it with data loader.