How to unwrap after auto_wrap in FSDP?
I am currently fine-tuning a LLM (LLaMA) and would like to retrieve the gradients of each weight (parameter) after every gradient update. However, I notice that weights are (auto) wrapped into stuff like “_fsdp_wrapped_module._flat_param” during training. I need to map these wrapped weights to the original LLaMA architecture such as “self_attn.v_proj”. Any code examples?

I guess “summon_full_params()” might be the function that I look for, but I am not sure if that is correct. I also have difficulty using this function. Thanks a lot for any help!
This is more a question for the PyTorch forums as it's purely related to FSDP. Still cc-ing @pacman100 in case he has any idea.<|||||>Hello, as Sylvain mentioned this question is for the PyTorch forums. `summon_full_params` usage can be found in these tests: https://github.com/pytorch/pytorch/blob/main/test/distributed/checkpoint/test_fsdp_optim_state.py#L56-L59

I am not sure if it contains the information related to the gradients of a given parameter. <|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.