executing layer. So if you need to access all parameters from all layers at once there is a specific method to do it.
Most likely you won't need it, but if you do please refer to Gathering Parameters
We do however use it internally in several places, one such example is when loading pretrained model weights in
from_pretrained. We load one layer at a time and immediately partition it to all participating GPUs, as for very
large models it won't be possible to load it on one GPU and then spread it out to multiple GPUs, due to memory
limitations.
Also under ZeRO-3, if you write your own code and run into a model parameter weight that looks like:
python
tensor([1.0], device="cuda:0", dtype=torch.float16, requires_grad=True)
stress on tensor([1.]), or if you get an error where it says the parameter is of size 1, instead of some much
