Distributed Training Gets Stuck
Hi,

When I freeze the BERT layers, distributed training works just fine. However, when I unfreeze the BERT layers, the first node continues training, and all other nodes wait on the training step with 100% GPU utilization on the first GPU. Is this expected behavior, or am I doing something wrong?

I'm using PyTorch 1.0.1post2, and 8 GTX 1080 GPUs per machine.

Best,
Moin
Hi,
One possible cause of this behavior may be the way you are freezing your parameters.

PyTorch's `DistributedDataParallel` is a rather sensitive beast as you can juge by the number of warnings in [its doc](https://pytorch.org/docs/stable/nn.html#torch.nn.parallel.DistributedDataParallel).

Two features are especially important to keep in mind:
- *You should never try to change your model’s parameters after wrapping up your model with DistributedDataParallel* or unexpected behaviors can happen, since some parameters’ gradient reduction functions might not get called.
- Another that is probably not important for you but you should be award of: Constructor, forward method, and differentiation of the output (backward pass) are *distributed synchronization point*.

So maybe you are freezing your parameters after having wrapped up your model with DistributedDataParallel?<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
