Hello PyTorch community,
I used pytorch’s DataParallel module in order to parallelize training of a large model.  I noticed the utilization of my primary GPU (coda:0) would be far larger than my other GPUs.  I checked previous topics and implemented recommended changes such as calculating loss in the forward pass.  However this problem persists.  I profiled my GPU usage and noticed it is evenly distributed until calling backward() on my loss.  Memory usage on the primary GPU then increases for several iterations until plateauing around the 10-20th iteration.  My training loop logic is pretty basic besides taking advantage of gradient accumulation so I am at a how to fix my problem.
# model is a DataParallel module
for i, batch in enumerate(dataloader):
    if i % freq == 0:
        optimizer.zero_grad()

    loss = model(**batch) # even GPU utilization
    loss = loss.sum()
    loss.backward() # increase in primary GPU utilization

    if (i+1) % freq == 0:
        optimizer.step()

GPU usage before calling loss.backward()
+-------------------------------+----------------------+----------------------+
|   2  GeForce GTX 108...  Off  | 00000000:08:00.0 Off |                  N/A |
| 25%   26C    P8    15W / 250W |   3599MiB / 11178MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  GeForce GTX 108...  Off  | 00000000:09:00.0 Off |                  N/A |
| 25%   35C    P8    16W / 250W |   3633MiB / 11178MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   4  GeForce GTX 108...  Off  | 00000000:83:00.0 Off |                  N/A |
| 25%   35C    P8    15W / 250W |   3633MiB / 11178MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   5  GeForce GTX 108...  Off  | 00000000:84:00.0 Off |                  N/A |
| 25%   33C    P8    15W / 250W |   3633MiB / 11178MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

GPU usage after calling loss.backward()
+-------------------------------+----------------------+----------------------+
|   2  GeForce GTX 108...  Off  | 00000000:08:00.0 Off |                  N/A |
| 25%   28C    P2    72W / 250W |   7101MiB / 11178MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  GeForce GTX 108...  Off  | 00000000:09:00.0 Off |                  N/A |
| 25%   37C    P2    72W / 250W |   4335MiB / 11178MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   4  GeForce GTX 108...  Off  | 00000000:83:00.0 Off |                  N/A |
| 25%   36C    P2    66W / 250W |   4335MiB / 11178MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   5  GeForce GTX 108...  Off  | 00000000:84:00.0 Off |                  N/A |
| 25%   35C    P2    70W / 250W |   4335MiB / 11178MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

GPU usage a couple iterations later
+-------------------------------+----------------------+----------------------+
|   2  GeForce GTX 108...  Off  | 00000000:08:00.0 Off |                  N/A |
| 25%   32C    P2    73W / 250W |   9957MiB / 11178MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  GeForce GTX 108...  Off  | 00000000:09:00.0 Off |                  N/A |
| 25%   42C    P2    74W / 250W |   4335MiB / 11178MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   4  GeForce GTX 108...  Off  | 00000000:83:00.0 Off |                  N/A |
| 25%   41C    P2    73W / 250W |   4335MiB / 11178MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   5  GeForce GTX 108...  Off  | 00000000:84:00.0 Off |                  N/A |
| 25%   39C    P2    70W / 250W |   4335MiB / 11178MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+