Yes, turns out that torch.no_grad was neither working as intended nor was it resolving the OOM issues.

I haven’t tried it random_unstructured yet but I’ll see if that’s any different.


Any fancy weight sharing? I’m asking because if a parameter is reused in different ways (for example when an LSTM is unrolled over the time-steps), then we might be having issues re-accessing the graph info after the first time. This would be a problem to pin down and address in pruning.

So for my configuration, I’m weight sharing the embedding tables for the encoder and the decoder. (I think this is somewhat standard practice in NMT). Of course, when I’m pruning them I only add the original module as subject to pruning (Not the “tied” one) Other than that, I don’t think there is any recurrence or weight sharing in standard transformer architecture.
Do you have some thoughts on whether how this routine might behave in a multiGPU training setting?
When I was getting the OOM error, I noticed that only one GPU was out of memory, whereas all other GPUs were basically empty. (Although during training all of them were being utilized)
2020-08-06 07:12:49 | INFO | fairseq.trainer | NOTE: Weights pruned, type: magnitude, amount: 0.009999324452827444, sparsity: 0.7000000065770643 2020-08-06 07:12:50 | WARNING | fairseq.trainer | OOM: Ran out of memory with exception: CUDA out of memory. Tried to allocate 246.00 MiB (GPU 0; 31.75 GiB total capacity; 26.00 GiB already allocated; 46
OOM on one GPU
2020-08-06 07:12:50 | WARNING | fairseq.trainer | |===========================================================================|
|                  PyTorch CUDA memory summary, device ID 0                 |
|---------------------------------------------------------------------------|
|           CUDA OOMs: 1             |       cudaMalloc retries: 6          |
|===========================================================================|
|        Metric         | Cur Usage  | Peak Usage | Tot Alloc  | Tot Freed  |
|---------------------------------------------------------------------------|
| Allocated memory      |   26377 MB |   26988 MB |  767743 GB |  767717 GB |
|       from large pool |   25981 MB |   26595 MB |  738567 GB |  738542 GB |
|       from small pool |     395 MB |     471 MB |   29175 GB |   29175 GB |
|---------------------------------------------------------------------------|
| Active memory         |   26377 MB |   26988 MB |  767743 GB |  767717 GB |
|       from large pool |   25981 MB |   26595 MB |  738567 GB |  738542 GB |
|       from small pool |     395 MB |     471 MB |   29175 GB |   29175 GB |
|---------------------------------------------------------------------------|
| GPU reserved memory   |   28544 MB |   28648 MB |   32826 MB |    4282 MB |
|       from large pool |   28138 MB |   28176 MB |   32024 MB |    3886 MB |
|       from small pool |     406 MB |     478 MB |     802 MB |     396 MB |
|---------------------------------------------------------------------------|
| Non-releasable memory |    1702 MB |    2343 MB |  670567 GB |  670565 GB |
|       from large pool |    1692 MB |    2325 MB |  641131 GB |  641129 GB |
|       from small pool |      10 MB |      74 MB |   29435 GB |   29435 GB |
|---------------------------------------------------------------------------|
| Allocations           |    1431    |    1533    |  296924 K  |  296923 K  |
|       from large pool |     420    |     434    |  133413 K  |  133413 K  |
|       from small pool |    1011    |    1299    |  163510 K  |  163509 K  |
|---------------------------------------------------------------------------|
| Active allocs         |    1431    |    1533    |  296924 K  |  296923 K  |
|       from large pool |     420    |     434    |  133413 K  |  133413 K  |
|       from small pool |    1011    |    1299    |  163510 K  |  163509 K  |
|---------------------------------------------------------------------------|
| GPU reserved segments |     343    |     398    |     616    |     273    |
|       from large pool |     140    |     159    |     215    |      75    |
|       from small pool |     203    |     239    |     401    |     198    |
|---------------------------------------------------------------------------|
| Non-releasable allocs |     154    |     237    |  195998 K  |  195998 K  |
|       from large pool |      88    |     147    |   96157 K  |   96157 K  |
|       from small pool |      66    |     124    |   99841 K  |   99841 K  |
|===========================================================================|

Nothing on other GPUs
2020-08-06 07:12:50 | WARNING | fairseq.trainer | |===========================================================================|
|                  PyTorch CUDA memory summary, device ID 2                 |
|---------------------------------------------------------------------------|
|           CUDA OOMs: 0             |       cudaMalloc retries: 0          |
|===========================================================================|
|        Metric         | Cur Usage  | Peak Usage | Tot Alloc  | Tot Freed  |
|---------------------------------------------------------------------------|
| Allocated memory      |       0 B  |       0 B  |       0 B  |       0 B  |
|       from large pool |       0 B  |       0 B  |       0 B  |       0 B  |
|       from small pool |       0 B  |       0 B  |       0 B  |       0 B  |
|---------------------------------------------------------------------------|
| Active memory         |       0 B  |       0 B  |       0 B  |       0 B  |
|       from large pool |       0 B  |       0 B  |       0 B  |       0 B  |
|       from small pool |       0 B  |       0 B  |       0 B  |       0 B  |
|---------------------------------------------------------------------------|
| GPU reserved memory   |       0 B  |       0 B  |       0 B  |       0 B  |
|       from large pool |       0 B  |       0 B  |       0 B  |       0 B  |
|       from small pool |       0 B  |       0 B  |       0 B  |       0 B  |
|---------------------------------------------------------------------------|
| Non-releasable memory |       0 B  |       0 B  |       0 B  |       0 B  |
|       from large pool |       0 B  |       0 B  |       0 B  |       0 B  |
|       from small pool |       0 B  |       0 B  |       0 B  |       0 B  |
|---------------------------------------------------------------------------|
| Allocations           |       0    |       0    |       0    |       0    |
|       from large pool |       0    |       0    |       0    |       0    |
|       from small pool |       0    |       0    |       0    |       0    |
|---------------------------------------------------------------------------|
| Active allocs         |       0    |       0    |       0    |       0    |
|       from large pool |       0    |       0    |       0    |       0    |
|       from small pool |       0    |       0    |       0    |       0    |
|---------------------------------------------------------------------------|
| GPU reserved segments |       0    |       0    |       0    |       0    |
|       from large pool |       0    |       0    |       0    |       0    |
|       from small pool |       0    |       0    |       0    |       0    |
|---------------------------------------------------------------------------|
| Non-releasable allocs |       0    |       0    |       0    |       0    |
|       from large pool |       0    |       0    |       0    |       0    |
|       from small pool |       0    |       0    |       0    |       0    |
|===========================================================================|