mrshenli:

I would try to print it every few iterations.


I adjusted the code like to print before during and after training:
    print(torch.cuda.memory_summary())
    sampler = torch.utils.data.DistributedSampler(dataset, 1, rank, shuffle=shuffle) if dist_sampler else None
    loader = torch.utils.data.DataLoader(dataset, batch_size=4, shuffle=False, num_workers=0, pin_memory=False,
                                         sampler=sampler, collate_fn=dataset.collate_fn)
    print("start trainig")
    start = datetime.now()
    for epoch in range(1):
        for i, inputs in enumerate(loader):
            for k, v in inputs.items():
                if isinstance(v, torch.Tensor):
                    inputs[k] = v.cuda(rank, non_blocking=True)
            if i % 10 == 0:
                print(torch.cuda.memory_summary())
            optimizer.zero_grad()
            outputs = model(**inputs)
            loss = outputs[0]
            loss = loss.mean()
            loss.backward()
            optimizer.step()
            if i >= n_steps:
                break
    print("Training complete in:", str(datetime.now() - start))
    print(torch.cuda.memory_summary())

and then called the function three times again like before:

    train_single()
    train_single(dist_sampler=True)
    train_single(dist_sampler=True, shuffle=True)

This is what happens during the third function call (DistributedSampler with shuffle=True). The other outputs did not show any irregularities so i’ll leave them out (and due to the character limit…). First output is before the dataloader is build, second print is after the first Iteration, third after the tenth iteration
|===========================================================================|
|                  PyTorch CUDA memory summary, device ID 0                 |
|---------------------------------------------------------------------------|
|            CUDA OOMs: 0            |        cudaMalloc retries: 0         |
|===========================================================================|
|        Metric         | Cur Usage  | Peak Usage | Tot Alloc  | Tot Freed  |
|---------------------------------------------------------------------------|
| Allocated memory      |  340029 KB |    1692 MB |  350966 MB |  350634 MB |
|       from large pool |  339456 KB |    1677 MB |  321965 MB |  321633 MB |
|       from small pool |     573 KB |     133 MB |   29001 MB |   29001 MB |
|---------------------------------------------------------------------------|
| Active memory         |  340029 KB |    1692 MB |  350966 MB |  350634 MB |
|       from large pool |  339456 KB |    1677 MB |  321965 MB |  321633 MB |
|       from small pool |     573 KB |     133 MB |   29001 MB |   29001 MB |
|---------------------------------------------------------------------------|
| GPU reserved memory   |    1842 MB |    1842 MB |    1842 MB |       0 B  |
|       from large pool |    1700 MB |    1700 MB |    1700 MB |       0 B  |
|       from small pool |     142 MB |     142 MB |     142 MB |       0 B  |
|---------------------------------------------------------------------------|
| Non-releasable memory |   51138 KB |  215041 KB |  393608 MB |  393558 MB |
|       from large pool |   49664 KB |  186288 KB |  362929 MB |  362880 MB |
|       from small pool |    1474 KB |   36244 KB |   30679 MB |   30678 MB |
|---------------------------------------------------------------------------|
| Allocations           |     204    |    1077    |  251092    |  250888    |
|       from large pool |      75    |     459    |  126113    |  126038    |
|       from small pool |     129    |     756    |  124979    |  124850    |
|---------------------------------------------------------------------------|
| Active allocs         |     204    |    1077    |  251092    |  250888    |
|       from large pool |      75    |     459    |  126113    |  126038    |
|       from small pool |     129    |     756    |  124979    |  124850    |
|---------------------------------------------------------------------------|
| GPU reserved segments |     156    |     156    |     156    |       0    |
|       from large pool |      85    |      85    |      85    |       0    |
|       from small pool |      71    |      71    |      71    |       0    |
|---------------------------------------------------------------------------|
| Non-releasable allocs |      20    |     177    |  133360    |  133340    |
|       from large pool |      19    |      80    |   87375    |   87356    |
|       from small pool |       1    |      98    |   45985    |   45984    |
|===========================================================================|

start trainig
|===========================================================================|
|                  PyTorch CUDA memory summary, device ID 0                 |
|---------------------------------------------------------------------------|
|            CUDA OOMs: 0            |        cudaMalloc retries: 0         |
|===========================================================================|
|        Metric         | Cur Usage  | Peak Usage | Tot Alloc  | Tot Freed  |
|---------------------------------------------------------------------------|
| Allocated memory      |  340079 KB |    1692 MB |  350966 MB |  350634 MB |
|       from large pool |  339456 KB |    1677 MB |  321965 MB |  321633 MB |
|       from small pool |     623 KB |     133 MB |   29001 MB |   29001 MB |
|---------------------------------------------------------------------------|
| Active memory         |  340079 KB |    1692 MB |  350966 MB |  350634 MB |
|       from large pool |  339456 KB |    1677 MB |  321965 MB |  321633 MB |
|       from small pool |     623 KB |     133 MB |   29001 MB |   29001 MB |
|---------------------------------------------------------------------------|
| GPU reserved memory   |    1842 MB |    1842 MB |    1842 MB |       0 B  |
|       from large pool |    1700 MB |    1700 MB |    1700 MB |       0 B  |
|       from small pool |     142 MB |     142 MB |     142 MB |       0 B  |
|---------------------------------------------------------------------------|
| Non-releasable memory |   51089 KB |  215041 KB |  393608 MB |  393558 MB |
|       from large pool |   49664 KB |  186288 KB |  362929 MB |  362880 MB |
|       from small pool |    1425 KB |   36244 KB |   30679 MB |   30678 MB |
|---------------------------------------------------------------------------|
| Allocations           |     207    |    1077    |  251095    |  250888    |
|       from large pool |      75    |     459    |  126113    |  126038    |
|       from small pool |     132    |     756    |  124982    |  124850    |
|---------------------------------------------------------------------------|
| Active allocs         |     207    |    1077    |  251095    |  250888    |
|       from large pool |      75    |     459    |  126113    |  126038    |
|       from small pool |     132    |     756    |  124982    |  124850    |
|---------------------------------------------------------------------------|
| GPU reserved segments |     156    |     156    |     156    |       0    |
|       from large pool |      85    |      85    |      85    |       0    |
|       from small pool |      71    |      71    |      71    |       0    |
|---------------------------------------------------------------------------|
| Non-releasable allocs |      20    |     177    |  133360    |  133340    |
|       from large pool |      19    |      80    |   87375    |   87356    |
|       from small pool |       1    |      98    |   45985    |   45984    |
|===========================================================================|

|===========================================================================|
|                  PyTorch CUDA memory summary, device ID 0                 |
|---------------------------------------------------------------------------|
|            CUDA OOMs: 0            |        cudaMalloc retries: 0         |
|===========================================================================|
|        Metric         | Cur Usage  | Peak Usage | Tot Alloc  | Tot Freed  |
|---------------------------------------------------------------------------|
| Allocated memory      |    1328 MB |   11131 MB |  494742 MB |  493413 MB |
|       from large pool |    1326 MB |   11128 MB |  465663 MB |  464337 MB |
|       from small pool |       2 MB |     133 MB |   29078 MB |   29075 MB |
|---------------------------------------------------------------------------|
| Active memory         |    1328 MB |   11131 MB |  494742 MB |  493413 MB |
|       from large pool |    1326 MB |   11128 MB |  465663 MB |  464337 MB |
|       from small pool |       2 MB |     133 MB |   29078 MB |   29075 MB |
|---------------------------------------------------------------------------|
| GPU reserved memory   |   13646 MB |   13646 MB |   13646 MB |       0 B  |
|       from large pool |   13504 MB |   13504 MB |   13504 MB |       0 B  |
|       from small pool |     142 MB |     142 MB |     142 MB |       0 B  |
|---------------------------------------------------------------------------|
| Non-releasable memory |  158950 KB |    3717 MB |  474414 MB |  474259 MB |
|       from large pool |  157520 KB |    3717 MB |  443648 MB |  443494 MB |
|       from small pool |    1430 KB |      35 MB |   30766 MB |   30765 MB |
|---------------------------------------------------------------------------|
| Allocations           |     816    |    1077    |  265399    |  264583    |
|       from large pool |     297    |     496    |  134867    |  134570    |
|       from small pool |     519    |     756    |  130532    |  130013    |
|---------------------------------------------------------------------------|
| Active allocs         |     816    |    1077    |  265399    |  264583    |
|       from large pool |     297    |     496    |  134867    |  134570    |
|       from small pool |     519    |     756    |  130532    |  130013    |
|---------------------------------------------------------------------------|
| GPU reserved segments |     297    |     297    |     297    |       0    |
|       from large pool |     226    |     226    |     226    |       0    |
|       from small pool |      71    |      71    |      71    |       0    |
|---------------------------------------------------------------------------|
| Non-releasable allocs |     109    |     191    |  140979    |  140870    |
|       from large pool |      74    |     154    |   92152    |   92078    |
|       from small pool |      35    |      98    |   48827    |   48792    |
|===========================================================================|