I watch RSS in top.(not nvidia-smi)
RSS value is increased to full, then previous call trace appeared.
Also, I found “main.py --no-cuda” execute many times(like 3 times) under bottleneck profiler.
Is there any good solution to solve this?
$ time python -m torch.utils.bottleneck main.py --no-cuda

RSS value is increased
top - 09:42:05 up 1 day,  9:58,  0 users,  load average: 1.42, 2.43, 2.34
Tasks:  15 total,   2 running,  13 sleeping,   0 stopped,   0 zombie
%Cpu(s): 25.8 us,  4.6 sy,  0.7 ni, 68.8 id,  0.0 wa,  0.0 hi,  0.1 si,  0.0 st
KiB Mem : 16025224 total,  2487640 free, 11505224 used,  2032360 buff/cache
KiB Swap: 16379900 total,  6105128 free, 10274772 used.  3980392 avail Mem

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
  2319 jovyan    20   0 23.346g 9.652g 318348 R 100.3 63.2  16:09.42 python
     1 root      20   0    4364      0      0 S   0.0  0.0   0:03.41 tini
     7 root      20   0   49152    180    176 S   0.0  0.0   0:00.00 sudo
    16 jovyan    20   0 2110016  25644   4456 S   0.0  0.2   0:51.18 jupyterhub-sing
    24 jovyan    20   0   20120    132    128 S   0.0  0.0   0:00.35 bash
    63 jovyan    20   0   20120     24     24 S   0.0  0.0   0:00.37 bash
   153 jovyan    20   0   20120    136    132 S   0.0  0.0   0:00.43 bash
   187 jovyan    20   0 2153728   7640    900 S   0.0  0.0   0:05.49 python
   262 jovyan    20   0   20132   2388   1996 S   0.0  0.0   0:00.59 bash
   539 jovyan    20   0   20120    256    252 S   0.0  0.0   0:00.43 bash
  1119 jovyan    20   0   20120    284    280 S   0.0  0.0   0:00.43 bash
  1286 jovyan    20   0   24332    300    296 S   0.0  0.0   0:00.00 git
  1287 jovyan    20   0    8376      0      0 S   0.0  0.0   0:00.00 pager
  1404 jovyan    20   0   20120   2248   1868 S   0.0  0.0   0:00.54 bash
  2352 jovyan    20   0   40392   3600   3128 R   0.0  0.0   0:00.28 top