hello,
recently, without changing anything knowingly, there is a change in the output of an nvidia-smi output i’m getting.
it used to be that every experiment that was running on a gpu would show up as one line in the output. now it iss broken down to the main process and next to it are many child processes all consisting of 305 MB.
it looks like this:
±----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A     54920      C   Insufficient Permissions         9009MiB |
|    1   N/A  N/A     22440      C   …3\envs\env_g\python.exe      305MiB |
|    1   N/A  N/A     51512      C   …3\envs\env_g\python.exe      305MiB |
|    1   N/A  N/A     59872      C   …3\envs\env_g\python.exe      305MiB |
|    1   N/A  N/A     72412      C   …3\envs\env_g\python.exe      305MiB |
|    1   N/A  N/A     80316      C   …3\envs\env_g\python.exe      305MiB |
|    1   N/A  N/A     84680      C   …3\envs\env_g\python.exe      305MiB |
|    1   N/A  N/A     95284      C   …3\envs\env_g\python.exe      305MiB |
|    1   N/A  N/A    102060      C   …3\envs\env_g\python.exe     9023MiB |
|    1   N/A  N/A    102312      C   …3\envs\env_g\python.exe     9023MiB |
|    1   N/A  N/A    108944      C   …3\envs\env_g\python.exe      305MiB |
|    1   N/A  N/A    115004      C   …3\envs\env_g\python.exe      305MiB |
|    1   N/A  N/A    118164      C   …3\envs\env_g\python.exe      305MiB |
|    1   N/A  N/A    129200      C   …3\envs\env_g\python.exe      305MiB |
|    1   N/A  N/A    158836      C   …3\envs\env_g\python.exe      305MiB |
|    2   N/A  N/A     25628      C   …3\envs\env_g\python.exe      305MiB |
|    2   N/A  N/A     34464      C   …3\envs\env_g\python.exe      305MiB |
|    2   N/A  N/A     39076      C   …3\envs\env_g\python.exe      305MiB |
|    2   N/A  N/A     44792      C   …3\envs\env_g\python.exe      305MiB |
|    2   N/A  N/A     46720      C   …3\envs\env_g\python.exe      305MiB |
|    2   N/A  N/A     55592      C   …3\envs\env_g\python.exe      305MiB |
|    2   N/A  N/A     57384      C   …3\envs\env_g\python.exe      305MiB |
|    2   N/A  N/A     74716      C   …3\envs\env_g\python.exe      305MiB |
|    2   N/A  N/A     83288      C   …3\envs\env_g\python.exe      305MiB |
|    2   N/A  N/A     88856      C   …3\envs\env_g\python.exe      305MiB |
|    2   N/A  N/A    124900      C   …3\envs\env_g\python.exe      305MiB |
|    2   N/A  N/A    136132      C   …3\envs\env_g\python.exe      305MiB |
|    2   N/A  N/A    137340      C   …3\envs\env_g\python.exe     8707MiB |
|    2   N/A  N/A    158128      C   …3\envs\env_g\python.exe     8799MiB |
|    3   N/A  N/A     20152      C   …3\envs\env_g\python.exe      305MiB |
|    3   N/A  N/A     32260      C   …3\envs\env_g\python.exe      305MiB |
|    3   N/A  N/A     49008      C   …3\envs\env_g\python.exe      305MiB |
|    3   N/A  N/A     69284      C   …3\envs\env_g\python.exe     8801MiB |
|    3   N/A  N/A     76820      C   …3\envs\env_g\python.exe      305MiB |
|    3   N/A  N/A     85804      C   …3\envs\env_g\python.exe      305MiB |
|    3   N/A  N/A     95104      C   …3\envs\env_g\python.exe      305MiB |
|    3   N/A  N/A     97580      C   …3\envs\env_g\python.exe      305MiB |
|    3   N/A  N/A    104040      C   …3\envs\env_g\python.exe      305MiB |
|    3   N/A  N/A    107568      C   …3\envs\env_g\python.exe      305MiB |
|    3   N/A  N/A    111028      C   …3\envs\env_g\python.exe      305MiB |
|    3   N/A  N/A    135936      C   …3\envs\env_g\python.exe     8801MiB |
|    3   N/A  N/A    136636      C   …3\envs\env_g\python.exe      305MiB |
|    3   N/A  N/A    158840      C   …3\envs\env_g\python.exe      305MiB |
±----------------------------------------------------------------------------+
notice a few things:


another user is running a process and it’s not showing any other processes.


on every card there are 1 or 2 main processes, and many other small ones.


looking up the mini process in the task manager, under “command line” it says: “from multiprocessing.spawn import spawn_main; spawn_main(parent_pid=10234, pipe_handle=34325)” "–multiprocessing-fork


i’m using 2 workers per main process running, so i assume it’s not that.


did anybody come across something similar? would appreciate any advice