Just so everything is on this page this is a summary of what should be happening:

saveAverageD is a function to be called in a backward hook to keep track of the average value in grad_out


valueTracker just has a single buffer to store this average


averageSaveConv: this keeps track of a single layer and a single instance of valueTracker to be used for average saving

I edited the code to have an extra print.  The following is now the output.  Again, this works exactly as you helped me get to with 1 GPU but fails with 2 GPUs.  As you can see with 1 GPU it loads both random inputs together and computes the average and saves with saveAverageD called one time on the one GPU.  on 2GPUs it calls saveAverageD twice, each GPU gets one of the inputs, but then the average buffer is not tracked.
$ CUDA_VISIBLE_DEVICES=1 python mainMinimalExampleMultiGPU.py 
calling with:
tensor([[[[-0.4504]],                                                                                                                                                                                                                                                          
                                                                                                                                                                                                                                                                               
         [[ 0.0442]],                                                                                                                                                                                                                                                          
                                                                                                                                                                                                                                                                               
         [[ 0.0555]],                                                                                                                                                                                                                                                          
                                                                                                                                                                                                                                                                               
         [[ 0.0622]],                                                                                                                                                                                                                                                          
                                                                                                                                                                                                                                                                               
         [[ 0.0781]],                                                                                                                                                                                                                                                          
                                                                                                                                                                                                                                                                               
         [[ 0.0507]],                                                                                                                                                                                                                                                          
                                                                                                                                                                                                                                                                               
         [[ 0.0317]],                                                                                                                                                                                                                                                          
                                                                                                                                                                                                                                                                               
         [[ 0.0323]],                                                                                                                                                                                                                                                          
                                                                                                                                                                                                                                                                               
         [[ 0.0349]],                                                                                                                                                                                                                                                          
                                                                                                                                                                                                                                                                               
         [[ 0.0607]]],                                                                                                                                                                                                                                                         
                                                                                                                                                                                                                                                                               
                                                                                                                                                                                                                                                                               
        [[[-0.4533]],                                                                                                                                                                                                                                                          
                                                                                                                                                                                                                                                                               
         [[ 0.0368]],                                                                                                                                                                                                                                                          
                                                                                                                                                                                                                                                                               
         [[ 0.0420]],                                                                                                                                                                                                                                                          
                                                                                                                                                                                                                                                                               
         [[ 0.0848]],                                                                                                                                                                                                                                                          
                                                                                                                                                                                                                                                                               
         [[ 0.0637]],                                                                                                                                                                                                                                                          
                                                                                                                                                                                                                                                                               
         [[ 0.0569]],                                                                                                                                                                                                                                                          
                                                                                                                                                                                                                                                                               
         [[ 0.0421]],                                                                                                                                                                                                                                                          
                                                                                                                                                                                                                                                                               
         [[ 0.0330]],                                                                                                                                                                                                                                                          
                                                                                                                                                                                                                                                                               
         [[ 0.0346]],                                                                                                                                                                                                                                                          
                                                                                                                                                                                                                                                                               
         [[ 0.0594]]]], device='cuda:0', dtype=torch.float64)                                                                                                                                                                                                                  
Conv Average Gradient:                                                                                                                                                                                                                                                         
[-0.009037267504024003, 0.0008102439568684652, 0.0009758195369250783, 0.001469522990143697, 0.001417696908934414, 0.0010763320891902935, 0.0007385695544331516, 0.0006529860781865106, 0.0006942532124915154, 0.0012018431768508766]                                           


$ CUDA_VISIBLE_DEVICES=0,1 python mainMinimalExampleMultiGPU.py                                                                                                                                               
calling with:                                                                                                                                                                                                                                                                  
calling with:                                                                                                                                                                                                                                                                  
tensor([[[[-0.4665]],                                                                                                                                                                                                                                                          
                                                                                                                                                                                                                                                                               
         [[ 0.0500]],

         [[ 0.0467]],

         [[ 0.0672]],

         [[ 0.0635]],

         [[ 0.0213]],

         [[ 0.0543]],

         [[ 0.0817]],

         [[ 0.0586]],

         [[ 0.0232]]]], device='cuda:1', dtype=torch.float64)
tensor([[[[-0.4679]],

         [[ 0.0354]],

         [[ 0.0464]],

         [[ 0.0746]],

         [[ 0.0570]],

         [[ 0.0386]],

         [[ 0.0673]],

         [[ 0.0740]],

         [[ 0.0512]],

         [[ 0.0235]]]], device='cuda:0', dtype=torch.float64)
Conv Average Gradient:
[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]