Sorry I jumped straight to the criticism, I am a big fan, keep up the good work.

No, I am not scaling the dropped out blocks with the inverse dropout probabilities. They are simply not dropped during inference.

Right, but it is being applied during training and it’s the difference between training and inference that I wonder about.
My intuition for why dropout uses that scaling factor is so that the mean value of the feature is the same before and after the dropout. But each example is independent, they don’t share statistics. In training the next layer sees layer(x)/keep_prob when the layer is kept, and 0 otherwise. No mixing. So the average across samples is preserved, but maybe the average value seen for any sample is not realistic.
And I see slightly better results (just 1 run) after dropping that factor:
Before:
Epoch 28/30
352/352 [==============================] - 13s 37ms/step - loss: 0.9469 - accuracy: 0.8020 - top-5-accuracy: 0.9899 - val_loss: 1.0130 - val_accuracy: 0.7766 - val_top-5-accuracy: 0.9870
Epoch 29/30
352/352 [==============================] - 13s 36ms/step - loss: 0.9326 - accuracy: 0.8079 - top-5-accuracy: 0.9901 - val_loss: 1.0455 - val_accuracy: 0.7674 - val_top-5-accuracy: 0.9844
Epoch 30/30
352/352 [==============================] - 13s 37ms/step - loss: 0.9296 - accuracy: 0.8097 - top-5-accuracy: 0.9902 - val_loss: 0.9982 - val_accuracy: 0.7822 - val_top-5-accuracy: 0.9838
313/313 [==============================] - 2s 8ms/step - loss: 1.0239 - accuracy: 0.7758 - top-5-accuracy: 0.9837
Test accuracy: 77.58%
Test top 5 accuracy: 98.37%

After:
Epoch 28/30
352/352 [==============================] - 13s 37ms/step - loss: 0.9268 - accuracy: 0.8117 - top-5-accuracy: 0.9908 - val_loss: 0.9599 - val_accuracy: 0.8050 - val_top-5-accuracy: 0.9872
Epoch 29/30
352/352 [==============================] - 13s 37ms/step - loss: 0.9255 - accuracy: 0.8136 - top-5-accuracy: 0.9910 - val_loss: 0.9751 - val_accuracy: 0.7942 - val_top-5-accuracy: 0.9868
Epoch 30/30
352/352 [==============================] - 13s 37ms/step - loss: 0.9132 - accuracy: 0.8181 - top-5-accuracy: 0.9923 - val_loss: 0.9745 - val_accuracy: 0.7952 - val_top-5-accuracy: 0.9870
313/313 [==============================] - 3s 9ms/step - loss: 0.9976 - accuracy: 0.7867 - top-5-accuracy: 0.9855
Test accuracy: 78.67%
Test top 5 accuracy: 98.55%

Interesting. IIUC, here’s that factor in the code from the original paper, so maybe I’m wrong:

  

      github.com
  

  
    yueatsprograms/Stochastic_Depth/blob/master/ResidualDrop.lua#L52 1



      
      
          end
          
          
function ResidualDrop:updateOutput(input)
              local skip_forward = self.skip:forward(input)
              self.output:resizeAs(skip_forward):copy(skip_forward)
              if self.train then
                if self.gate then -- only compute convolutional output when gate is open
                  self.output:add(self.net:forward(input))
                end
              else
                self.output:add(self.net:forward(input):mul(1-self.deathRate))
              end
              return self.output
          end
          
          
function ResidualDrop:updateGradInput(input, gradOutput)
             self.gradInput = self.gradInput or input.new()
             self.gradInput:resizeAs(input):copy(self.skip:updateGradInput(input, gradOutput))
             if self.gate then
                self.gradInput:add(self.net:updateGradInput(input, gradOutput))
             end
      
    
  


  

  
    
    
  

  



Is wrapped the block we are applying?

Yes, that’s what I meant here that this layer is like “maybe apply the wrapped layer”,  (It should also check the training flag…)
Either way, fun stuff.
Thanks again Sayak!