I haven’t used this approach in practice, but I can understand the procedure. I doubt you can measure confidence using dropout however, but you can evaluate the resilience to co-adaptation between neurons, and therefore the regularization of the network…
As to the actual procedure, I think you really need to scale by hand, because as you said, PyTorch doesn’t do anything with dropout at test time. Therefore, you will have fewer nodes in the layer, which won’t contribute as much to the next layer’s input as in the default network. This will produce very different results than the untouched network.
Note that it’s easier to use nn.functional.dropout if you want it active during evaluation, as model.eval() will deactivate the nn.Dropout layers…