Solved by lmnt in post #2 
              
              
                So it turns out that when using torch.functional.dropout(layer, p=1), the default value for training=False, which did not use dropout each time hence it makes sense the network would learn. 
I’ve now set it to take torch.functional.dropout(layer, p=1, training=self.training) and now the output of th…