Solved by ptrblck in post #2 
              
              
                You are initializing all parameters and the inputs with torch::rand, which will sample values from a uniform distribution on the interval [0, 1) and will thus yield only positive numbers. 
Increasing the layer size will thus easily saturate the activation and thus yield these saturated values. 
Use …