Solved by ptrblck in post #7 
              
              
                Yes, I would interpret these samples in a different range as samples from another data distribution. 
By saturation I mean that the parameters of your model were trained to create useful features using a normalized input. If you provide now much larger input values, the magnitude of your activation …