Hi, ptrblck.



 ptrblck:

Could you check. If the outputs converge towards a constant value, e.g. a zero tensor?


Yes they converge to zero. The following code, after few iterations, prints:
print(anchor)
print(loss)

# tensor([[0., 0., 0., ..... 0., 0.]], grad_fn=<ReluBackward0>)
# tensor(0.5000, grad_fn=<MeanBackward1>)




 ptrblck:

Could you try to lower the learning rate a bit and check, if the loss decreases further?


With a learning rate of 1e-3 the same happens, it just takes more iterations for the values to converge. With a learning rate of 1e-4 I haven’t run the code long enough for it to converge but with every iteration there is fewer non zero values in the output vectors(i.e, it’s converging towards a zero vector).
The above results are when using random input data, and the above results, kind of make sense to me. Since the input data is random, the network won’t be able to do any better than .05, and the best way to converge to that error value is to always return output vectors of zeros.
Is my reasoning wright?
On the other hand when using real data the following happens:
Figure.png1214×707 54.4 KB

The above plot is the mean(100 values) error for 10000 iterations using a learning rate of 1e-4. The above is great, but what bothers is that the feature vectors returned by the network are mostly zeros:
tensor([[0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,
         0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,
         0.0000, 0.0000, 0.0000, 0.0000, 0.1780, 0.0000, 0.0000, 0.0000, 0.0000,
         0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.2754,
         0.0534, 0.2608, 0.0000, 0.0000, 0.2360, 0.0000, 0.0000, 0.0000, 0.0000,
         0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0242, 0.0000, 0.0000, 0.3901,
         0.0000, 0.0557, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,
         0.0000]], grad_fn=<ReluBackward0>)

To my understanding this is because of the Relu activation functions I am using between every layer of the network. Could I get better results removing some of the relu layers in my network.
Thanks.