The -nan values are introduced by the Linear layers. The first layer introduces a handfull of those nans and it is getting worse with data traversing through each additional layer.
Most of the input data consists of zeros with usually a hand full of values differing from zero. Input data seems fine otherwise. No nan values to be found there.
Any suggestions/ideas on this?
Edit 1: Turns out that some bias values in the first layer are nan … How comes?
Edit 2: Bias values are fine upon initialization. Nans seem to appear when updating the network…
Edit 3: After learning on a first batch of size 64 a couple of nans already appear in the layer’s bias and weight values among some impressively big negative numbers. I’m afraid bias values are overflowing … My input data has not been scaled/normalized yet. I’ll try that and see if it helps…