Hello.
I am programming the ladder network and realize about a possible bug in your backward function. When ever the result of a variable which is part of the cost is 0, the backward method evaluate a NaN.
In this particular case the denoising function is given by:
mu_i=a1·nn.functional.sigmoid(a2·z_proj+a3)+a4·z_proj+a5
v_i=a6·nn.functional.sigmoid(a7·z_proj+a8)+a9·z_proj+a10
z_denoised = (z_level-mu_i)·v_i+mu_i
where z_level is the corrupted signal before the activation and the addition of BN params, and z_proj is the latent variable projected using BN(V·z^(l+1)).
I have checked the parameter initialization from the github AI ladder network code and use same initialization. In this case all the denoising function params “a” are initialized to zero, execpt a2 and a7. This means that in the first batch epoch the z_denoised value is 0 because mu_i and v_i are zero. In this case I am getting nan in the grad attribute from all the parameters of the ladder network (because all of them influence either z_proj or z_level). Whenever I change the initialization of “a” to 0.00001, everything works fine and I get a test error of 0.76 more or less (which means my implementation seems to be nearly ok).
Moreover If I use as denoising function something like:
mu_i=a1+a2·z_proj·a3+a4+a5
v_i=a6+a7·z_proj·a8+a9+a10
I also get nan. However if I use simply:
mu_i=a1+a2·z_proj+a3+a4+a5
v_i=a6+a7·z_proj+a8+a9+a10
Everything works fine. I think there is a problem in backward when evaluating a gradient at 0. Remark that the SSE function is doing:
MSE(x-x_recon), where x_recon is 0 in the first batch epoch, however x is not, as it is the clean projection, this means there is error to backpropagate. Moreover the error in the first epoch is like 180 so it is not as big that the update could cause NaN, and NaN are observed before optimizer.step()
Thanks. I could provide the code I am using if you request.