thanks @ptrblck! I start to narrow down the problem… now just need to understand what is causing it and how to fix it.
so the debugging showed me that @ layer2:


inputs don’t have any nan


outputs (that’s what I was looking at before, do as mentioned always at a column (which changes according to runs)

parameters have nans at weights (more specifically at weights_v



label: bias - torch.Size([2048]) - 0
label: weight_g - torch.Size([2048, 1]) - 0
label: weight_v - torch.Size([2048, 2048]) - 2048 [columns rows with nan: 60, ]

tried m.weight[60,:] and m.weight_v[60,:] which also shows all nan
So the questions now are… what is causing this and how to fix?
Could you help me once again at digging further or explain what is causing this?