tornadots:

But when I train the network all layers have zero-grads but last.
[…]
Because of the size of the process I use half and fp16 on model and input,output.


Yeah, that’s kind of dangerous, since your model and training might suffer from gradient underflow.
This might happen, if the gradients have a small magnitude and may thus not be representable in float16. You can use the built-in mixed-precision training utility via torch.cuda.amp, which uses a gradient scaling mechanism to prevent the underflow of the gradients as explained here 1.
You can find more information about it here.