Hi,
When you do  W1 = torch.randn(D_in,H,requires_grad=True).cuda() what is returned and stored as W1 is not a leaf anymore: it is the result of the differentiable op .cuda().
You should do W1 = torch.randn(D_in,H, device="cuda", requires_grad=True) to make sure W1 is a leaf and thus will get a .grad field.