So, within the leanring loop I did something like this:
count=0
for k,v in model.named_parameters():
    if not torch.isclose(v.grad.mean(),torch.tensor(0.0, device=device)):
        count +=1
print(count)

count=0
for k,v in model.named_parameters():
    if v.grad.mean() != 0:
        count +=1
print(count)

Turns out none of the means are zero but all of them are super close to zero.
At this stage, I somehow got the feeling that I am forgetting about something silly that is happening under the hood. I mean there has to be a reason why constructing tensors in the forward function makes it work and doing the same externally or in the init results in near zero gradients.