The majority of the parameters have a valid gradient, while some don’t and you can check it via:
for name, param in model.named_parameters():
    if param.grad == None:
        print(name, 'is None')
    else:
        print('param {}: {}'.format(name, param.grad.abs().sum()))

The parameters without a .grad attribute are:
transformer.mask_emb is None
transformer.layer.0.rel_attn.r_s_bias is None
transformer.layer.0.rel_attn.seg_embed is None
...

I’m not familiar with this model implementation, but guess this is on purpose.
@stas would most likely be familiar with the implementation.