You could add the weight’s L2 norm to the loss before optimizing
l2_lambda = 0.01
l2_reg = torch.tensor(0.)
for name, param in model.named_parameters():
    if 'conv' in name:
        l2_reg += torch.linalg.norm(param)
loss += l2_lambda * l2_reg

AFAIK the weight_decay parameter in your optimizer will apply to all parameters in the network, and not just the conv layers.