Have you guys tried setting the fill values of the parameters manually in the model init?
For example:
fill_val=0.000000001
for m in self.modules():
    if isinstance(m, nn.Conv2d):
        if not isinstance(m.bias, type(None)):
            m.bias.data.fill_(fill_val)

for m in self.modules():
    if isinstance(m, nn.Linear):
        if not isinstance(m.bias, type(None)):
            m.bias.data.fill_(fill_val)

Can also try lowering your learning rate.