Most likely you could stabilize it using a more suitable weight initialization.
I’m not sure which method would work the best for your model, so you might want to try out some methods.
Here is a small example using xavier_uniform:
def weights_init(m):
    if isinstance(m, nn.Linear):
        nn.init.xavier_uniform_(m.weight.data)
        nn.init.xavier_uniform_(m.bias.data)
    

model = MyModel()
model.apply(weights_init)