Before training, I set the params to be non-negative.
for param in r.parameters():
    param.data = param.data.abs()

I get the following warning

UserWarning: RNN module weights are not part of single contiguous chunk of memory. This means they need to be compacted at every call, possibly greately increasing memory usage. To compact weights again call flatten_parameters().

However, now my problem seems to be solved. Happy to close this thread if someone can verify if the non-negative weight initialization is the right approach.