Three throughts:

To use nn.Parameter in the forward is almost certainly a mistake. Parameters should be defined in __init__ and then used in the forward.
I’m not sure that unit variance is a good initialization.

tanh is a classic activation function, but I’m not sure that it is widely used these days.