I call model.w_init and model.w_hidden before and after an epoch or more and realise the initial values aren’t too far off from the lastly updated ones and very similar amongst each other. For example, with a learning rate of 0.001, starts at:

and after one epoch:

With a learning rate of 0.01, ends at: