Parameters are update inplace by optimizers as seen here 70.
@Raphael_Emeka if you think this might be the issue, you might be running into this issue 61 and would need to check your wokflow to make sure no stale forward activations are used.