Ok, so the results are:
999 item_emb.weight tensor(0.0596, device='cuda:0')
999 user_emb.weight tensor(0.0473, device='cuda:0')
999 item_layers.0.weight tensor(0.3596, device='cuda:0')
999 item_layers.0.bias tensor(0.0378, device='cuda:0')
999 user_layers.0.weight tensor(0.4759, device='cuda:0')
999 user_layers.0.bias tensor(0.0414, device='cuda:0')
1999 item_emb.weight tensor(0.0013, device='cuda:0')
1999 user_emb.weight tensor(0.0053, device='cuda:0')
1999 item_layers.0.weight tensor(0.0060, device='cuda:0')
1999 item_layers.0.bias tensor(0.0005, device='cuda:0')
1999 user_layers.0.weight tensor(0.1563, device='cuda:0')
1999 user_layers.0.bias tensor(0.0128, device='cuda:0')
2999 item_emb.weight tensor(0., device='cuda:0')
2999 user_emb.weight tensor(0., device='cuda:0')
2999 item_layers.0.weight tensor(0., device='cuda:0')
2999 item_layers.0.bias tensor(0., device='cuda:0')
2999 user_layers.0.weight tensor(0., device='cuda:0')
2999 user_layers.0.bias tensor(0., device='cuda:0')
3999 item_emb.weight tensor(0., device='cuda:0')
3999 user_emb.weight tensor(0., device='cuda:0')
3999 item_layers.0.weight tensor(0., device='cuda:0')
3999 item_layers.0.bias tensor(0., device='cuda:0')
3999 user_layers.0.weight tensor(0., device='cuda:0')
3999 user_layers.0.bias tensor(0., device='cuda:0')

The number in front of each row is the update step. I guess this was what you thought was happening.