I’m a bit confused by the currently results.
It seems you are not only seeing a speedup using the main process to load the data (instead of another process), but could also reduce the time to update the weights?
Did you change anything else in the code, which might explain the runtime changes?