Thanks for the update.
Could you execute these runs and see, if the behavior changes, as a sync wouldn’t be strictly necessary, but would help us to isolate is an internal method is broken:

add synchronizations in the train_data loop before and after each line of code
set non_blocking=False

rerun the test with and without pin_memory in the DataLoader


Also, do you have a code snippet to reproduce the issue and could post the output of python -m torch.utils.collect_env here?