Hi,
I’m training a simple NN and I noticed a strange behavior during training and evaluation. I’m using DataLoaders with TensorDataset as follows
dataset_train = TensorDataset(X_train, Y_train)
dataset_test = TensorDataset(X_test, Y_test)

dataloader_train = DataLoader(dataset_train, batch_size=512, shuffle=True, drop_last=True,  
                                      pin_memory=False, num_workers=0)
dataloader_test = DataLoader(dataset_test, batch_size=512, shuffle=True, 
                                     pin_memory=True, num_workers=0)

Since the dataset is huge and since I want a fine grain evaluation of the network, instead of waiting an entire epoch to be completed, every X iterations (i.e. after X batches) it stops the training process, I evaluate the network and, finally, it continues with optimization.
This is how the loss function looks like testing every 50 and 250 iterations

Schermata 2018-07-02 alle 6.34.18 PM.jpg2076×1370 301 KB

This is very weird, as it seems that the randomness of the dataloader is somehow broken when I restart the process. The same spike in the loss (maybe caused by the same batch?) is repeating exactly every X iteration (50 as well as 250).
Note that I’m not manually setting any seed and I’m not using CUDA/GPUs.
Do you have any idea of what might cause this behaviour?