Thank you ptrblck for great answer, as always. It does work but I stumbled upon a strange issue.
I split my training set into training and validation set using a deterministic seed as mentioned:
torch.manual_seed(0)
train_dataset, val_dataset = torch.utils.data.random_split(trainval_dataset, [train_size, val_size])

I wanted to test the CNN then on a validation set (using torchvision CIFAR10). When I test it on a testset, the accuracy is always the same as expected. However, when I test it on validation set, the accuracy changes. When delving into the code, I realized the problem is not with the validation set (whose targets are the same every time I run an instance of a script) but actually the network spits different outputs.
How is it possible? Especially, that it appears when feeding the validation set but not a test set?