Solved by JanoschMenke in post #2 
              
              
                50% of all Answers are found within 5 minutes after posting the questions.  
Rather than using a Sampler one can use  torch.utils.data.Subset(dataset, train_index)