Solved by 0phoff in post #8 
              
              
                Writing this for future people stumbling upon this issue. 
Whilst the solution proposed by @SimonW would certainly work, I decided to solve my problem by using a custom BatchSampler, as suggested by @apaszke on github. 
The reason being that this solution allows me to use any collate function, sampl…