I think that you are initializing the network before the dataloader. For this reason, when you change the network size, the samples generated by the dataloader also change. Because, as you know all filters and bias need to be initialized normally using random methods. A change of the number of times that random method is used implies a change to the state of the random generator and this can be the cause that your dataloader is generating different values.