Do you mean the batch size by “packet size”? If so, could you explain what you mean by



 slavavs:

Accordingly, I do loss.backward () in increments of 128.


loss.backward() will calculate the average loss for the input and target batch by default and will thus compute the average gradients for all parameters w.r.t. this loss.



 slavavs:

If I have a packet size of 128 * 8 = 1024, then my model cannot learn.


If you are increasing the batch size in a data parallel model you might need to adapt some hyperparameters such as the learning rate if your model is sensitive to the increased batch size.