The size of the parameter doesn’t change, you only have parameters for batch size=1. But when you create several copies of it (for each batch element), you compute the gradient relatively to each batch element. So, when you update the parameter, you do it using the average direction all the elements in the batch. I debugged my code to confirm this behaviour before posting here