Are you using the summed loss or the mean (which is the default)?
Also, did you try to use MSELoss instead of L1Loss?
Since you have a fixed number of balls (1 to 10), you could also see this use case as a multi-class classification and use nn.CrossEntropyLoss instead. However, extrapolating wouldn’t work, if that’s in your use case.