agt:

How are the values updated by default? Will it just be an average of the 3 gradients as if they form a batch in gradient descent?


Yes



 agt:

What if the output has a 1 element dimension, then how will the previous layers with 3 elements train?


This means that you combine elements somewhere, so they’re not independent. Your 3 intermediate outputs are just like hidden features, so it makes less sense to train them as independent (1x3 vs 3x3 map) or scale gradients, as they’ll be entangled later anyway.



 agt:

How do you do that in pytorch?


Pytorch loss classes have reduction=‘mean’ argument. If you use reduction=‘none’, you can adjust elementwise losses before computing mean.