agt:

If the second layer has 3D, but the output is 1D, how will the weights for the second layer get updated? Will it calculate 3 separate gradients for all 3 elements only for the second layer, and deeper layers will have only 1 shared gradient?


That sounds right.
But this description somehow ignores hidden layer and non-linearity, needed to get a universal function approximator.
So, minimalistic network looks like: 3->64->relu->1, it has linear maps: 3x64,64x1. Thing is, your proposed 1d linear transformations, prepended to this network, won’t do anything beyond what 3x64 map does.
You can of course prepend 1d non-linear transformations to that network, but I don’t see a point.