Hi pvardanis!



 pvardanis:

What I mean by “bottleneck” is, that regression with 512 units as outputs is a more difficult task than classification,


If I understand your problem, the correct comparison is 512
“regressions” vs. 512 classifiers, each with 128 classes.  Not
only do I think that the classifier is not really the problem you
want to solve, I also don’t see that it would be easier.


especially since you are not supressing your output in any case


I’m not sure what you mean by this.  If you mean that for a label
of 0, a prediction of 0 and of 2*pi and of 1000*pi are all equally
good, so that your predictions could drift off and become very
large, I think you’re right that this could be a problem.


Still, I totally get the thing you said on penalizing the model on neighboring predictions and you’re right.


I do think that your real problem is to predict an angular variable (or
more precisely, a set of 512 angular variables), and the fact that your
targets are angular variables discretized to 128 different values is just
a technical detail.
There are probably standard approaches for predicting angular
variables with neural networks, but I don’t know what they are.
Having said that, here’s what I would suggest:
Represent each of your 512 angular outputs with two real numbers.
If those two number were forced to lie on the unit circle, they would
have only one degree of freedom, and that degree of freedom would
be an angle.
We will “encourage” (but not fully constrain) each pair of such numbers
to lie on the unit circle.  (Call them x and y.)
So build a network that outputs 1024 numbers, understood to be
512 pairs, (x_i, y_i).  Let your loss function have two terms: first,
something like
   ( (x_i**2 + y_i**2) - 1 )**2,

to penalize the pair (x_i, y_i) from moving off the unit circle; and
   1 - cos (theta_i - theta_pred_i),

where theta_i = atan2 (y_i, x_i) is the angle predicted by your
network, as represented by the pair (x_i, y_i).
(Again, 1 - cos (delta_theta) is the angular analog of the
conventional squared error, delta_x**2, that you would use for
a linear regression.)
I think that this is very much along the lines of what you and Mr. Meerkat
were discussing, except it deals more cleanly with the constraint implicit
in representing an angle with a pair of numbers, and uses the natural
angular version of squared error for the loss.
To be clear, in this scheme I imagine the last layer of your network to be
nn.Linear (n, 1024), fed into the loss function outlined above.
Good luck.
K. Frank