Hi Janosch!



 JanoschMenke:

But if a is learnable, would the netowkr not start minimizing the loss function which is easier to minimize and then just fit a to only value that loss function?


Conceptually, yes, a would train to prefer the easier loss.  But, as
written, a could train to become a very large positive number or a
very large negative number, with your loss becoming unbounded
below.
(You could train (1 - a)**2 * loss_reg + a**2 * loss_clf,
but now a will just train to 0 or 1 to ignore the “harder” loss.)
As to your original question, it does make sense to consider weighting
the two losses, e.g., something like:
loss = loss_reg + alpha * loss_clf

But alpha should now be a non-trainable hyperparameter whose
value you tune “by hand.”  (You could also contemplate using some
automated hyperparameter-tuning scheme, but those are often more
complicated than they are worth.)
Best.
K. Frank