Thank you Alex for the feedback and your time 



 googlebot:

You should understand that with L1 loss (and non-quadaratic loss zones in general) you’re doing gradient descent based on error sign only, this may converge with help from lr scheduler and/or adaptive optimizer, but it is intrinsically harder, and big (compared to LR) initial errors may be uncorrectable w/o downscaling.


Ok, got that. So downscaling my values is necessary ;-).



 googlebot:

Interesting log-transform results… I’d guess that your optimizer’s steps are too big, if you end up with 10^5 errors. This stuff with clusters may actually be bad, as gradients are scaled ~100-1000 times for your last clusters; this can mess up shared parameters and gradient variance estimates too much. Other possibilities include inappropriate LR & scheduler params or non-quadratic loss issues I’ve mentioned.


You’ve guessed right that my optimizer steps were big, lowering it though improved the output but not good enough. So far scaling by a factor of 100 seems to perform better than everything else, though not good enough as well.
The main issue, is that my dataset is heavily imbalanced. For example if you consider that almost 90% or more of the values are laying within the range of 0-2000. It is normal that the network will learn to predict this range of values and would consider the rest in a less meaningful way. Weighting though I am not sure yet why it didn’t help, since in principle it should improve.
Another idea that came to my mind is whether it would be better (or it is possible) if I create a separate loss output for each cluster and then either combine them to an overall loss or somehow have a different regression branch for each cluster in the architecture. But I am not sure whether this would make any difference.