Ok, after playing a bit with almost all the proposals (weights, transformations e.g. log and scaling (dividing by a value), gradient accumulation of mini-batches) for modulating the gradient magnitudes per sample I am still facing more or less the same behavior, still the network cannot handle this discrepancy to the high range values.
I’ve tried to make a better visualization of the problem. I’ve split my data in 10 clusters, and this is how my whole testing dataset looks like:
image2560×1406 113 KB
and this is the amount of values per cluster:
       1      165197
       2      187126
       3      137637
       4       94646
       5       65813
       6       40318
       7       19070
       8        6263
       9         651
      10          79

as it can be noticed the majority of my values apply within the first more or less 7 clusters.
Then as you can see below my estimated values are capped more or less to the limit of the values of these 7 clusters:
image2560×1406 632 KB
end especially on the last cluster the values are underestimated, something that can be seen clearer in the following graph (green=overestimation, red=underestimation):
image2560×1406 70 KB
The above graphs made based on the L1 loss.
Now if I apply a log transform for example though it seems to be following the pattern better it is kind of amplified by a lot:
image2560×1406 50.6 KB
Scaling (dividing targets by 100) seems to work a bit better but still output values are underestimated especially in the last clusters with the high range values:
image2560×1406 81.2 KB
As I see it, the main problem is the big imbalance of points with high range values.