googlebot:

Huber loss or some similar step balancing method may still be necessary, but yea, I said “MLP” but meant deep networks in general; and more complex architectures are more prone to introduce bias.


I was also trying the reverse huber loss (go to the loss function section) which seems to perform quite nicely so far, except of course the high range values.



 googlebot:

Note that you can do gradient accumulation (i.e. do backward() without optimizer.step()), to increase effective batch size. Scheduled increasing may be optimal for faster convergence.


That’s actually a good point 



 googlebot:

So, log transform is the simplest correcting (and variance stabilizing) transformation for positive values, not necessarily the best one though…


I didn’t understand that part clearly, where and when I should apply this transformation. You also suggested it in one of your previous messages. Should I apply this before I get the error between my targets and the regressed values in my loss function or elsewhere