You should understand that with L1 loss (and non-quadaratic loss zones in general) you’re doing gradient descent based on error sign only, this may converge with help from lr scheduler and/or adaptive optimizer, but it is intrinsically harder, and big (compared to LR) initial errors  may be uncorrectable w/o downscaling.
Interesting log-transform results… I’d guess that your optimizer’s steps are too big, if you end up with 10^5 errors. This stuff with clusters may actually be bad, as gradients are scaled ~100-1000 times for your last clusters; this can mess up shared parameters and gradient variance estimates too much. Other possibilities include inappropriate LR & scheduler params or non-quadratic loss issues I’ve mentioned.
You can also try https://en.wikipedia.org/wiki/Box-Cox_transformation 1 to deal with exploding predictions, but that’s rather an auxiliary measure