I am working on a regression problem, where I want to modify the loss function so that to address a set of data which has outliers with high importance. This characteristic can be seen in the following images of two sample cases (on the left the raw values and on the right the sorted ones):
image695×535 114 KB
So as you can see, usually the majority of the values are within a specific lower range and then I have a couple ones with a much higher intensity.
I’ve tried quite a few different common losses, e.g. MAE, MSE, RMSE, normal/reverse Huber loss which are failing to model the high range values since as I guess they consider them as outliers. Thus, I’ve tried to create a custom loss where for the low error it uses MAE and for the higher error MSE (which is know to be sensitive to outliers):
class CustomLoss(torch.nn.Module):
    def __init__(self):
        super(CustomLoss, self).__init__()

    def forward(self, predicted, true):
        errors = torch.abs(predicted - true)
        mask = errors > 250
        return torch.mean((0.5 * mask * (errors ** 2)) + ~mask * errors)

However, this one still did not help much. More or less in all cases I am getting a poor prediction at the high range values (a bit better with MSE).
Thus I would like to ask whether someone has any idea how I could modify my loss so that it can address these high range values without considering them as outliers.
Thanks.