It doesn’t seem to converge. Indeed, starting with a value around 1.5, then when it’s approaching 1 it just remains in the same value range, and optim.lr_scheduler.ReduceLROnPlateau doesn’t seem to help. It’s still a long way to go of course, as this could be due to several reasons, and not the loss function. To sum up what I use as of now:

VGG-16 with batch_norm pre-trained weights. I only train the classifier part.
Adam and SGD optimizers with several learning rates, to name a few: 0.0001, 0.0003, 0.001, 0.003.
Loss function is the one you suggested.
Dataset consists of 12k images, splitted randomly with a percent of .9 for training examples.

I’m wondering if after all the problem may be the sparsity of the artificial images I’m creating, or the architecture itself. My images have dimensions (3, 224, 224) and number of points on those images ranges from 1 to 8 for now. Each image represents an area of measurements (in reality each combination of particles exist in a 12x12cm area, thus do the math and you have the resolution you can get with 224x224 pixels). That means that in the image 5x5 area of pixels * number of points is colored (what each color represent doesn’t matter for now) and the rest is black. I.e. if I have 10 particles then 10 * (5 ** 2) pixels are different than black, the rest is black.