Hello, KFrank,
Thank you for your detailed and thoughtful reply.

Leaving aside your current network and the kind of annotations you
have for your data, ideally, what would you like the output of your
network to be?  Would you like the full probability distribution of
particle sizes – that is, roughly speaking, a histogram of particle sizes
with lots of bins?

Five bins is standard for this application. But yes, I’m basically looking for a histogram, aggregated to these five bins and normalized so the total of all bins is 1.0.

Is it important that you use neural networks / machine learning for this
task, or would you be open to using other computer-vision techniques?

We’ve actually tried some analytical machine vision to identify individual particles. Results were quite poor. I am hoping ML can do better.


My output is a 5 element vector of values that represent the relative proportion of particles in different size categories.



What, in more detail, is the meaning of these five numbers?  Are your
ground-truth annotations analogous length-5 vectors that have the
same meaning?

Yes, ground truth labels mean the same thing. These images were manually acquired from lab samples prepared to have different distributions.

What loss function are you using to compare your network outputs
to your annotations?

RMSE

What does the final layer of your network look like – that is what is
the layer that produces this length-5 output?

nn.linear(32,5), followed by softmax to normalize to 0-1 range.

Are the test images that your network does poorly on in any way
different in character than the images you used to train your network,
or are they basically the same?  For example, did you randomly
select some test images from the same larger annotated set from
which you selected your training set?

I have randomly selected 20% of the images as validation images. These do reasonably well though the average loss is higher than for the training images (as I’d expect). I then tried other images which have the same type of annotation but which are not from the original data set. Our goal is transfer to other images.

In general, if you are getting significantly worse results on test images
that are of the same character as you training images, you are seeing
signs of overfitting.  That is, your network has been trained enough
times on the same images that in addition to “learning” the properties
you want it to learn, it is also learning recognize individual images
by their individual peculiarities.  Having recognized a specific image,
the network now “parrots back” the “right” answer.

Yes, I have come to that conclusion also. However, I realized that even without the overfitting issue I see no sign of the generalization that I am trying for.

The “rigorous” approach is to split your annotated data set into three
pieces (usually randomly) – a training set, a validation set, and a
test set.  You use your training set to train with backpropagation and
optimizer parameter updates.  At times of your choosing (for example,
once every epoch), you calculate your loss and potentially other
evaluation metrics for your validation set, and use this information to
decide when to stop, perhaps because the results are good enough,
or training has slowed down, or overfitting has set in.
Only after you have settled on your “final” trained network, do you
apply it to your test set to see how well it actually performs.  (The
purist notion of rigor here is that using your validation set to evaluate
the performance of your final network would constitute a “data leak”
because you did use the validation set as part of your training process.)

Yes, of course… but I’m in exploration mode at the moment. I’ve actually trained several different networks with variations in input image size and augmentation techniques. The results vary somewhat but the pattern is more or less the same.


With data augmentation, I currently have about 22,000 input images



How many unaugmented annotated images do you have?  How big
are the images, and are different regions of a given images  relatively
independent so that it might be reasonable to consider different regions
of a single image as independent data samples?

About 2600 different images. They are about 1500x1500 to start. My latest attempt downsamples them to 224x224 (earlier versions used 64x64 and 128x128).
The entire image area is relevant to the answer, because the particles are not uniformly distributed in the image area. One of my earlier attempts used subsetting for augmentation, but I realized the subsets might in fact reflect a significantly different distribution.

What is a “distribution group?”  What is the relationship of a distribution
group and the 5-element output of your network?

As noted above, these images were synthetically created by preparing a gravel sample with a particular distribution of particle sizes, spreading it in a frame, and taking a series of photos. There were 25 different samples prepared, each one with a different distribution of particle sizes. For each distribution, the students took about 1000 images, from different locations. After a few images, they would “disturb” the gravel, moving it around to change the patterns, but not the underlying distribution.
What I mean to convey with the term “distribution group” is a set of images all of which have the same 5-element histogram.

As you say, upstream ResNet layers recognize “textures” so it’s
plausible that pre-trained, upstream ResNet layers could be a useful
starting point for your training.  But your ultimate task (as I understand
it) is really quite different than what ResNet was trained for, so I’m not
convinced that you would get that much benefit using it as a starting
point.

That is more or less the conclusion I’ve come to. I’ve been looking for papers about using texture in ML. However, pretty much every one is considering a classification task – rather than a “characterization” or “description” task.
Any suggestions would be welcome! (Including your thoughts about non-ML methods.)
Thank you!