Hi Sally!



 Sally_Goldin:

I have images of gravel with a mix of particle sizes.


May I ask some clarifying questions about your use case?
Leaving aside your current network and the kind of annotations you
have for your data, ideally, what would you like the output of your
network to be?  Would you like the full probability distribution of
particle sizes – that is, roughly speaking, a histogram of particle sizes
with lots of bins?
Is it important that you use neural networks / machine learning for this
task, or would you be open to using other computer-vision techniques?


My output is a 5 element vector of values that represent the relative proportion of particles in different size categories.


What, in more detail, is the meaning of these five numbers?  Are your
ground-truth annotations analogous length-5 vectors that have the
same meaning?
What loss function are you using to compare your network outputs
to your annotations?
What does the final layer of your network look like – that is what is
the layer that produces this length-5 output?


if the input image more or less matches one of the particle size distributions in the training data set. However, if I test an image with a different distribution than any the network has seen during training, the results are typically poor.


Are the test images that your network does poorly on in any way
different in character than the images you used to train your network,
or are they basically the same?  For example, did you randomly
select some test images from the same larger annotated set from
which you selected your training set?
In general, if you are getting significantly worse results on test images
that are of the same character as you training images, you are seeing
signs of overfitting.  That is, your network has been trained enough
times on the same images that in addition to “learning” the properties
you want it to learn, it is also learning recognize individual images
by their individual peculiarities.  Having recognized a specific image,
the network now “parrots back” the “right” answer.
The “rigorous” approach is to split your annotated data set into three
pieces (usually randomly) – a training set, a validation set, and a
test set.  You use your training set to train with backpropagation and
optimizer parameter updates.  At times of your choosing (for example,
once every epoch), you calculate your loss and potentially other
evaluation metrics for your validation set, and use this information to
decide when to stop, perhaps because the results are good enough,
or training has slowed down, or overfitting has set in.
Only after you have settled on your “final” trained network, do you
apply it to your test set to see how well it actually performs.  (The
purist notion of rigor here is that using your validation set to evaluate
the performance of your final network would constitute a “data leak”
because you did use the validation set as part of your training process.)


With data augmentation, I currently have about 22,000 input images


How many unaugmented annotated images do you have?  How big
are the images, and are different regions of a given images  relatively
independent so that it might be reasonable to consider different regions
of a single image as independent data samples?


in 25 different distribution groups.


What is a “distribution group?”  What is the relationship of a distribution
group and the 5-element output of your network?
Related to the question of whether your ground-truth annotations have
the same structure as your network output, what is the relationship of
a distribution group to your annotations?


I’ve considered transfer learning. For instance, some papers have claimed that ResNet learns many texture features
…
Can I somehow adapt it to my more general objectives?


My gut feeling is that ResNet is unlikely to be helpful for you (but I
could well be wrong).
My intuition – not based on any real experience or evidence – about
transfer learning goes something like this:
The upstream layers of something like ResNet “learn” low-level
“features” – things like edges and arcs and maybe blobs.  In the
middle these get aggregated together into mid-level features like
nose-like blobs and eye-like blobs.  In the final, down-stream layers
they get aggregated into faces, cat-like faces, dog-like faces, and
ultimately, into cat-vs-dog predictions.
As you say, upstream ResNet layers recognize “textures” so it’s
plausible that pre-trained, upstream ResNet layers could be a useful
starting point for your training.  But your ultimate task (as I understand
it) is really quite different than what ResNet was trained for, so I’m not
convinced that you would get that much benefit using it as a starting
point.
Best.
K. Frank