Hi Sally!



 Sally_Goldin:


Loss function -
I chose RMSE because the end users would like to be able to compare the output “error” from the model with the results from the classical machine vision approach.
…
Cross-entropy loss would be more difficult from them to interpret.



I agree with G. Philip’s comment.  It is perfectly reasonable to use one
loss function for training and one or more distinct loss-like functions
as evaluation metrics.  These evaluation metrics might be in some
substantive sense better metrics, or they might be technically not as
good but easier for non-specialists to interpret.  Either way, it’s common
practice to use multiple evaluation metrics.
It’s also true, as G. Philip notes, that metric X may improve more
rapidly and end up at a better level if you train with loss A than if
you trained directly with metric X.
In any event, I would strongly recommend that you try training
with cross entropy (or perhaps the Kullback-Leibler divergence,
a related measure of the discrepancy between two probability
distributions).  Doing so might not work better than training with
root-mean-squared-error (RMSE), but it certainly could, even
with RMSE as your evaluation metric.



Wouldn’t a network designed to process 1500x1500 images be enormous?



The network, in and of itself, would not be significantly larger, as it
would likely consist of upstream convolutional and downsampling
layers whose sizes are independent of the size of the input images.
However the memory and computation time required to train on
(and evaluate) the large images would be potentially enormous.


In any case, the particles are quite visible in a 224x224 image (see below).


The larger particles look distinct to me.  In the upper right of your
sample image, around two o’clock, the smaller particles – if that’s
what they are – look to my eye to be harder to distinguish.  It’s a
judgement call, but if it’s hard for your eye to see, you would be
making it harder on your network,  Perhaps something like 500x500
could provide a happy medium.


I’m also switching from color to gray scale. Do you think this is a bad idea?


My intuition is that gray scale should be fine.  Are different particles
of significantly different hues?  Are the particles significantly easier
to distinguish by eye in the color image?  If not, it doesn’t seem like
a gray-scale conversion would discard much relevant information.


(The images are normalized before training. This has not been.)


This is perfectly reasonable and possibly helpful.  If you do train
with normalization, be sure to evaluate and infer with similarly
normalized inputs.



for training, they had to draw polygons around every single particle in each training image!



Indeed!



My hardware resources are fairly modest, but the ML machine is more or less dedicated to that task, so I can start it up and let it run.



If you can perform a forward pass and backpropagation on a single
image with your hardware – typically an issue of gpu memory – you
should be able to train, at the cost of hours or days (or even weeks)
of clock / calendar time.  (If you can’t run a single sample through
your model, things become much more painful, and it wouldn’t bode
well for managing to carry out training in a reasonable amount of time.)
Good luck.
K. Frank