Hello Steve!



 stevew77:

…
I have a need to find a fairly short string of text ( 2-4 digits ) in a photo of a cows head ( its ID tag ) .
The text may be rotated slightly by up to +/-  20 degrees.
Then I need to extract the text from the photo.
…
Does anyone have a suggestion as to an approach for this please?
…


The general task you are asking about is called “object
detection.”  There is a lot of literature about it that discusses
a number of well-developed methods and algorithms.
Let me suggest one standard approach.  It isn’t necessarily
the “best,” but it is conceptually straightforward, and
relatively easy to implement.  This effective, but somewhat
old-school method (that isn’t necessarily the most efficient)
runs as follows:
First build a binary object classifier – that is, is this object
X or not? – and then apply your object classifier to windows
(of various sizes) that you slide across your image.
In more detail:
Manually annotate your training data.  That is, make a
set of scaled, centered, cropped images that contain
cow ids snipped out of your larger images.  These are
your “positive” – “yes, cow id” training images.  Also
make a similar set of scaled, cropped images snipped
out of your larger images – a cow’s tail, part of a tractor,
a bale of hay.  These are your “negative” – “no cow id”
training images.
These should be smallish images.  Depending on the typical
aspect ratio of a cow id, these might be, say, 32x16 pixel
images.  I doubt that you will need to orient (rotate to the
horizontal) your cow ids – 20 degrees shouldn’t be a problem
for training a classifier.
Now train a binary cow-id classifier.  (Since this is the
pytorch forum, we imagine that you train a simple
convolutional neural network – although a fully-connected
network would likely work fine – but you could also build
a non-neural-network classifier.)
To make your detector, slide windows of various sizes (and
possibly aspect ratios) across your image, and rescale the
contents of your window to the 32x16 size used to train your
classifier.  When a window slides over a cow id, your classifier
should return “yes”, and you have a “hit.”  When the window is
over something else, your classifier should return “no.”
Lastly, as your window slides across a cow id, you are
likely to get several hits for the same cow id.  (Windows
of different sizes are also likely to give you duplicate
hits.)  You want the best hit for any give cow id, so you
apply what is called “non-maximal suppression.”  For
this you want your classifier to provide not just a “yes”
or “no,” but also the “probability” or “confidence”
underlying the “yes” or “no.”  (Let’s say that your
network has a single output – a “logit” – that is fed
into BCEWithLogitsLoss when training your network.
You could use this logit as your confidence.)  For
non-maximal suppression, you look at the confidence
of each of the hits in a cluster of neighboring hits,
and keep only the hit with the largest confidence.
The location of the sliding window, together with its
size (and possibly aspect ratio) for this largest-
confidence hit is the bounding box of your detected
cow id that you can then feed to further processing.


I’m thinking I could maybe identify the shape of a cow, then zoom in on the cow head area so I’m in the general area where the text is, then find the text? How would I do that ( in broad terms? ) or is this a dumb idea?


This isn’t a dumb idea – you would be using substantive
“semantic” structure of your images, that is, cow ids appear
on cow heads, and not on bales of hay, etc.
I would image that unless text appears in lots of other places
in your images – license plates on tractors, cow id labels
stuck to the sides of barns – just detecting the cow ids
directly will be simpler and easier to implement.
(Even if your images contain a lot of other text, as long as
the cow-id text is somehow distinctive – a certain font, thick
black borders – you could still train just on the cow ids,
provided you make sure to include representative non-cow-id
text in your “not” training samples.)
Good luck.
K. Frank