  This paper proposes a technique for training a neural network by minimizing a
surrogate loss that approximates the target evaluation metric, which may be
non-differentiable. The surrogate is learned via a deep embedding where the
Euclidean distance between the prediction and the ground truth corresponds to
the value of the evaluation metric. The effectiveness of the proposed technique
is demonstrated in a post-tuning setup, where a trained model is tuned using
the learned surrogate. Without a significant computational overhead and any
bells and whistles, improvements are demonstrated on challenging and practical
tasks of scene-text recognition and detection. In the recognition task, the
model is tuned using a surrogate approximating the edit distance metric and
achieves up to $39\%$ relative improvement in the total edit distance. In the
detection task, the surrogate approximates the intersection over union metric
for rotated bounding boxes and yields up to $4.25\%$ relative improvement in
the $F_{1}$ score.

Computer Vision and Pattern Recognition