After receiving an image 500 foreground text 550 is segmented from a background portion by classifying each pixel as text or background. A segmentation prediction 520 is produced using an additional convolutional layer. A key vector representing features such as texture of the text may be generated by applying a cosine similarity, bias, and softmax to the prediction. A neural network refines the prediction using the key vector. The image may be encoded using a ResNet architecture to produce a feature map 510 and decoded 515 to produce the segmentation prediction. Decoding may comprise applying a convolutional layer to the feature map, a first bias to an output of the layer, and a softmax to the output of the first bias. An attention map 530 and combined feature map 535 may be produced by combining the feature map and key vector. The key vector is generated 525 using a cosine similarity function, a second softmax, a second bias, and a pooling layer. The neural network is trained by segmenting a set of training images and updating the network based on loss functions between a ground truth, initial segmentation prediction, and refined segmentation predictions.
Text refinement network