  The Brazilian Supreme Court receives tens of thousands of cases each
semester. Court employees spend thousands of hours to execute the initial
analysis and classification of those cases -- which takes effort away from
posterior, more complex stages of the case management workflow. In this paper,
we explore multimodal classification of documents from Brazil's Supreme Court.
We train and evaluate our methods on a novel multimodal dataset of 6,510
lawsuits (339,478 pages) with manual annotation assigning each page to one of
six classes. Each lawsuit is an ordered sequence of pages, which are stored
both as an image and as a corresponding text extracted through optical
character recognition. We first train two unimodal classifiers: a ResNet
pre-trained on ImageNet is fine-tuned on the images, and a convolutional
network with filters of multiple kernel sizes is trained from scratch on
document texts. We use them as extractors of visual and textual features, which
are then combined through our proposed Fusion Module. Our Fusion Module can
handle missing textual or visual input by using learned embeddings for missing
data. Moreover, we experiment with bi-directional Long Short-Term Memory
(biLSTM) networks and linear-chain conditional random fields to model the
sequential nature of the pages. The multimodal approaches outperform both
textual and visual classifiers, especially when leveraging the sequential
nature of the pages.

Computation and Language