We propose a deep and interpretable probabilistic generative model to analyze glyph shapes in printed Early Modern documents. We focus on clustering extracted glyph images into underlying templates in the presence of multiple confounding sources of variance. Our approach introduces a neural editor model that first generates well-understood printing phenomena like spatial perturbations from template parameters via interpertable latent variables, and then modifies the result by generating a non-interpretable latent vector responsible for inking variations, jitter, noise from the archiving process, and other unforeseen phenomena associated with Early Modern printing. Critically, by introducing an inference network whose input is restricted to the visual residual between the observation and the interpretably-modified template, we are able to control and isolate what the vector-valued latent variable captures. We show that our approach outperforms rigid interpretable clustering baselines (Ocular) and overly-flexible deep generative models (VAE) alike on the task of completely unsupervised discovery of typefaces in mixed-font documents. Scholars interested in understanding details related to production and provenance of historical documents rely on methods of analysis ranging from the study of orthographic differences and stylometrics, to visual analysis of layout, font, and printed characters. Recently developed tools like Ocular (Berg-Kirkpatrick et al., 2013) for OCR of historical documents have helped automate and scale some textual analysis methods for tasks like compositor attribution (Ryskina et al., 2017) and digitization of historical documents (Garrette et al., 2015). However, researchers often find the need to go beyond Figure 1: We desire a generative model that can be biased to cluster according to typeface characteristics (e.g. the length of the middle arm) rather than other more visually salient sources of variation like inking. textual analysis for establishing provenance of historical documents. For example, Hinman (1963)'s study of typesetting in Shakespeare's First Folio relied on the discovery of pieces of damaged or distinctive type through manual inspection of every glyph in the document. More recently, Warren et al. (2020) examine pieces of distinctive types across several printers of the early modern period to posit the identity of clandestine printers of John Milton's Areopagitica (1644). In such work, researchers frequently aim to determine whether a book was produced by a single or multiple printers (Weiss (1992); Malcolm (2014); Takano (2016)). Hence, in order to aid these visual methods of analyses, we propose here a novel probabilistic generative model for analyzing extracted images of individual printed characters in historical documents. We draw from work on both deep generative modeling and interpretable models of the printing press to develop an approach that is both flexible and controllable  the later being a critical requirement for such analysis tools. As depicted in Figure 1, we are interested in identifying clusters of subtly distinctive glyph shapes as these correspond to distinct metal stamps in the type-cases used by printers. However, other sources of variation (inking, for example, as depicted in Figure 1) are likely to dominate conventional clustering methods. For example, powerful models like the variational autoencoder (VAE) (Kingma and Welling, 2014) capture the more visually salient variance in inking rather than typeface, while more rigid models (e.g. the emission model of Ocular (Berg-Kirkpatrick et al., 2013)), fail to fit the data. The goal of our approach is to account for these confounding sources of variance, while isolating the variables pertinent to clustering. Hence, we propose a generative clustering model that introduces a neural editing process to add expressivity , but includes interpretable latent variables that model well-understood variance in the printing process: bi-axial translation, shear, and rotation of canonical type shapes. In order to make our model controllable and prevent deep latent variables from explaining all variance in the data, we introduce a restricted inference network. By only allowing the inference network to observe the visual residual of the observation after interpretable modifications have been applied, we bias the posterior approximation on the neural editor (and thus the model itself) to capture residual sources of variance in the editor  for example, inking levels, ink bleeds, and imaging noise. This approach is related to recently introduced neural editor models for text generation (Guu et al., 2018). In experiments, we compare our model with rigid interpretable models (Ocular) and powerful generative models (VAE) at the task of unsupervised clustering subtly distinct typeface in scanned images early modern documents sourced from Early English Books Online (EEBO). Our model reasons about the printed appearances of a symbol (say majuscule F ) in a document via a mixture model whose K components correspond to different metal stamps used by a printer for the document. During various stages of printing, random transformations result in varying printed manifestations of a metal cast on the paper. Figure 2 depicts our model. We denote an observed image of the extracted character by X . We denote choice of typeface by latent variable c (the mixture component) with prior  . We represent the shape of the k -th stamp by template T k , a square matrix of parameters. We denote the interpretable latent variables corresponding to spatial adjustment of Figure 2: Proposed generative model for clustering images of a symbol by typeface. the metal stamp by  , and the editor latent variable responsible for residual sources of variation by z . As illustrated in Fig. 2, after a cluster component c = k is selected, the corresponding template T k undergoes a transformation to yield  T k . This transformation occurs in two stages: first, the interpretable spatial adjustment variables (  ) produce an adjusted template (  2.1),  T k = warp ( T k ,  ) , and then the neural latent variable transforms the adjusted template (  2.2),  T k = filter (  T k , z ) . The marginal probability under our model is p ( X ) = (cid:88) k  k (cid:90) p ( X | , z ; T k ) p (  ) p ( z ) dzd, where p ( X | , z ; T k ) refers to the distribution over the binary pixels of X where each pixel has a bernoulli distribution parametrized by the value of the corresponding pixel-entry in  T k . Early typesetting was noisy, and the metal pieces were often arranged with slight variations which resulted in the printed characters being positioned with small amounts of offset, rotation and shear. These real-valued spatial adjustment variables are denoted by  = ( r, o, s, a ) , where r represents the rotation variable, o = ( o h , o v ) represents offsets along the horizontal and vertical axes, s = ( s h , s v ) denotes shear along the two axes. A scale fac-tor,  a = 1 . 0 + a , accounts for minor scale variations arising due to the archiving and extraction processes. All variables in  are generated from a Gaussian prior with zero mean and fixed variance as the transformations due to these variables tend to be subtle. In order to incorporate these deterministic transformations in a differentiable manner, we map  to a template sized attention map H ij for each output pixel position ( i, j ) in  T as depicted in Figure 3. The attention map for each output pixel is formed in order to attend to the corresponding shifted (or scaled or sheared) portion of the input template and is shaped according to a Gaussian distribution with mean determined by an affine transform. This approach allows for strong inductive bias which contrasts with related work on spatial-VAE (Bepler et al., 2019) that learns arbitrary transformations. Apart from spatial perturbations, other major sources of deviation in early printing include random inking perturbations caused by inconsistent application of the stamps, unpredictable ink bleeds, and noise associated with digital archiving of the documents. Unlike in the case of spatial perturbations which could be handled by deterministic affine transformation operators, it is not possible to analytically define a transformation operator due to these variables. Hence we propose to introduce a non-interpretable real-valued latent vector z , with a Gaussian prior N ( 0 , I ) , that transforms  T into a final template  T via neurally-parametrized function filter (  T , z ;  ) with neural network parameters  . This function is a convolution over  T whose kernel is parametrized by z , followed by non-linear operations. Intuitively, parametrizing the filter by z results in the latent variable accounting for variations like inking appropriately because convolution filters capture local variations in appearance. Srivatsan et al. (2019) also observed the effectiveness of using z to define a deconvolutional kernel for c X Observation Template choice  T c Warped template R c Residual z Inference parameters  Posterior Approximation q ( z | X , c ;  ) R c = X   T c z = InferNet ( R c ,c ) Figure 4: Inference network for z conditions on the mixture component and only the residual image left after subtracting the  -transformed template from the image. Our aim is to maximize the log likelihood of the observed data ( { X d | d  N , d < n } ) of n images wrt. model parameters: LL ( T 1 ,...,k ,  ) = max T, (cid:88) d log (cid:104) (cid:88) k  k (cid:90) p ( X d |  d , z d ; T k ,  ) p (  d ) p ( z d ) dz d d d (cid:105) During training, we maximize the likelihood wrt. instead of marginalizing, which is an approximation inspired by iterated conditional modes (Besag, 1986): max T, (cid:88) d log (cid:88) k max  k,d  k (cid:90) p ( X d |  d =  k,d , z d ; T k ,  ) p (  d =  k,d ) p ( z d ) dz d However, marginalizing over z remains intractable. Therefore we perform amortized variational inference to define and maximize a lower bound on the above objective (Kingma and Welling, 2014). We use a convolutional inference neural network parametrized by  (Fig. 4), that takes as input, the mixture component k , the residual image R k = X   T k , and produces mean and variance parameters for an isotropic gaussian proposal distribution q ( z | R k , k ;  ) . This results in the final training objective: max T,, (cid:88) d log (cid:88) k E q ( z d | R d,k ,k ;  ) (cid:2) max  k,d (cid:0)  k p ( X d |  =  k,d , z d ; T k ,  ) p (  =  k,d ) (cid:1)(cid:3)  KL (cid:0) q ( z d | R d,k , k ;  ) || p ( z ) (cid:1) We use stochastic gradient ascent to maximize this objective with respect to T, ,  and  . We train our models on printed occurrences of 10 different uppercase character classes that scholars have found useful for bibliographic analysis (Warren et al., 2020) because of their distinctiveness. As a preprocessing step, we ran Ocular (Berg-Kirkpatrick et al., 2013) on the grayscale scanned images of historical books in EEBO dataset and extracted the estimated image segments for the letters of interest. We show that our model is superior to strong baselines at clustering subtly distinct typefaces (using realistic synthetic data), as well as in terms of fit-ting the real data from historical books. Ocular: Based on the emission model of Ocular that uses discrete latent variables for the ver-tical/horizontal offset and inking variables, and hence has limited expressivity. -only: This model only has the interpretable continuous latent variables pertaining to spatial adjustment. VAE-only: This model is expressive but doesn't have any interpretable latent variables for explicit control. It is an extension of Kingma et al. (2014)'s model for semi-supervised learning with a continuous latent variable vector in which we obtain tighter bounds by marginalizing over the cluster identities explicitly. For fair comparison, the encoder and decoder convolutional architectures are the same as the ones in our full model. The corresponding training objective for this baseline is: max T,, (cid:88) d log (cid:88) k E q ( z d | X d ,k ;  ) (cid:2)  k p ( X d | z d ; T k ,  ) (cid:3)  KL (cid:0) q ( z d | X d , k ;  ) || p ( z ) (cid:1) No-residual: The only difference from the full model is that the encoder for the inference network conditions the variational distribution q ( z ) on the entire input image X instead of just the residual image X   T . Early modern books were frequently composed from two or more type cases, resulting in documents with mixed fonts. We aim to learn the dif-V-measure Mutual Info F&M NLL Ocular 0.42 0.45 0.61 379.21  -only 0.49 0.51 0.70 322.04 VAE-only 0.22 0.29 0.38 263.45 No-residual 0.54 0.58 0.73 264.27 Our Model 0.73 0.74 0.85 257.92 Table 1: ferent shapes of metal stamps that were used as templates for each cluster component in our model. Data: In order to quantitatively evaluate our model's performance, we experiment with synthetically generated realistic dataset for which we know the ground truth cluster identities in the following manner: For each character of interest, we pick three distinct images from scanned segmented EEBO images, corresponding to three different metal casts. Then we randomly add spatial perur-bations related to scale, offset, rotation and shear. To incorporate varying inking levels and other distortions, we randomly either perform erosion , dilation , or a combination of these warpings using OpenCV (Bradski, 2000) with randomly selected kernel sizes. Finally, we add a small Gaussian noise to the pixel intensities and generate 300 perturbed examples per character class. Results: We report macro-averaged results across all the character classes on three different clustering measures, V-measure (Rosenberg and Hirschberg, 2007), Mutual Information and Fowlkes and Mallows Index (Fowlkes and Mallows, 1983). In Table 1, we see that our model significantly outperforms all other baselines on every metric. Ocular and  -only models fail because they lack expressiveness to explain the variations due to random jitters, erosions and dilations. The VAE-only model, while very expressive, performs poorly because it lacks the inductive bias needed for successful clustering. The No-residual model performs decently but our model's superior performance emphasizes the importance of designing a restrictive inference network such that z only focuses on extraneous sources of variation. For the analysis of real books, we selected three books from the EEBO dataset printed by different printers. We modeled each character class for each book separately and report the macro-aggregated upper bounds on the negative log likelihood (NLL) in Table 1. We observe that adding a small amount of expressiveness makes our  -only model better than Ocular . The upper bounds of other inference network based models are much better than the tight 1 bounds of both the interpretable models. Our model has the lowest upper bound of all the models while retaining interpretability and control. We provide visual evidence of desirable behavior of our model on collections of character extractions from historical books with mixed fonts. Specifically, we discus the performance of our model on the mysterious edition of Thomas Hobbes' Leviathan known as the 25 Ornaments edition. (Hobbes, 1651 [really 1700?]). The 25 Ornaments Leviathan is an interesting test case for several reasons. While its title page indicates a publisher and year of publication, both are fabricated (Mal-colm, 2014). The identities of its printer(s) remain speculative, and the actual year of publication is uncertain. Further, the 25 Ornaments exhibits two distinct fonts. Our model is successful in discovering distinctly shaped typefaces in the 25 Ornaments Leviathan . We focus on the case study of majuscule letters F and R , each of which have two different typefaces mixed in throughout. The two typefaces for F differ in the length of the middle arm (Fig. 1), and the two typefaces for R have differently shaped legs . In Fig. 5, we show that our model successfully learns the two desired templates T 1 and T 2 for both the characters which indicates that the clusters in our 1 For Ocular and  -only models, we report the upper bound obtained via maximization over the interpretable latent variables. Intuitively, these latent variables are likely to have unimodal posterior distributions with low variance, hence this approximation is likely tight. model mainly focus on subtle differences in underlying glyph shapes. We also illustrate how the latent variables transform the model templates T to  T for four example F images. The model learns complex functions to transform the templates which go beyond simple affine and morphological transformations in order to account for inking differences, random jitter, contrast variations etc. 3.2.2 Interpretable variables (  ) and Control 1 2 3 Avg. Finally, we visualize the ability of our model to separate responsibility of modelling variation among the interpretable and non-interpretable variables appropriately. We use the inferred values of the interpretable (  ) variable for each image in the dataset to adjust the corresponding image. Since the templates represent the canonical shape of the letters, the  variables which shift the templates to explain the images can be reverse applied to the input images themselves in order to align them by accounting for offset, rotation, shear and minor size variations. In Fig. 6, we see that the input images (top row) are uneven and vary by size and orientation. By reverse applying the inferred  values, we are able to project the images to a fixed size such that they are aligned and any remaining variations in the data are caused by other sources of variation. Moreover, this alignment method would be crucial for automating certain aspects of bibliographic studies that focus on comparing specific imprints. Beyond applications to typeface clustering, the general approach we take might apply more broadly to other clustering problems, and the model we developed might be incorporated into OCR models for historical text. This project is funded in part by the NSF under grants 1618044 and 1936155, and by the NEH under grant HAA256044-17.
objective method objective method result abstain abstain result abstain abstain abstain abstain objective objective method abstain abstain abstain objective objective abstain result abstain method method abstain abstain method abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain other abstain method other result abstain result abstain abstain abstain result abstain other abstain abstain abstain abstain abstain abstain abstain result result abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain abstain objective other