summaries: This paper introduces the compression risk in domain-invariant representations. Learning domain-invariant representations leads to larger compression risks and potentially worse adaptability. To this end, the authors presents gamma(H) to measure the compression risk. Learning weighted representations to control source error, domain discrepancy, and compression simultaneously leads to a better tradeoff between invariance and compression, which is verified by experimental results.

summaries: This paper presents a revisit of existing theoretical frameworks in unsupervised domain adaptation in the context of learning invariant representation. They propose a novel bound that involves trainable terms taking into account some compression information and a novel interpretation of adaptability. The authors mention also contribution showing that weighting representations can be a way to improve the analysis.

summaries: This submission provides a new theoretical framework for domain adaptation. In order to tackle the adaptability term in the classical domain adaptation theory, this submission proposes a new upper bound that enlarge the hypothesis space in the adaptability term. A weighted version of this theory is also given. Authors further support their conclusion by empirical results.

abstractText: Learning Invariant Representations to adapt deep classifiers of a source domain to a new target domain has recently attracted much attention. In this paper, we show that the search for invariance favors the compression of representations. We point out this may have a bad impact on adaptability of representations expressed as a minimal combined domain error. By considering the risk of compression, we show that weighting representations can align representation distributions without impacting their adaptability. This supports the claim that representation invariance is too strict a constraint. First, we introduce a new bound on the target risk that reveals a trade-off between compression and invariance of learned representations. More precisely, our results show that the adaptability of a representation can be better controlled when the compression risk is taken into account. In contrast, preserving adaptability may overestimate the risk of compression that makes the bound impracticable. We support these statements with a theoretical analysis illustrated on a standard domain adaptation benchmark. Second, we show that learning weighted representations plays a key role in relaxing the constraint of invariance and then preserving the risk of compression. Taking advantage of this trade-off may open up promising directions for the design of new adaptation methods.

authors:
Dario Amodei
Chris Olah
Jacob Steinhardt
Paul Christiano
John Schulman
Dan Mané

title: Concrete problems in ai safety

venue: arXiv preprint arXiv:1606.06565,

authors:
Martin Arjovsky
Léon Bottou
Ishaan Gulrajani
David Lopez-Paz

title: Invariant risk minimization

venue: arXiv preprint arXiv:1907.02893,

authors:
Shai Ben-David
John Blitzer
Koby Crammer
Fernando Pereira

title: Analysis of representations for domain adaptation

venue: In Advances in neural information processing systems,

authors:
Shai Ben-David
John Blitzer
Koby Crammer
Alex Kulesza
Fernando Pereira
Jennifer Wortman Vaughan

title: A theory of learning from different domains

venue: Machine learning,

authors:
Konstantinos Bousmalis
George Trigeorgis
Nathan Silberman
Dilip Krishnan
Dumitru Erhan

title: Domain separation networks

authors:
Corinna Cortes
Yishay Mansour
Mehryar Mohri

title: Learning bounds for importance weighting

authors:
Nicolas Courty
Rémi Flamary
Devis Tuia
Alain Rakotomamonjy

title: Optimal transport for domain adaptation

venue: IEEE transactions on pattern analysis and machine intelligence,

authors:
Nicolas Courty
Rémi Flamary
Amaury Habrard
Alain Rakotomamonjy

title: Joint distribution optimal transportation for domain adaptation

venue: In Advances in Neural Information Processing Systems,

authors:
Yaroslav Ganin
Victor Lempitsky

title: Unsupervised domain adaptation by backpropagation

venue: In International Conference on Machine Learning,

authors:
Judy Hoffman
Eric Tzeng
Taesung Park
Jun-Yan Zhu
Phillip Isola
Kate Saenko
Alexei Efros
Trevor Darrell

title: Cycada: Cycle-consistent adversarial domain adaptation

authors:
Jiayuan Huang
Arthur Gretton
Karsten Borgwardt
Bernhard Schölkopf
Alex J Smola

title: Correcting sample selection bias by unlabeled data

authors:
Fredrik Johansson
David Sontag
Rajesh Ranganath

title: Support and invertibility in domaininvariant representations

venue: In The 22nd International Conference on Artificial Intelligence and Statistics,

authors:
Wouter M Kouw
Marco Loog

title: A review of single-source unsupervised domain adaptation

venue: arXiv preprint arXiv:1901.05335,

authors:
Meelis Kull
Peter Flach

title: Patterns of dataset shift

venue: In First International Workshop on Learning over Multiple Contexts (LMCE) at ECML-PKDD,

authors:
Yann LeCun
Léon Bottou
Yoshua Bengio
Patrick Haffner

title: Gradient-based learning applied to document recognition

venue: Proceedings of the IEEE,

authors:
Hong Liu
Mingsheng Long
Jianmin Wang
Michael Jordan

title: Transferable adversarial training: A general approach to adapting deep classifiers

authors:
Mingsheng Long
Yue Cao
Jianmin Wang
Michael I Jordan

title: Learning transferable features with deep adaptation networks

venue: In Proceedings of the 32nd International Conference on International Conference on Machine Learning-Volume

authors:
Mingsheng Long
Han Zhu
Jianmin Wang
Michael I Jordan

title: Unsupervised domain adaptation with residual transfer networks

authors:
Mingsheng Long
Zhangjie Cao
Jianmin Wang
Michael I Jordan

title: Conditional adversarial domain adaptation

authors:
Yishay Mansour
Mehryar Mohri
Afshin Rostamizadeh

title: Domain adaptation: Learning bounds and algorithms

venue: In 22nd Conference on Learning

authors:
Sinno Jialin Pan
Qiang Yang

title: A survey on transfer learning

venue: IEEE Transactions on knowledge and data engineering,

authors:
Joaquin Quionero-Candela
Masashi Sugiyama
Anton Schwaighofer
Neil D Lawrence

title: Dataset shift in machine learning

authors:
Swami Sankaranarayanan
Yogesh Balaji
Carlos D Castillo
Rama Chellappa

title: Generate to adapt: Aligning domains using generative adversarial networks

venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,

authors:
Jian Shen
Yanru Qu
Weinan Zhang
Yong Yu

title: Wasserstein distance guided representation learning for domain adaptation

venue: In Thirty-Second AAAI Conference on Artificial Intelligence,

title: Improving predictive inference under covariate shift by weighting the loglikelihood function

venue: Journal of statistical planning and inference,

authors:
Masashi Sugiyama
Shinichi Nakajima
Hisashi Kashima
Paul V Buenau
Motoaki Kawanabe

title: Direct importance estimation with model selection and its application to covariate shift adaptation

authors:
Eric Tzeng
Judy Hoffman
Ning Zhang
Kate Saenko
Trevor Darrell

title: Deep domain confusion: Maximizing for domain invariance

venue: arXiv preprint arXiv:1412.3474,

authors:
Yifan Wu
Ezra Winston
Divyansh Kaushik
Zachary Lipton

title: Domain adaptation with asymmetrically-relaxed distribution alignment

authors:
Han Zhao
Remi Tachet Des Combes
Kun Zhang
Geoffrey Gordon

title: On learning invariant representations for domain adaptation

heading: 1 INTRODUCTION

text:
In a standard Supervised Learning setup, we consider that training data is representative of testing data. This ideal framework is often violated in practical applications since the data generative process can be altered during the application phase e.g. different pre-processing, sample rejection, imbalanced classes, condition of data collection, time evolving data... This so-called Distributional Shift situation (Quionero-Candela et al. (2009); Kull & Flach (2014)) is recognized as a major challenge in Machine Learning (Amodei et al. (2016)). Domain Adaptation (DA) (Pan & Yang (2009)) is a widely adopted strategy to mitigate its effects. It consists in leveraging unlabeled testing data (the target domain) in order to transfer knowledge learned on labeled training data (the source domain).
A first line of study, named Importance Sampling (IS), assumes some stationarity in distributions in order to approximate the risk in the target domain by weighting sample contribution (Shimodaira (2000); Huang et al. (2007)). To be applied, IS needs statistical support sharing across domains limiting its range of application when addressing high dimensional data. In that challenging context, Ganin & Lempitsky (2015) have shown that deep classifiers can be adapted to a new target domain by learning domain Invariant Representations (IR) and find theoretical support in the work of Ben-David et al. (2007; 2010). It assumes that a classifier trained on representations of labeled data sampled from both domains will achieve a negligible combined error (adaptability). However, enforcing distribution invariance in the representation space can badly impact its adaptability, for instance in the context of Target Shift (Zhao et al. (2019)) or when the support of the distributions do not overlap enough (Johansson et al. (2019)). In the most general case of DA, it is not clear how IR helps aligning domains and removing spurious correlations as pointed by Arjovsky et al. (2019).
In the present work, we show that the search of invariance favors learning representations with a significant loss of information from the original features. We refer to this phenomenon as the risk of Compression. In our understanding, this risk is not addressed in the original work from Ben-David et al. (2007; 2010). We demonstrate that taking into account this risk has two major advantages. First, it provides a better control on (non-trainable) terms involving the labels in the
target domain. Second, it introduces a new trainable term which embodies a risk of Compression and then may open-up the path to new learning methods. We provide a theoretical analysis illustrated with experiments on a standard Domain Adaptation benchmark. We underline that a better control on non-trainable terms may lead to an over-pessimistic risk of compression. This brings to light an inherent trade-off when learning representations for domain adaptation: How much compression risk is acceptable in order to obtain invariance? As a first try to answer to this question, we point that Learning Weighted Representations plays a key role in aligning representation distributions in order to obtain invariance. More precisely, we show that under a constraint of strong source risk conservation, weighting representations does not impact its adaptability.

heading: 2 BACKGROUND

heading: 2.1 A DECADE OF DOMAIN ADAPTATION

text:
The classical setup for modelling distributional shift is to introduce two domains: the source domain i.e. dataset where the model is trained with supervision and the target domain i.e. where the model is tested or applied. Formally, for two random variables (X,Y ) of a given space X × Y , we introduce two distributions on X × Y: the source distribution Ps(X,Y ) and the target distribution Pt(X,Y ). We use the exponent notation s and t to differentiate source and target terms. We define the hypothesis space H as a subset of functions from X to Y . The distributional shift situation is then characterized by the shift: Ps(X,Y ) 6= Pt(X,Y ). Learning under Distributional Shift (Quionero-Candela et al. (2009)) consists in minimizing the risk on the target domain εt(h) = Et[`(h(X), Y )] for a given loss ` : Y × Y → R+ and an hypothesis function h ∈ H. Assuming that w(X,Y ) = Pt(X,Y )/Ps(X,Y ) is tractable, it is straightforward to show that:
εt(h) = Es[w(X,Y )`(h(X), Y )] (1)
Roughly speaking, this equation shows that it is possible to minimize the target risk using data (xsi , y s i )1≤i≤ns sampled from the source domain (i.e. (xi, yi) ∼ Ps(X,Y )) while re-weighting their contribution in the source expectation with a factor w(xi, yi) (Importance Sampling). The difficulty of Learning under distributional shift arises when w(xi, yi) is not tractable. The first reason is that data sampled from Pt(X,Y ) is often not easily available. In order to get a reasonable estimation of w, practitioners are used to reformulate the problem as Unsupervised Domain Adaptation. Definition 1 (Unsupervised Domain Adaptation). Given a loss `, an hypothesis class H, a source domain (X ×Y,Ps(X,Y )) and a target domain (X ×Y,Pt(X,Y )), Unsupervised Domain Adaptation consists in minimizing the target risk (i.e. finding h? = arg minh∈H Et[`(h(X), Y )]) with finite labeled sampling of (xsi , ysi ) ∼ Ps(X,Y ) of the source domain and a finite unlabeled sampling (xtj) ∼ Pt(X) of the target domain.
If unlabeled samples in the target domain are available, a first idea is then to approximate w(X,Y ) by w(X) = Pt(X)/Ps(X). This is equivalent to assuming that conditional distributions are conserved across domains (i.e. Ps(Y |X) = Pt(Y |X)), a situation known as Covariate Shift, that has already been the subject of an extensive literature (Sugiyama et al. (2008); Huang et al. (2007)). This formulation becomes ill-posed when feature domains (X ,Ps(X)) and (X ,Pt(X)) suffer from non-overlapping support: w(X) is then undefined. To address this issue, Invariant Representations learning (Long et al. (2015)) performs adaptation by looking for a representation Z = ϕ(X) whose distribution is conserved across domains Ps(Z) = Pt(Z). A common strategy consists in extracting features Z to learn labels Y in the source domain while controlling representation invariance. That is formulated as follows: Learning Objective 1 (Learning Invariant Representations for Domain Adaptation).
g?, ϕ? = arg min (g,ϕ)∈G×Φ (1− α) · εs(g ◦ ϕ) + α · D(s,t)(ϕ) (2)
where G is the set of classifiers and Φ is the set of representations. The hypothesis space is then defined as H = {g ◦ ϕ : g ∈ G, ϕ ∈ Φ}. D(s,t)(ϕ) is a distance between distributions Ps(Z) and Pt(Z) and α is a trade-off parameter.
The choice of D(s,t)(ϕ) has been the object of an extensive literature including f− divergence measures such as domain adversarial learning (Ganin & Lempitsky (2015); Tzeng et al. (2014); Long
et al. (2018)), Integral Probability Measures such as Maximum Mean Discrepancy (Long et al. (2015; 2016)) or Optimal Transport (Shen et al. (2018)). See Kouw & Loog (2019) for an extensive review. Interestingly, learning invariant representations to perform adaptation finds theoretical support from the work of Ben-David et al. (2007; 2010) that we recall in the next section.

heading: 2.2 THEORETICAL GUARANTEES OF DOMAIN ADAPTATION

text:
We consider the set of representations Φ and classifiers G. The hypothesis space is denoted H = {g ◦ ϕ : g ∈ G, ϕ ∈ Φ}. We introduce the loss ` : Y × Y → R+ which is assumed symmetric and verifies the triangular inequality. We note the risk ε(h) = E[`(Y, h(X))] for h ∈ H and ε(h, h′) = E[`(h(X), h′(X))] for (h, h′) ∈ H2. To emphasize the role of a representation Z = ϕ(X), we introduce f̃(Z) = E[Y |Z] and H̃ = {g ◦ϕ : g ∈ G} ⊂ H. We underline that ·̃ quantities depend on ϕ where, for the ease of reading, this dependence is omitted in notations when it is not ambiguous. Ben-David et al. (2007; 2010) have derived a theoretical bound of the additional risk when using a representation Z = ϕ(X) rather than raw features X: Inequality 1 (Ben-David et al. (2007; 2010)). For a given ϕ ∈ Φ and g ∈ G, there is:
εt(g ◦ ϕ)︸ ︷︷ ︸ Target risk ≤ εs(g ◦ ϕ)︸ ︷︷ ︸ Source risk + sup (h,h′)∈H̃ ∣∣εs(h, h′)− εt(h, h′)∣∣︸ ︷︷ ︸ = 12 ·d(H̃∆H̃): Disagreement + inf h∈H̃ εt(h) + εs(h)︸ ︷︷ ︸ =λ(H̃): Adaptability
(3)
The inequality 1 ensures that the target risk is bounded by the sum of the source risk, the disagreement risk between two classifiers from representations (d(H̃∆H̃) named H̃∆H̃ distance), and a third term (λ(H̃)) which quantifies the ability to perform well in both domains from representations. In the rest of the paper, we refer to the latter as the adaptability. Since Et[Y |X] is unknown, λ(H̃) is intractable in practical applications. By conveniently assuming that λ(H̃) is negligible, it is possible to derive a Learning Objective 1 which minimizes ε̃s(g◦ϕ)+ 12d(H̃∆H̃). Indeed, minimizing D(s,t) on ϕ leads to learn representations Z = ϕ(X) such that Ps(Z) = Pt(Z), where such situation is sufficient to ensure that d(H̃∆H̃) = 0.

heading: 2.3 THE CURSE OF INVARIANCE IN DOMAIN ADAPTATION

text:
Compressing representations is an easy way to achieve invariance. Assume Xs ∼ N (0, 1), Xt ∼ U(−1, 1) and Y = 1X>0 then a trivial way to obtain representations invariance while separating source data is to consider ϕ : x 7→ 1x>0. The main drawback is a potential loss of information from the original feature space. To explain this phenomenon, we first introduce formally the notion of Hypothesis space compression: Definition 2 (Hypothesis space compression). Let (ϕ1, ϕ2) ∈ Φ2, ϕ1 compresses more H if H̃(ϕ1) ⊂ H̃(ϕ2). By language abuse, we will say that Z1 = ϕ1(X) is more compressed than Z2 = ϕ2(X).
Second, we build two possible solutions (g1, ϕ1) and (g2, ϕ2) of Learning Objective 1 by considering the following compositions X →ϕ Z1 →ψ Z2 →g Ŷ such that ϕ and ψ ◦ ϕ are both in Φ and g and g ◦ ψ are both in G. We set g1 = g ◦ ψ, ϕ1 = ϕ and g2 = g, ϕ2 = ψ ◦ ϕ. These solutions are chosen to ensure that g1 ◦ ϕ1 = g2 ◦ ϕ2 and Z2 = ϕ2(X) is more compressed than Z1 = ϕ1(X). In the case when D(s,t) is the Jensen divergence, the data processing inequality (see Lemma 4.6 in Zhao et al. (2019)) implies that (g2, ϕ2) is a better optimum than (g1, ϕ1) according to Learning Objective 1 since εs(g1 ◦ ϕ1) = εs(g2 ◦ ϕ2) and D(s,t)(ϕ2) ≤ D(s,t)(ϕ1). To conclude, learning under objective 1 will favor compressed representations to enforce invariance. However, as noted in Johansson et al. (2019); Zhao et al. (2019); Liu et al. (2019), more compressed representations will degrade adaptability. Indeed,
λ(H̃(ϕ2)) = inf h∈H̃(ϕ2) εt(h) + εs(h) ≤ inf h∈H̃(ϕ1) εt(h) + εs(h) = λ(H̃(ϕ1)) (4)
since H̃(ϕ1) ⊂ H̃(ϕ2). A question which remains not addressed in the literature (except in Johansson et al. (2019) under the Covariate Shift assumption) is:
How compression may increase the risk of bad adaptability?

heading: 3 THEORETICAL ANALYSIS OF COMPRESSION

heading: 3.1 WORKING PLAN

text: Our strategy aims to compare a representation ϕ with representations ϕ′ which act similarly on the source domain but differently on the target domain (source equivalence). First, by allowing to compute the minimal combined error on source equivalent representations ϕ′, we limit the risk of bad adaptability of a fixed representation. Through an inequality called Target Compression, we show this new control of adaptability comes at the cost of a new risk which embodies the Compression. This is done by studying the projection of an hypothesis defined as its best approximation from representations in the target domain among those that conserve the source domain error (source conservation). Second, we rely on the H̃∆H̃ inequality (Ben-David et al. (2007; 2010)) for stating a new bound of the target risk (Invariant and Compressed Representations inequality). Finally, we expose the particular interest of this bound: it allows a better control on adaptability by considering compression. This is formulated by the Invariance - Compression trade-off.

heading: 3.2 NOTATIONS AND DEFINITIONS

text:
In the following, we fix a given ϕ ∈ Φ. We restrict our analysis to the case of ` : (y, y′) ∈ Y2 7→ (y− y′)2 as a convenient choice for manipulating properly f̃(Z) = E[Y |Z]. Note that in the binary classification case (Y = {0, 1}), ` is the accuracy. We define the set of representations which act similarly on the source domain but differently in the target domain:
Definition 3 (Source equivalence). We introduce the set of source equivalent representations Φ̃s = {ϕ′ ∈ Φ, ϕ′ = ϕ on Ds} (where Ds = supp(Ps(X))) and the source equivalent hypothesis space H̃s = {g ◦ ϕ′ : g ∈ G, ϕ′ ∈ Φ̃s}. Two elements of Φ̃s are said source equivalent.
The source equivalent hypothesis space H̃s is an extension of H̃ adding hypothesis built using representations which are source equivalent to ϕ. For a given h ∈ H̃s, we exhibit an interesting subset of H̃ considering hypothesis with the same source error than h. Definition 4 (Source conservation). For a given h ∈ H̃s, we note H̃h = {h′ ∈ H̃ : εs(h, h′) = 0}.
It is worth noting that for all h ∈ H̃s, H̃h is not empty (see Appendix A for details). We are now ready to introduce the projection of an hypothesis h ∈ H̃s and to this purpose, we note ϕ′ ∈ Φ̃s and g′ ∈ G such that h = g′ ◦ ϕ′. Intuitively, the projection of h is defined as its best approximation from ϕ in the target domain among those which verify the source conservation i.e. from hypothesis g ◦ ϕ with g ∈ G such that εs(g ◦ ϕ, h) = 0. Definition 5 (Projection). π : H̃s → H̃ is defined as follows: π(h) = arg minh′∈H̃h εt(h, h′) for h ∈ H̃s. π(h) is called the projected hypothesis. If there is no uniqueness of the minimum, any solution works.
Illustrations of introduced definitions are provided in Figure 1. If h = g ◦ ϕ ∈ H̃, the projection aims to map h to g′ ◦ϕ with g′ = g both in source and target L2 norms. If, h = g′ ◦ϕ′ with ϕ′ ∈ Φ̃, the projection aims to map h to g ◦ ϕ such that g = g′ in source L2 norm while g ◦ ϕ achieves a minimal distance with g′ ◦ ϕ′ in target L2 norm. We can observe that ∀h ∈ Hs, π ◦ π(h) = π(h) in both source and target L2 norms hence the (abusive) analogy with projection.
For ease of reading, we remind the notations and definitions in appendix B.

heading: 3.3 OUR CONTRIBUTION

text:
In the following, we fix a given H◦ such that H̃ ⊂ H◦ ⊂ H̃s. We are now ready to state a new bound of the target risk:
Inequality 2 (Target Compression). Given g ∈ G and noting h̃ = π(h), we have:
εt(g ◦ ϕ) ≤ εt(g ◦ ϕ, f̃s ◦ ϕ) + sup h∈H◦ εt(h, h̃)︸ ︷︷ ︸ γ̃(H◦): Compression + inf h∈H◦ {εt(h) + εt(h̃, f̃s ◦ ϕ)}︸ ︷︷ ︸ λ̃t(H◦): Adaptability
(5)
Proof. See Appendix A.
This inequality introduces two interesting terms γ̃(H◦) and λ̃t(H◦) with opposite roles; the former involves a supremum while the latter involves an infremum onH◦. The term λ̃t(H◦) is very similar to the adaptability λ(H̃) from Inequality 1 and will be further investigated. Before, we focus on the term γ̃(H◦) which reflects the possibility of finding hypotheses in H◦ that can not be well approximated from target representations. The more information is lost from raw features, the higher the risk, hence its name Compression. This can be formally stated as follows:
Theorem 1 (γ embodies the risk of compression). Let (ϕ1, ϕ2) ∈ Φ2 two source equivalent source representations. If ϕ1 is more compressed than ϕ2, then γ(ϕ1,H◦) ≥ γ(ϕ2,H◦).
Proof. If ϕ1 is more compressed than ϕ2, there is H̃(ϕ1) ⊂ H̃(ϕ2) ⊂ H̃s then ∀h ∈ H̃s, H̃h(ϕ1) ⊂ H̃h(ϕ2). The projection involves an infremum then εt(h, h̃(ϕ2)) ≤ εt(h, h̃(ϕ1)) where h̃(ϕ) = arg minh′∈H̃h(ϕ) ε t(h, h′). The supremum on h ∈ H◦ conserves the inequality.
We now focus our attention on λ̃t(H◦). It differs with λ(H̃) in two ways. First, the source labelling function is involved only with the intermediate f̃s(Z) = Es[Y |Z]. Second, the risk associated with it (εt(h̃, f̃s◦ϕ)) involves a target expectation. In order to obtain a comparable bound with Inequality 1, we state a new bound of the target risk which is our main contribution:
Inequality 3 (Invariant and Compressed Representations). Given g ∈ G and noting h̃ = π(h), β = 2 infh∈H̃ ε s(h) + εt(h, f̃s), we have:
εt(g ◦ ϕ) ≤ εs(g ◦ ϕ) + d(H̃∆H̃) + β + sup h∈H◦ εt(h, h̃)︸ ︷︷ ︸ γ̃(H◦): Compression + inf h∈H◦ εt(h) + εs(h)︸ ︷︷ ︸ λ̃(H◦): Adaptability
(6)
Proof. See AppendixA. The main idea is to bound εt(h, f̃s ◦ϕ) and εt(h̃, f̃s) using H̃∆H̃ inequality. This introduces respectively εs(h, f̃s ◦ϕ) and εs(h̃, f̃s ◦ϕ) with residual terms 12d(H̃∆H̃)+ 12β (two times). We use the source risk conservation to bound εs(h̃, f̃s ◦ ϕ) ≤ εs(h, f̃s ◦ ϕ). Then we bound εs(h, f̃s ◦ ϕ) ≤ εs(h) leveraging the Pythagorean theorem since ` is the L2 norm.
We are now ready to show the better control of the adaptability of the representation ϕ:
Theorem 2 (Better control of adaptability). λ̃(H◦) ≤ λ(H̃).
The proof is given by the infremum on H◦ ⊃ H̃. Compression γ̃(H◦) and adaptability λ̃(H◦) have an opposite behavior. This reflects an interesting trade-off:
Theorem 3 (Invariance and Compression trade-off). With H1◦ and H2◦ such that H̃ ⊂ H1◦ ⊂ H2◦ ⊂ H̃s, there is λ̃(H1◦) ≥ λ̃(H2◦) and γ̃(H1◦) ≤ γ̃(H2◦).
The proof is obtained by observing that H1◦ ⊂ H2◦. Even if invariance may have a bad impact on adaptability, our bound shows it is possible to have a better control of the adaptability while conserving the same degree of invariance. This control comes at the cost of an additional term involving compression of representations. The trade-off between invariance and compression emphasizes than the more control there is over adaptability, the higher the risk associated with compression. Furthermore, this new bound has a particular interest in providing theoretical support for the construction of new learning objectives. The difficulty in DA comes from in the fact that the adaptability is not trainable since it uses the labelling function in the target domain. Our study shows that better control of adaptability (i.e. making the adaptation more reliable) is possible. The two additional terms β and γ̃(H◦) can indeed be trained in an Unsupervised Domain Adaptation setting1.

heading: 4 EXPERIMENTS

text:
We are now conducting an experimental analysis on a widely used DA benchmark of digits recognition (MNIST, USPS). We study two tasks M→U and U→M under the Learning Objective 1. The representation ϕ is a convolutional neural network with a LeNet architecture (LeCun et al. (1998)). We set the dimension of representations to 50 and G is the set of linear classifiers from the representation space to classes (with a softmax layer on top) and the Domain Adversarial Neural Network (DANN Ganin & Lempitsky (2015)) for D(s,t). We use the implementation from Long et al. (2018) for good reproductibility. Models are trained during 30 epochs and we note ϕ̂ the learned representation. In the following, we note CE the cross-entropy.
We suggest to study the trade-off between compression and invariance with the family:
Hη◦ = {g ◦ ϕ′ ∈ H̃s : Et||ϕ′(X)− ϕ(X)|| ≤ ηEt||ϕ(X)||}, η > 0 (7) We interpret this choice as a method for testing robustness of the learned representation to adversarial attacks in the target domain.
Enforcing h ∈ Hη◦ . Enforcing h ∈ Hη◦ is challenging in practrice. We suggest to rely on a penalization L◦ to promote ϕ ≈ ϕ̂ (in the sense of Ps) and ||ϕ− ϕ̂|| . η · ||ϕ̂|| (in the sense of Pt). We define the penalization L◦ as follows:
L◦(ϕ) = Es||ϕ(X)− ϕ̂(X)|| Es||ϕ̂(X)|| + ReLU ( Et||ϕ(X)− ϕ̂(X)|| Et||ϕ̂(X)|| − η )
(8)
Source conservation. For a given h, the projected hypothesis is defined as h̃ = g̃ ◦ ϕ̂ where g̃ is obtained by minimizing on g:
Π(g, h) = λπ · CEs(g ◦ ϕ̂(X), h(X)) + (1− λπ) · CEt(g ◦ ϕ̂(X), h(X)) (9) with λπ = 0.9 for enforcing the source conservation.
1The trainability of β is shown in Appendix A
Estimation of λ̃(H◦). We suggest to minimize on (g, ϕ) ∈ H the following loss: Λ(g, ϕ) = CEt(g ◦ ϕ(X), Y ) + CEs(g ◦ ϕ(X), Y ) + λ◦ · L◦(ϕ) (10)
with λ◦ = 102. CEt(g ◦ϕ(X), Y ) and CEs(g ◦ϕ(X), Y ) are respectively two proxies of εt(h) and εs(h). L◦(ϕ) promotes h ∈ Hη◦ .
Estimation of γ̃(H◦). We follow an adversarial training procedure alternating between
1. maximizing an objective Γ(g, ϕ) on h = g ◦ ϕ ∈ H◦ (sup step) 2. computing the projected hypothesis (inf step)
During experimentation, we have observed that simply setting Γ(g, ϕ) = −Π(ĝ◦ϕ̂, g◦ϕ)+λ◦L◦(ϕ) (where ĝ ◦ ϕ̂ is an estimation of the projected hypothesis) leads to learn noisy labels. We suggest to prevent such pathological solutions by enforcing a low entropy adding to Γ a regularizer λent · CE(h(X), Ŷ ) with (Ŷ = y) = 1(h(X) = y) with λent = −1. Then, we set:
Γ(g, ϕ) = −Π(ĝ ◦ ϕ̂, g ◦ ϕ) + λ◦L◦(ϕ) + λent · CE(h(X), Ŷ ) (11) Furthermore, for a fair comparison, we correct γ by comparing it with a baseline. This baseline is allowed to train representations to fit the adversary.
Analysis We build new target domains modifying the label distribution which is known to badly impact λ(H̃) (Johansson et al. (2019); Zhao et al. (2019)). For a given target dataset D with classes balanced, we define D1 ⊂ ... ⊂ D6 as follows: for 1 ≤ i ≤ 6, Di is obtained by removing half of digit < i from Di−1 and setting D0 = D. Main results are reported in Figure 2. Although estimator are trained using cross-entropy, the results are reported computing the accuracy on a test set in order to be consistent with our theoretical analysis.
We can observe that stronger the adversary is (α→ 1) (and the more the target shift (i→ 6 forDi)), the more significant the difference between λ(H̃) and λ̃(H◦). This demonstrates a better control over adaptability especially when IR fails to perform adaptation. By playing with the value of η, we see that the better control on adaptability is paid to the cost of a higher γ. In our example, the risk of compression is estimated to a potential drop of performances from ∼12.5% to ∼27.5% of accuracy in the target domain while the gain on adaptability is marginal by reducing it of ∼4%. To conclude, a better control on adaptability is obtained with a pessimistic estimation of the risk of compression.

heading: 5 WEIGHTS FOR INVARIANCE RELAXATION

text:
The fact that compression and invariance are conflicting objectives supports the point from Wu et al. (2019) which claims that invariance is too strict a constraint. In this section, we show that invariance can be relaxed leveraging Weighted Representations. More specifically, we show that taking into account the risk of compression leads to introduce an adaptability term which does not depend on learned weights. Then, weighting representations can reduce the discrepancy between the source and the target distributions without impacting the adaptability. We introduce the set of weightsW , a subset of functions w : X → R+ such that Es[w(X)] = 1 and we note for w ∈ W,Pw·s(X) := w(X)Ps(X). The condition Es[w(X)] = 1 ensures that Pw·s(X) is a distribution. We state a similar result than Inequality 1.
Inequality 4 (Weighted version of Inequality1). For a given ϕ ∈ Φ and g ∈ H̃,
εt(g ◦ ϕ) ≤ εw·s(g ◦ ϕ) + 1 2 dw(H̃∆H̃) + λw(H̃) (12)
with dw(H̃∆H̃) = sup(h,h′)∈H̃ |εw·s(h, h′)− εt(h, h′)| and λw = infh∈H̃ εt(h) + εw·s(h).
Interestingly, w and ϕ play similar role in inequality 4 since they are actionable parameters to minimize εw·s(g ◦ ϕ) + 12dw(H̃∆H̃) while we can not quantify their impact on λw(H̃). We show that taking advantage of compression leads to an adaptability which does not depend on weights. This needs to introduce some definitions:
Definition 6 (Strong source conservation). For a given h ∈ H̃s, we note H̃h = {h′ ∈ H̃ : h = h′ on Ds} where Ds = supp(Pt(X)). Definition 7 (Strong projection). Π : H̃s → H̃ is defined as follows π(h) = arg minh′∈H̃h εt(h, h′) for h ∈ H̃s. If there is no uniqueness of the minimum, any solution works.
We are now ready to state the role of weights as a relaxed invariance for a lower compression: Inequality 5 (Weighting Invariant and Compressed Representations). Given g ∈ G and noting h = Π(h), βw = 2 infh∈H̃ ε w·s(h) + εt(h, f̃s), we have:
εt(g◦ϕ) ≤ inf w∈W
{ εw·s(g ◦ ϕ) + 1
2 dw(H̃∆H̃) +
1 2 βw } + sup h∈H◦ εt(h,h)+ inf h∈H◦ εt(h)+εt(h, f̃s◦ϕ)
Proof. This is a straightforward extension of inequality 2 applied to Pw·s. Furthermore, infh∈H◦ ε
t(h) + εt(h, f̃s ◦ ϕ) does not depend on w due to strong risk conservation (risk conservation would have been dependent on w through the constraint εw·s(h, h̃) = 0).
Adding weights exposes to the risk of increasing the variance of estimators when considering finite sampling analysis (see Cortes et al. (2010) for an extensive theoretical analysis of Importance Sampling). In inequality 5, this can be controlled with a good choice ofW e.g. by considering weights w ≤M for a given M > 0.

heading: 6 RELATED WORK

text: Generalization bounds for DA have been the object of an extensive literature for both Importance Sampling Cortes et al. (2010) and Invariant Representations Ben-David et al. (2007; 2010); Mansour et al. (2009). It has attracted a lot of attention in order to understand failure cases observed when using IR (Zhao et al. (2019); Johansson et al. (2019)). The theoretical analysis in Johansson et al. (2019) is the work closest to ours. They introduce a loss information metric which quantifies the risk of using non-invertible representations. The present work differs in three ways. First, our work follows the formalism introduced in Ben-David et al. (2007) which is widely adopted in the community. Second, contrary to the information loss, our theory provides a tractable risk of compressing representations. Finally, we do not rely on the Covariate Shift assumption which gives our work a broader range of applications.

heading: 7 CONCLUSION

text:
By addressing domain generalization under the perspective of compression, we have bounded the target risk by a sum of trainable terms (εs(g ◦ ϕ) + d(H̃∆H̃) + β + γ̃(H◦)) and a term λ̃(H◦) which embeds a new adaptability criterion of representations. We have shown that λ̃(H◦) can be better controlled by enriching the hypothesis class H◦. This control comes at a cost of a higher risk of compression underlying a trade-off between invariance and compression when learning IR for DA. Through an empirical study on a standard benchmark, we have exhibited the better control of adaptability. Furthermore, we have shown how our framework can be applied for studying adaptation robustness to adversarial attacks. Finally, as an attempt to obtain representation distributions invariance with a lower compression, we have emphasized the role of weighted representations for relaxing the constraint of invariance.
Achieving representation invariance can reduce the generalization gap between two domains. However, all invariances will not have the same guarantees for better generalization. Our work shows that we will look for representations which preserve the best the information in the original target features spaces. We leave both the design on new Domain Adaptation methods which incorporate such consideration and relevant design ofH◦ as future works.
Promising directions (1) Our analysis is restricted to the comparison among source equivalent representations of a given representation ϕ ∈ Φ. Then, for two given representations ϕ1 and ϕ2 of Φ which does not verify the source equivalence, our analysis does not allow to state if ϕ1 presents, or not, better guarantees on the success of the adaptation. Understanding how the source equivalence can be relaxed can provide to the present work a broader range of applications. (2) Obtaining a robust estimation of γ̃(H◦) by following an adversarial training procedure appeared to be challenging in practice. Finding fast and robust estimations of γ̃(H◦) (or a proxy of it) may help to derive new learning objectives or policy for model selection. For instance, Information Theory gives better suited devices for modelling the phenomenon of compression. (3) A strategy may to enforce invariance directly in the target domain by transporting source samples. Formally, assuming Ds ∩ Dt = ∅, we set ϕ such that ϕ(X) = X on Dt and ϕ(Ds) ⊂ Dt. If ϕ is fixed in the source domain (i.e. ϕ|Ds is fixed) the optimal representation from the compression perspective is ϕ|Dt(X) = X . This consideration is connected with two recent trends which perform adaptation by generating cross domain samples (Sankaranarayanan et al. (2018); Hoffman et al. (2018); Bousmalis et al. (2016)) or by finding Optimal Transport between the source and the target domains (Courty et al. (2016; 2017)). (4) The Covariate Shift assumption can be stated as H◦ = {h ∈ H, εs(h, ĥ) = 0 with ĥ = arg minh∈H εs(h)}. Such H◦ may provide a new framework for addressing the case of non-overlapping supports in Covariate Shift adaptation.

heading: A MAIN PROOFS

text:
Proposition 1 (The source conservation hypothesis is not empty). For a given ϕ ∈ Φ and h ∈ H̃s, H̃h is not empty.
Proof. If h ∈ H̃s, it exists g ∈ G and ϕ′ ∈ Φ such that h = g ◦ ϕ′. We set h′ = g ◦ ϕ ∈ H̃, there is εs(h, h′) = εs(g ◦ ϕ′, g ◦ ϕ) = εs(g ◦ ϕ, g ◦ ϕ) = 0 since ϕ = ϕ′ on Ds then h′ ∈ H̃h.
Inequality 6 (Target Compression). Given ϕ ∈ Φ, g ∈ G and noting h̃ = π(h), there is for anyH◦ such that H̃ ⊂ H◦ ⊂ H̃s:
εt(g ◦ ϕ) ≤ εt(g ◦ ϕ, f̃s ◦ ϕ) + sup h∈H◦ εt(h, h̃) + inf h∈H◦ {εt(h) + εt(h̃, f̃s ◦ ϕ)} (13)
Proof. Let ϕ ∈ Φ, g ∈ G and h ∈ H◦:
εt(g ◦ ϕ) ≤ εt(f̃s ◦ ϕ) + εt(f̃s ◦ ϕ, g ◦ ϕ) (14) ≤ εt(h) + εt(h, f̃s ◦ ϕ) + εt(f̃s ◦ ϕ, g ◦ ϕ) (15) ≤ εt(h) + εt(h, h̃) + εt(h̃, f̃s ◦ ϕ) + εt(f̃s ◦ ϕ, g ◦ ϕ) (16) ≤ εt(f̃s ◦ ϕ, g ◦ ϕ) + εt(h, h̃) + εt(h) + εt(h̃, f̃s ◦ ϕ) (17)
This is a succession of triangular inequalities which involve respectively f̃s ◦ ϕ, h and h̃. The last inequality is a convenient arrangement of terms. We bound εt(h, h̃) by its supremal value for h ∈ H◦, then εt(g ◦ ϕ) ≤ εt(f̃s ◦ ϕ, g ◦ ϕ) + suph∈H◦{εt(h, h̃)} + εt(h) + εt(h̃, f̃s ◦ ϕ). Since this holds for all h ∈ H◦, we bound εt(g ◦ ϕ) by the infremum for h ∈ H◦.
Inequality 7 (Invariant and Compressed Representations). Given ϕ ∈ Φ, g ∈ G and noting h̃ = π(h), β = 2 infh∈H̃ ε s(h) + εt(h, f̃s), there is for anyH◦ such that H̃ ⊂ H◦ ⊂ H̃s:
εt(g ◦ ϕ) ≤ εs(g ◦ ϕ) + d(H̃∆H̃) + β + sup h∈H◦ εt(h, h̃)︸ ︷︷ ︸ γ̃(H◦): Compression + inf h∈H◦ εt(h) + εs(h)︸ ︷︷ ︸ λ̃(H◦): Adaptability
(18)
Proof. First of all, we recall the H̃∆H̃ inequality from Ben-David et al. (2007; 2010), for any fs and f t, there is:
∀h ∈ H̃, εt(h, f t) ≤ εs(h, fs) + 1 2 d(H̃∆H̃) + inf h∈H̃ εt(h, f t) + εs(h, fs) (19)
A particular case of that inequality applied to fs = f t = f̃s ◦ ϕ leads to bound:
εt(g ◦ ϕ, f̃s ◦ ϕ) ≤ εs(g ◦ ϕ, f̃s ◦ ϕ) + 1 2 d(H̃∆H̃) + inf h∈H̃ εs(h, f̃s ◦ ϕ) + εt(h, f̃s ◦ ϕ)
≤ εs(g ◦ ϕ) + 1 2 d(H̃∆H̃) + inf h∈H̃ εs(h) + εt(h, f̃s ◦ ϕ) (20)
where bounding εs(g ◦ ϕ, f̃s ◦ ϕ) ≤ εs(g ◦ ϕ) is a direct property of `(y, y′) = (y − y′)2 and f̃s(Z) = Es[Y |Z]: εs(g ◦ ϕ) = εs(g ◦ ϕ, f̃s ◦ ϕ) + εs(f̃s ◦ ϕ). Note that, it is not possible to do so for εt(g ◦ ϕ, f̃s ◦ ϕ) since the expectation is computed in the target domain. We define β = 2 infh∈H̃ ε
s(h) + εt(h, f̃s ◦ ϕ). We apply a second time H̃∆H̃ inequality, there is for any h ∈ H◦:
εt(h̃, f̃s ◦ ϕ) ≤ εs(h̃, f̃s ◦ ϕ) + 1 2 d(H̃∆H̃) + inf h∈H̃ εt(h, f̃s ◦ ϕ) + εs(h, f̃s ◦ ϕ)
≤ εs(h̃) + 1 2 d(H̃∆H̃) + inf h∈H̃ εt(h, f̃s) + εs(h) (21)
which is licit since h̃ ∈ H̃. Relying on the projected hypothesis h̃ allows us to preserve the invariance: even if h ∈ H◦, d(H̃∆H̃) is involved, not d(H◦∆H◦). Then, we use the property of source risk conservation h ∈ H which enforces εs(h̃, h) = 0 for noting that εs(h̃) ≤ εs(h) + εs(h, h̃) = εs(h) then:
εt(h̃, f̃s ◦ ϕ) ≤ εs(h) + 1 2 d(H̃∆H̃) + inf h∈H̃ εt(h, f̃s) + εs(h)
Finally, taking the infremum:
inf h∈H◦ εt(h) + εt(h̃, f̃s ◦ ϕ) ≤ 1 2 β + 1 2 d(H̃∆H̃) + inf h∈H◦ εt(h) + εt(h) (22)
Combining all inequalities leads to the stated one.
Theorem 4 (Better control on the adaptability). λ̃(H◦) ≤ λ(H̃).
Proof. We simply use the fact that H̃ ⊂ H◦ then using the property of the infremum:
λ̃(H◦) = inf h∈H◦ εt(h) + εs(h) ≤ inf h∈H̃ εt(h) + εs(h) = λ(H̃) (23)
Theorem 5 (γ embodies the risk of compression). Let (ϕ1, ϕ2) ∈ Φ2 two source equivalent representations. If ϕ1 is more compressed than ϕ2, then γ(ϕ1,H◦) ≥ γ(ϕ2,H◦) for H̃ ⊂ H◦ ⊂ H̃s.
Proof. If ϕ1 and ϕ2 are two equivalent representations, there is H̃(ϕ1) ⊂ H̃(ϕ2) ⊂ H̃s then
∀h ∈ H̃s, (H̃(ϕ1) ∩ {h′ ∈ H : εs(h, h′) = 0}) ⊂ (H̃(ϕ2) ∩ {h′ ∈ H : εs(h, h′) = 0})
which guarantees H̃h(ϕ1) ⊂ H̃h(ϕ2). Finally
εt(h, h̃(ϕ2)) ≤ εt(h, h̃(ϕ1))
invoking the property of the infremum where h̃(ϕ) = arg minh′∈H̃h(ϕ) ε t(h, h′). The supremum on h ∈ H◦ conserves the inequality. Proposition 2 (β is trainable from the data). Let (gn, ϕn)n∈N a sequence such that α · ε̃s(gn ◦ϕn)+ (1− α) · D(ϕn)→ 0 then βn → 0.
Proof. First of all, since D(ϕn)→ 0 then Pt(ϕn(X))→ Ps(ϕn(X)). We note that for any n ∈ N, β ≤ 2·(εs(gn◦ϕn)+εt(gn◦ϕn, f̃sn◦ϕn)) ≤ 2·(2·εs(gn◦ϕn)+(εt(gn◦ϕn, f̃sn◦ϕn)−εs(gn◦ϕn))) Since εs(gn ◦ϕn)→ and (εt(gn ◦ϕn, f̃s)−εs(gn ◦ϕn))→ 0 (because Pt(ϕn(X))→ Ps(ϕn(X))) there is βn → 0.

heading: B NOTATIONS

text:
• Φ is the set of representations. • G is the set of classifiers. • H is the hypothesis spaceH = G ◦ Φ • For a given ϕ ∈ Φ, we note H̃(ϕ) = {g ◦ϕ : g ∈ G} = G ◦ϕ. When there is no ambiguity,
we note H̃(ϕ) = H̃. • For a given h ∈ H, we note ε(h) = E[(Y − h(X))2]. For d = s, t, we note a domain risk εd(h) = Ed[(Y − h(X))2].
• For d = s, t and a given ϕ ∈ Φ, we note the conditional expectation f̃d(Z) = Ed[Y |Z] with Z = ϕ(X).
• Equivalence. For a given ϕ ∈ Φ, the set of source equivalent representations Φ̃s(ϕ) = {ϕ′ ∈ Φ, ϕ′ = ϕ on Ds} (where Ds = supp(Ps(X))) and the source equivalent hypothesis space H̃s(ϕ) = {g ◦ ϕ′ : g ∈ G, ϕ′ ∈ Φ̃s}. Two elements of Φ̃s are said source equivalent. When there is no ambiguity, we note Φ̃s and H̃s.
• Conservation. – For a given h ∈ H̃s, we note H̃h = {h′ ∈ H̃ : εs(h, h′) = 0}. – For a given h ∈ H̃s, we note H̃h = {h′ ∈ H̃ : h = h′ on Ds} where Ds = supp(Pt(X)). • Projection.
– π : H̃s → H̃ is defined as follows: π(h) = arg minh′∈H̃h εt(h, h′) for h ∈ H̃s. π(h) is called the projected hypothesis. If there is no uniqueness of the minimum, any solution works. – Π : H̃s → H̃ is defined as follows π(h) = arg minh′∈H̃h εt(h, h′) for h ∈ H̃s. If there is no uniqueness of the minimum, any solution works.


summaries: Authors proposed an algorithm to predict the attribute of GUI elements from rasterized design images. The problem is separated into two steps. The first step is to predict initial values of the attributes (border width, color, padding etc) from the image where the type of UI element and set of attributes are already known. Authors designed a typical convolutional DNN for each of the attributes. The second step is to learn a policy \pi to iteratively adjust one attribute a time until the final rendering matches input image pixel-perfect.

summaries: This paper proposes an approach for reverse-engineering webpages using Siamese networks and imitation learning. While the idea of using synthetic data (which can be easily procedurally generated) to do this reverse-engineer training is very clever, prior work has exploited it also. Novel elements include the attribute refinement using imitation learning, and the authors show the effect of this step, but the improvement is small. Thus, the limited novelty and not very convincing results make the question the potential impact of this paper.

summaries: The paper proposes an approach to infer the attribute values of an input image representing a user interface. The model first infers the most likely initial attribute values, and iteratively refine them to improve the similarity between the input image and the interface generated from the newly inferred attributes. The model is trained on synthetic datasets generated by a black box rendering engine, and generalizes well to real-world datasets. To address the issues of pixel based metrics and mean squared error, the authors instead uses the probability that two images are equal in the attribute space to define the cost between these two images.

abstractText: We explore a new domain of learning to infer user interface attributes that helps developers automate the process of user interface implementation. Concretely, given an input image created by a designer, we learn to infer its implementation which when rendered, looks visually the same as the input image. To achieve this, we take a black box rendering engine and a set of attributes it supports (e.g., colors, border radius, shadow or text properties), use it to generate a suitable synthetic training dataset, and then train specialized neural models to predict each of the attribute values. To improve pixel-level accuracy, we also use imitation learning to train a neural policy that refines the predicted attribute values by learning to compute the similarity of the original and rendered images in their attribute space, rather than based on the difference of pixel values. We instantiate our approach to the task of inferring Android Button attribute values and achieve 92.5% accuracy on a dataset consisting of real-world Google Play Store applications.

title: Pix2code: Generating code from a graphical user interface screenshot

venue: In Proceedings of the ACM SIGCHI Symposium on Engineering Interactive Computing Systems, EICS ’18,

authors:
Pavol Bielik
Marc Fischer
Martin Vechev

title: Robust relational layout synthesis from examples for android

venue: Proc. ACM Program. Lang.,

authors:
Chunyang Chen
Ting Su
Guozhu Meng
Zhenchang Xing
Yang Liu

title: From ui design image to gui skeleton: A neural machine translator to bootstrap mobile gui implementation

venue: In Proceedings of the 40th International Conference on Software Engineering,

authors:
Guang Chen
Jianchao Yang
Hailin Jin
Jonathan Brandt
Eli Shechtman
Aseem Agarwala
Tony X. Han

title: Large-scale visual font recognition

venue: In Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition,

authors:
Biplab Deka
Zifeng Huang
Chad Franzen
Joshua Hibschman
Daniel Afergan
Yang Li
Jeffrey Nichols
Ranjitha Kumar

title: Rico: A mobile app dataset for building data-driven design applications

venue: In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology,

authors:
Kevin Ellis
Daniel Ritchie
Armando Solar-Lezama
Josh Tenenbaum

title: Learning to infer graphics programs from hand-drawn images

venue: Advances in Neural Information Processing Systems

authors:
Yaroslav Ganin
Tejas Kulkarni
Igor Babuschkin
S.M. Ali Eslami
Oriol Vinyals

title: Synthesizing programs for images using reinforced adversarial learning

authors:
Ankush Gupta
Andrea Vedaldi
Andrew Zisserman

title: Synthetic data for text localisation in natural images

venue: In IEEE Conference on Computer Vision and Pattern Recognition,

authors:
Forrest Huang
John F. Canny
Jeffrey Nichols

title: Swire: Sketch-based user interface retrieval

venue: In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems,

authors:
M Jaderberg
K Simonyan
A Vedaldi
A Zisserman

title: Synthetic data and artificial neural networks for natural scene text recognition

venue: Neural Information Processing Systems,

authors:
Max Jaderberg
Karen Simonyan
Andrea Vedaldi
Andrew Zisserman

title: Reading text in the wild with convolutional neural networks

venue: Int. J. Comput. Vision,

authors:
Justin Johnson
Agrim Gupta
Li Fei-Fei

title: Image generation from scene graphs

venue: In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR),

authors:
Tejas D Kulkarni
William F. Whitney
Pushmeet Kohli
Josh Tenenbaum

title: Deep convolutional inverse graphics network

authors:
Abhijit Kundu
Yin Li
James M. Rehg

title: 3d-rcnn: Instance-level 3d object reconstruction via render-and-compare

authors:
Pengyuan Lyu
Minghui Liao
Cong Yao
Wenhao Wu
Xiang Bai

title: Mask TextSpotter: An endto-end trainable neural network for spotting text with arbitrary shapes

venue: Computer Vision – ECCV

authors:
Stéphane Ross
Geoffrey Gordon
Drew Bagnell

title: A reduction of imitation learning and structured prediction to no-regret online learning

venue: In Proceedings of the fourteenth international conference on artificial intelligence and statistics,

title: Colorimetry: understanding the CIE system

authors:
Ashish Shrivastava
Tomas Pfister
Oncel Tuzel
Joshua Susskind
Wenda Wang
Russell Webb

title: Learning from simulated and unsupervised images through adversarial training

authors:
Karen Simonyan
Andrea Vedaldi
Andrew Zisserman

title: Deep inside convolutional networks: Visualising image classification models and saliency

venue: maps. CoRR,

authors:
Amanda Swearngin
Mira Dontcheva
Wilmot Li
Joel Brandt
Morgan Dixon
Andrew J. Ko

title: Rewire: Interface design assistance from examples

venue: In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems,

authors:
Zhangyang Wang
Jianchao Yang
Hailin Jin
Eli Shechtman
Aseem Agarwala
Jonathan Brandt
Thomas S. Huang

title: Deepfont: Identify your font from an image

venue: In Proceedings of the 23rd ACM International Conference on Multimedia, MM

authors:
Zhou Wang
Alan C. Bovik
Hamid R. Sheikh
Eero P. Simoncelli

title: Image quality assessment: From error visibility to structural similarity

venue: IEEE TRANSACTIONS ON IMAGE PROCESSING,

title: Simple statistical gradient-following algorithms for connectionist reinforcement learning

venue: Machine Learning,

authors:
Jiajun Wu
Joshua B Tenenbaum
Pushmeet Kohli

title: Neural scene de-rendering

authors:
Jiajun Wu
Tianfan Xue
Joseph J Lim
Yuandong Tian
Joshua B Tenenbaum
Antonio Torralba
William T Freeman

title: 3d interpreter networks for viewer-centered wireframe modeling

venue: International Journal of Computer Vision (IJCV),

authors:
Nanxuan Zhao
Ying Cao
Rynson W.H. Lau

title: Modeling fonts in context: Font prediction on web designs

venue: Computer Graphics Forum (Proc. Pacific Graphics 2018),

heading: 1 INTRODUCTION

text:
With over 5 million applications in Google Play Store and Apple App Store and over a billion webpages, a significant amount of time can be saved by automating even small parts of their development. To achieve this, several tools have been recently developed that help user interface designers explore and quickly prototype different ideas, including Sketch2Code (Microsoft, 2018) and InkToCode (Corrado et al., 2018), which generate user interface sketches from hand-drawn images, Swire (Huang et al., 2019) and Rico (Deka et al., 2017), which allow retrieving designs similar to the one supplied by the user and Rewire (Swearngin et al., 2018), which transforms images into vector representations consisting of rectangles, circles and lines. At the same time, to help developers implement the design, a number of approaches have been proposed that generate layout code that places the user interface components at the desired position (e.g., when resizing the application). These include both symbolic synthesis approaches such as InferUI (Bielik et al., 2018), which encodes the problem as a satisfiability query of a first-order logic formula, as well as statistical approaches (Beltramelli, 2018; Chen et al., 2018), which use encoder-decoder neural networks to process the input image and output the corresponding implementation.
In this work, we explore a new domain of inferring an implementation of an user interface component from an image which when rendered, looks visually the same as the input image. Going from an image to a concrete implementation is a time consuming, yet necessary task, which is often outsourced to a company for a high fee (replia, 2019; psd2android, 2019; psd2mobi, 2019). Compared to prior work, we focus on the pixel-accurate implementation, rather than on producing sketches or the complementary task of synthesizing layouts that place the components at the desired positions.
Concretely, given a black box rendering engine that defines a set of categorical and numerical attributes of a component, we design a two step process which predicts the attribute values from an input image – (i) first, we train a neural model to predict the most likely initial attribute values, and then (ii) we use imitation learning to iteratively refine the attribute values to achieve pixel-level accuracy. Crucially, all our models are trained using synthetic datasets that are obtained by sampling the black box rendering engine, which makes it easy to train models for other attributes in the future. We instantiate our approach to the task of inferring the implementation of Android Button attributes and show that it generalizes well to a real-world dataset consisting of buttons found in existing Google Play Store applications. In particular, our approach successfully infers the correct attribute values in 94.8% and 92.5% of the cases for the synthetic and the real-world datasets, respectively.

heading: 2 RELATED WORK

text:
As an application, our work is related to a number of recently developed tools in the domain of user interface design and implementation with the goal of making developers more productive, as discussed in Section 1. Here we give overview of the related research from a technical perspective.
Inverting rendering engines to interpret images The most closely related work to ours phrases the task of inferring attributes from images as the more general task of learning to invert the rendering engines used to produce the images. For example, Wu et al. (2017) use reinforcement learning to train a neural pipeline that given an image of a cartoon scene or a Minecraft screenshot, identifies objects and a small number of high level features (e.g., whether the object is oriented left or right). Ganin et al. (2018) also use reinforcement learning, but with an adversarially learned reward signal, to generate a program executed by a graphics engine that draws simple CAD programs or handwritten symbols and digits. Johnson et al. (2018) and Ellis et al. (2018) design a neural architecture that generates a program that when rendered, produces that same 2D or 3D shape as in the input image. While Johnson et al. (2018) train the network using a combination of supervised pretraining and reinforcement learning with a custom reward function (using Chamfer distance to measure similarity of two objects), Ellis et al. (2018) use a two step process that first uses supervised learning to predict a set of objects in the image and then synthesizes a program (e.g., containing loops) that draws them.
In comparison to these prior works, our approach differs in three key aspects. First, the main challenge in prior works is predicting the set of objects contained in the image and how to compose them. Instead, the focus of our work is in predicting a set of object properties after the objects in the image were already identified. Second, instead of using the expensive REINFORCE (Williams, 1992) algorithm (or its variation) to train our models, we use a two step process that first pretrains the network to make an initial prediction and then uses imitation learning to refine it. This is possible because, in our setting, there is a fixed set of attributes known in advance for which we can generate a suitable synthetic dataset used by both of these steps. Finally, because our goal is to learn pixel-accurate attribute values, the refinement loop takes as input both the original image, as well as the rendered image of the current attribute predictions. As a result, we do not require our models to predict pixel-accurate rendering of an attribute value but instead, to only predict whether the attribute values in two images are the same or in which direction they should be adjusted.
Attribute prediction Optical character recognition (Jaderberg et al., 2016; Lyu et al., 2018; Jaderberg et al., 2014; Gupta et al., 2016) is a well studied example of predicting an attribute from an image with a large number of real-world applications. Other examples include predicting text fonts (Zhao et al., 2018; Wang et al., 2015; Chen et al., 2014), predicting eye gaze (Shrivastava et al., 2017), face pose and lighting (Kulkarni et al., 2015), chair pose and content (Wu et al., 2018) or 3D object shapes and pose (Kundu et al., 2018), to name just a few. The attribute prediction network used in our work to predict the initial attribute value is similar to these existing approaches, except that it is applied to a new domain of inferring user interface attributes. As a result, while some of the challenges remain the same (e.g., how to effectively generate synthetic datasets), our main challenge is designing a pipeline, together with a network architecture capable of achieving pixel-level accuracies on a range of diverse attributes.

heading: 3 BACKGROUND: USER INTERFACE ATTRIBUTES

text:
Visual design of user interface components can be specified in many different ways – by defining a program that draws on a canvas, by defining a program that instantiates components at runtime and manipulates their properties, declaratively by defining attribute values in a configuration file (e.g., using CSS), or by using a bitmap image that is rendered in place of the components. In our work, we follow the best practices and consider the setting where the visual design is defined declaratively, thus allowing separating the design from the logic that controls the application behaviour.
Formally, let C denote a component with a set of attributes A. The domain of possible values of each attribute ai ∈A is denoted as Θi. As all the attributes are rendered on a physical device, their domains are finite sets containing measurements in pixels or a set of categorical values. For example, the domain for the text color attribute is three RGB channels N3×[0,255], the domain for text gravity is {top, left, center, right, bottom} and the domain for border width is N[0,20]. We distinguish
two domain types: (i) comparable (e.g., colors, shadows or sizes) for which a valid distance metric d : Θ×Θ→N[0,∞) exists, and (ii) uncomparable (e.g., font types or text gravity) for which the distance between any two attribute values is equal to one. We use Θ ⊆ Θ1×· · ·×Θn to denote the space of all possible attribute configurations, and use the function render : Θ→R3×h×w to denote an image with width w, height h and three color channels obtained by rendering the attribute configuration y∈Θ. Furthermore, we use the notation y∼Θ to denote a random sample of attribute values from the space of all valid attribute configurations. Finally, we note that attributes often affect the same parts of the rendered image (e.g., the shadow is overlayed on top of the background) and they are in general not independent of each other (e.g, changing the border width affects the border radius).

heading: 4 LEARNING TO INFER USER INTERFACE ATTRIBUTES FROM IMAGES

text:
We now present our approach for learning to infer user interface component attributes from images.
Problem statement Given an input image I ∈ R3×h×w, our goal is to find an attribute configuration y ∈ Θ which when rendered, produces an image most visually similar to I:
arg miny∈Θ cost(I, render(y))
where cost : I × I → R[0,∞) is a function that computes the visual similarity of a given user interface component in two images. It returns zero if the component looks visually the same in both images or a positive real value denoting the degree to which the attributes are dissimilar.
The first challenge that arises from the problem statement above is how to define the cost function. Pixel based metrics, such as mean squared error of pixel differences, are not suitable and instead of producing images with similar attribute values, produce images that have on average similar colors. Training a discriminator also does not work, as all the generated images are produced by rendering a set of attributes and are true images by definition. Finally, the cost can be computed not over the rendered image but by comparing the predicted attributes y with the ground-truth labels. Unfortunately, even if we would spend the effort and annotated a large number of images with their ground-truth attributes, using a manually annotated dataset restricts the space of models that can be used to infer y to only those that do supervised learning. In what follows we address this challenge by showing how to define the cost function over attributes (used for supervise learning) as well as over images (used for reinforcement learning), both by using a synthetically generated dataset.
Our approach To address the task of inferring user interface attributes from images, we propose a two step process that – (i) first selects the most likely initial attribute values arg maxy∈Θ p(y | I) by learning a probability distribution of attribute values conditioned on the input image, and then (ii) iteratively refines the attribute values by learning a policy π(∆y(i) |I, render(y(i))) that represents the probability distribution of how each attribute should be changed, conditioned on both the original image, as well as the rendered image of the attribute configuration y(i) at iteration i. We use the policy π to define the cost between two images as cost(I, I ′) := 1−π(∆y=0 | I, I ′). That is, the cost is defined as the probability that the two images are not equal in the attribute space.
We illustrate both steps in Figure 1 with an example that predicts attributes of a Button component. In Figure 1 (a), the input image is passed to a set of convolutional neural networks, each of which is trained to predict a single attribute value. In our example, the most likely value predicted for the border width is 2dp while the most likely color of the border is #4a4a4a. Then, instead of returning the most likely attribute configuration y, we take advantage of the fact that it can be rendered and compared to the original input image. This give us additional information that is used to refine the predictions as shown in Figure 1 (b). Here, we use a pair of siamese networks (pretrained on the prediction task) to learn the probability distribution over changes required to make the component attributes in both images the same. In our example, the network predicts that the border color and the text gravity attributes have already the correct values but the border width should be decreased by 2dp and the shadow should be increased by 4dp. Then, due to the large number of different attributes that affect each other, instead of applying all the changes at once, we select and apply a single attribute change. In our example, the ∆y corresponds to adjusting the value of border width by −2dp. Since the change is supposed to correct a mispredicted attribute value, we accept it only if it indeed makes the model more confident that the prediction is correct.
(a) Attribute Prediction (Section 4.1)
CNNCNN
y border width 2dp border color
shadow 0dp text gravity center
. . .
p(y | I) 0.77 0.99 0.91 0.96
D = {( render(y(i)),y(i) ∼ Θ )}N
i=1
Training
Input Image I p(y | I) Predicted Attributes render(y)
Synthetic datasets We instantiate our approach by training it purely using synthetic datasets, while ensuring that it generalizes well to real-world images. This allows us to avoid the expensive task of collecting and annotating real-world datasets and more importantly, makes our approach easily applicable to new domains and attributes. In particular, given a space of possible attribute configurations Θ and a rendering function render, we generate two different datasets D and D∆ used to train the attribute prediction network and the policy π, respectively. The dataset D = {(render(y(i)),y(i) ∼ Θ)}Ni=1 is constructed by sampling a valid attribute configuration y(i) ∼ Θ and rendering it to produce the input image. To generateD∆, we sample two attribute configurations y(i),y(j)∼Θ that are used to render two input images and train the network to predict the difference of their attributes, that is,D∆ ={(〈render(y(i) ∼ Θ), render(y(j) ∼ Θ)〉,∆(y(i)−y(j)))}Mi,j=1. For both datasets, to avoid overfitting when training models for attributes with large domain of possible values, we sample only a subset of attributes, while setting the remaining attributes to values from the previous example y(i−1). As a result, every two consecutive samples are similar to each other, since a subset of their attributes is the same. Further, because the real-world images do not contain components in isolation but together with other components that fill the rest of the screen, we introduce three additional attributes xpos∈N, ypos∈N and background. We use xpos and ypos to denote the horizontal and vertical position of the component in the image, respectively. This allows the network to learn robust predictions regardless of the component position in the image. We use background to select the background on which the component is rendered. We experimented with three different choices of backgrounds – only while color, random solid color and overlaying the component on top of an existing application, all of which are evaluated in Section 5.

heading: 4.1 ATTRIBUTE PREDICTION

text:
The attribute prediction network architecture is a multilayer convolutional neural network (CNN) followed by a set of fully connected layers. The multilayer convolutional part consists of 6 repetitive sequences of convolutional layers with ReLU activations, followed by batch normalization and a max-pooling layer of size 2 and stride 2. For the convolutional layers we use 3×3 filters of size 32, 32, 64, 64, 128 and 128, respectively. The result of the convolutional part is then flattened and
connected to a fully connected layer of size 256 with ReLU activation followed by a final softmax layer (or a single neuron for regression). We note that this is not a fixed architecture and instead, it is adapted to a given attribute by performing an architecture search, as shown in Section 5.
Supporting multiple input sizes To support user interface components of different sizes, we select the input image dimension such that it is large enough to contain them. This is necessary as our goal is to infer pixel-accurate attribute values and scaling down or resizing the image to a fixed dimension is not an option, as it leads to severe performance degradation. However, this is problematic, as most of the input images are smaller or cropped in order to remove other components. As a result, before feeding the image to the network we need to increase the dimension of the input without resizing or scaling. To achieve this, we pad the missing pixels with the values of edge pixels, which improves the generalization to real-world images as shown in Section 5.1 and illustrated in Appendix C.
Optimizations To improve the accuracy and reduce the variance of the attribute prediction network, we perform the following two optimizations. First, we compute the most likely attribute value by combining multiple predictions and selecting the most likely among them. This is achieved by generating several perturbations of the input image, each of which shifts the image randomly, horizontally by x ∼ U(−t, t) and vertically by y ∼ U(−t, t), where t ∈ N. This is similar to ensemble models but instead of training multiple models we generate and evaluate multiple inputs.
Second, to improve the accuracy of the color attributes, we perform color clipping by picking the closest color to one of those present in the input image. To reduce the set of all possible colors, we use saliency maps (Simonyan et al., 2013) to select a subset of the pixels most relevant for the prediction. In our experiments we keep only the pixels with the normalized saliency value above 0.8. Then, we clip the predicted color to the closest color from the top five most common colors among the pixels selected by the saliency map. We provide illustration of the color clipping in Appendix D.

heading: 4.2 ATTRIBUTE REFINEMENT LOOP

text:
We now describe how to learn a function π(∆y | I, render(y)) that represents probability distribution of how each attribute should be changed, conditioned on both the original image as well as the rendered image of the current attribute configuration y. We can think of π as a policy, where the actions correspond to changing an attribute value and the state is a tuple of the original and the currently rendered image. We can then apply imitation learning to train the policy π on a synthetic dataset D∆ ={(〈render(y(i)∼Θ), render(y(j)∼Θ)〉,∆(y(i)−y(j)))}Mi,j=1. Because the range of possible values ∆(y(i)−y(j)) can be large and sparse, we limit the range by clipping it to an interval [−c, c] (where c is a hyperparameter set to c= 5 for all the attributes in our evaluation). To fix a change larger than c, we perform sequence of small changes. For comparable attributes, the delta between two attribute values is defined as their distance ∆(y(i)k −y (j) k ) := d(y (i) k −y (j) k ). For uncomparable attributes, the delta is binary and determines whether the value is correct or not.
The model architecture used to represent π consists of two convolutional neural networks with shared weights θ, also called siamese neural networks, each of which computes a latent representation of the input image hx = fθ(I) and hr = fθ(render(y)). The function fθ has the same architecture as the attribute prediction network, except that we replace the fully connected layer with one that has a bigger size and remove the last softmax layer. Then, we combine the latent features hx and hr into a single vector h = [hx;hr;hx +hr;hx−hr;hx hr], where denotes element-wise multiplication. Finally, the vector h is passed to a fully connected layer of size 256 with ReLU activations, followed by a final softmax layer. Once the models are trained, we perform the refinement loop as follows:
Select attribute to change. As in general attributes interfere with each other, in each refinement iteration we adjust only a single attribute, which is chosen by sampling from the following distribution:
P [A = ai] = 1− π(∆yi = 0 | I, render(y))∑|A| k=1 1− π(∆yk = 0 | I, render(y))
where π(∆yk = 0 | I, render(y)) denotes the probability that the k-th attribute should not be changed, that is, the predicted change is zero. Since we train a separate model for each attribute, the probability that the given attribute should be changed is 1− π(∆yk = 0 | I, render(y)).
Select attribute’s new value. For comparable attributes, we adjust their value by sampling from the probability distribution computed by the policy π, which contains changes in range [−c, c]. For uncomparable attributes another approach has to be chosen, since the delta prediction network computes only whether the attribute is correct and not how to change it. Instead, we select the new value by sampling from the probability distribution computed by the corresponding attribute prediction network.
Accept or reject the change. In a typical setting, we would accept the proposed changes as long as the model predicts that an attribute should be changed. However, in our domain we can render the proposed changes and check whether the result is consistent with the model. Concretely, we accept the change y′ if it reduces the cost, that is, cost(I, render(y′))< cost(I, render(y)). Note that this optimization is possible only if the change was supposed to fix the attribute value, that is, the change was in the range (−c, c) or the attribute is uncomparable.

heading: 5 EVALUATION

text:
To evaluate the effectiveness of our approach, we apply it to the task of generating Android Button implementations. Concretely, we predict the following 12 attributes – border color, border width, border radius, height, width, padding, shadow, main color, text color, text font type, text gravity and text size. We do not predict the text content for which specialized models already exist (Jaderberg et al., 2016; Lyu et al., 2018; Jaderberg et al., 2014; Gupta et al., 2016). We provide domains and the visualization of all these attributes in Appendix A. In what follows we first describe our datasets and evaluation metrics, then we present a detailed evaluation of our approach consisting of the attribute prediction network (Section 4.1) and the refinement loop (Section 4.2).
Datasets To train the attribute prediction network we use a synthetic datasetD containing≈20,000 images and their corresponding attributes as described in Section 4. To train the refinement loop we use a second synthetic dataset D∆, also containing ≈20,000 image pairs. During training we perform two iterations of DAgger (Ross et al., 2011), each of which generates ≈20,000 samples obtained by running the policy on the initial training dataset. To evaluate our models we use two datasets – (i) syntheticDsyn generated in the same way as for training, and (ii) real-worldDgplay we obtained by manually implementing 110 buttons in existing Google Play Store applications. The illustration of samples and our inferred implementations for both datasets are provided in Appendix F.
Evaluation metrics To remove clutter, we introduce an uniform unit to measure attribute similarity called perceivable difference. We say that two attributes have the same (=) perceivable difference if their values are the same or almost indistinguishable. For example, the text size is perceivably the same, if the distance of the predicted y and the ground-truth y∗ value is d(y, y∗) ≤ 1, while the border width is perceivably the same only if it is predicted perfectly, i.e., d(y, y∗) = 0. The formal definition of perceivable difference with visualizations of all attributes is provided in Appendix E.

heading: 5.1 ATTRIBUTE PREDICTION

text:
A detailed summary of the variations of our attribute prediction models and their effect on performance is shown in Table 1. To enable easy comparison, we selected a good performing instantiation of our models, denoted as core, against which all variations in the rows (A)-(E) can be directly compared. Based on our experiments, we then select the best configuration that achieves accuracy 93.6% and 91.4% on the synthetic and the real-world datasets, respectively. All models were trained for 500 epochs, with early-stopping of 15 epochs, using a batch size of 128 and initial learning rate of 0.01. In what follows we provide a short discussion of each of the variations from Table 1.
Image background (A) We trained our models on synthetic datasets with three different component backgrounds of increasing complexity – white color, random solid color and user interface screenshot. Unsurprisingly, the models trained with white background fail to generalize to real-world datasets and achieve only 56.7% accuracy. Perhaps surprisingly, although the models trained with the screenshot background improve significantly, they also fail to generalize and achieve only 75.6% accuracy. Upon closer inspection, this is because overlaying components over existing images often introduces small visual artefacts around the component. On the other hand, random color backgrounds generalize well to real-world dataset as they have enough variety and no visual artefacts.
Affine transformations (B) Since the components can appear in any part of the input image, we use three methods to generate the training datasets – tr1 places the component randomly at any position with a margin of at least 20 pixels of the image border, tr2 places the component in the middle of the image with a horizontal offset x∼U(−13, 13) and vertical offset y∼U(−19, 19), and center always places the component exactly in the center. We can see that using either tr1 or tr2 leads to significantly more robust model and increases the real-world accuracy by ≈ 8%. Input image size & padding (C) As our goal is to perform pixel-accurate predictions, we do not scale down or resize the input images. However, since large images include additional noise (e.g., parts of the application unrelated to the predicted component), we measure how robust our model is to such noise by training models for three different input sizes – 135×310, 150×330 and 180×350. While the accuracy on the synthetic dataset is not affected, the real-world accuracy shows a slight decrease for larger sizes that contain more noise. However, note that the decrease is so small because of our padding technique, which extends the component to a larger size by padding the missing pixels with edge pixel values. When using no padding, the accuracy of the real-world dataset drops to 72% and when padding with a white color the accuracy drops even further to 71% (not shown in Table 1).
Color clipping (D) We experimented with different color clipping techniques – saliencytop5 that considers the top 5 colors in the saliency map of a given attribute, imagetop5 that considers the top 5 colors in the image, and imageall that considers all the colors in the image. The color clipping using saliency maps performs the best and leads to more than 3% and 16% improvements over other types of clipping or using no clipping, respectively. While the other types of color clipping also perform reasonably well, they typically fail for images that include many colors, where the saliency map helps focusing only on the colors relevant to the prediction. We note that the color clipping works best for components with solid colors and the improvement for gradient colors is limited.
Network architecture (E) We adapt the architecture presented in Section 4.1 for each attribute, by performing a small scale architecture search. Concretely, we choose between using classification (C) or regression (R), the kernel sizes, the number of output channels, whether we use pooling layer and whether we use additional fully connected layer before the softmax layer. Although the results in Table 1 are not directly comparable, as they provide only the aggregate accuracy over all attributes (additionally for regression experiments we consider only numerical attributes), they do show that such architectural choices have a significant impact on the network’s performance.

heading: 5.2 ATTRIBUTE REFINEMENT

text:
To evaluate our attribute refinement loop, we perform two experiments that refine the attribute values: (i) starting from random initial values, and (ii) starting from values computed by the attribute prediction network. For both experiments, we show that the refinement loop improves the accuracy of the predicted attribute values, as well as significantly outperforms other similarity metrics used as a baseline. We trained all our models for 500 epochs, with early-stopping of 15 epochs, using a batch size of 64, learning rate of 0.01 and gradient clipping of 3, which is necessary to make the training stable. Further, we initialize the siamese networks with the pretrained weights of the best attribute prediction network, which leads to both improved accuracy of 4%, as well as faster convergence when compared to training from scratch. Finally, we introduce a hyperparameter that controls which attributes are refined. This is useful as it allows the refinement loop to improve the overall accuracy even if only a subset of the attributes values can be refined successfully.
Attribute refinement improves accuracy The top row in Table 2 (right) shows the accuracy of our refinement loop when applied starting from values predicted by the best attribute prediction network. Based on our hyperparameters, we refined the following six attributes – text size, text gravity, text font, shadow, width and height. The refined attributes are additionally set to random initial values for the experiment in Table 2 (left). The overall improvement for both synthetic and real-world dataset is≈1.1% when starting from the values predicted by the attribute prediction network. When starting from random values the refinement loop can still recover predictions of almost the same quality, although with ≈ 12× more refinement iterations. The reason why the improvement is not higher is mainly because ≈ 5% of the errors are performed when predicting the text color and the text padding, for which both the attribute prediction networks and the refinement loop work poorly. This suggest that a better network architecture is needed to improve the accuracy of these attributes.
Effectiveness of our learned similarity metric To evaluate the quality of our learned cost function, which computes image similarity in the attribute space, we use the following similarity metrics as a baseline – pairwise pixel difference, structural similarity (Wang et al., 2004), and Wasserstein distance. As the baseline metrics depend heavily on the fact that the input components are aligned in the two images (e.g., when computing pairwise pixel difference), for a fair comparison we add a manual preprocessing step that centers the components in the input image. The results from Table 2 show that all of these metrics are significantly worse compared to our learned cost function. Even though they provide some improvement when starting from random attributes, the improvement is limited and all of them result in accuracy decrease when used starting from good attributes.

heading: 6 CONCLUSION

text: We present an approach for learning to infer user interface attributes from images. We instantiate it to the task of learning the implementation of the Android Button component and achieve 92.5% accuracy on a dataset consisting of Google Play Store applications. We show that this can be achieved by training purely using suitable datasets generated synthetically. This result indicates that our method is a promising step towards automating the process of user interface implementation.

heading: A ANDROID BUTTON ATTRIBUTES

text: We provide definition of all the attributes considered in our work as well as their visualization in Table 3. For border radius we use a special value∞ to denote round buttons.

heading: B REFINEMENT LOOP

text:
Stopping criterion For a given similarity metric (e.g., our learned attribute distance, pixel similarity, etc.) the stopping criterion of the refinement loop is defined using two hyperparameters:
• early stopping: stop if the similarity metric has not improved in the last n iterations. In our experiments we use n = 4.
• maximum number of iterations: stop if the maximum number of iterations was reached. In our experiments we set maximum number of iterations to 8 when starting from the best prediction and to 100 if starting from random attribute values.
Per-attribute accuracy We provide per-attribute accuracy of the refinement loop in Table 4. As described in Section 5.2, we refined the following six attributes – text size, text gravity, text font, shadow, width and height. When considering only these six attributes, the refinement loop improves the accuracy by 2.3%, from 93.5% to 95.8%.
The remaining errors are mainly due to text color, padding and text font attributes. The padding achieves accuracy 81% and is difficult to learn as it often interferes with the text alignment and the same position of the text can be achieved with different attribute values. The text font attribute accuracy is 85% which is slightly improved by using the refinement loop but could be improved further. The worst attribute overall is the text color that achieves 69% accuracy. Upon closer inspection, the predictions of the text color are challenging due to the fact that the text is typically narrow and when rendered, it does not have a solid background. Instead, the color is an interpolation of the text color and background color as computed by the anti-aliasing in the rendering engine.
Attribute dependencies All the results presented in Table 4, as well as in our evaluation, are computed starting from the initial set of attribute values that typically contains mistakes. Since different attributes can (and do) dependent on each other, mispredicting one attribute can negatively affect the predictions of other attributes. As a concrete example, refining text font while assuming that all the other attribute values are correct leads to an improvement of 12% which is significantly higher than ≈2% from Table 4. To partially address this issue, the refinement loop uses a heuristic which ensures that all attributes have different values when used as input to the refinement loop. Concretely, if two attributes would have the same value, one of the values is temporarily changed to a random valid value and returned to the original value at the end of each refinement iteration.
C IMAGE PADDING
To improve robustness of our models on real-world images we experimented with three techniques of image padding shown in Figure 2. In (a) the image is padded with the edge values, in (b) the image is padded with a constant solid color and in (c) the image is simply extended to the required input size.

heading: D COLOR CLIPPING USING SALIENCY MAPS

text: To improve color clipping results we are limiting the colors to which the predicted colors can be clipped by only considering the top 5 colors within the thresholded saliency map of the input image. An illustration of this process is shown in Figure 3, where (a) shows an initial input image, (b) the saliency map of the prediction, and (c) and (d) the thresholded saliency map (we use threshold 0.8) and the colors it contains.

heading: E PERCEIVABLE ATTRIBUTE DIFFERENCE

text: We define the perceivable difference for each attribute in Table 5. We use to denote the distance between two attribute values. For all numerical attributes except colors, the distance is defined as the attribute value difference, i.e., d(yi, yj) = yi − yj . To better capture the difference between colors, we define their distance using the CIE76 formula (Schanda, 2007), denoted as dE. Furthermore, we provide illustration of the worse case perceivable difference for each attribute in Table 6.

heading: F DATASETS AND INFERRED IMPLEMENTATION VISUALIZATIONS

text: We provide illustrations of our approach for inferring Android Button implementations from images. Concretely, we include examples of images for which our approach works well, as well as examples where our models make mistakes. The visualizations for the synthetic Dsyn and realworld Dgplay dataset of buttons found in Google Play Store applications are shown in Table 7 and Table 8, respectively. Each table row is divided into 4 parts: an image of the input, the preprocessed input image, a rendering of the predicted Button and a rendering of the refined Button.