LEARNING ROBUST REPRESENTATIONS BY PROJECT- ING SUPERFICIAL STATISTICS OUT
Despite impressive performance as evaluated on i.i.d. holdout data, deep neural networks depend heavily on superficial statistics of the training data and are liable to break under distribution shift. For example, subtle changes to the background or texture of an image can break a seemingly powerful classifier. Building on previous work on domain generalization, we hope to produce a classifier that will generalize to previously unseen domains, even when domain identifiers are not available during training. This setting is challenging because the model may extract many distribution-specific (superficial) signals together with distribution-agnostic (semantic) signals. To overcome this challenge, we incorporate the gray-level cooccurrence matrix (GLCM) to extract patterns that our prior knowledge suggests are superficial: they are sensitive to texture but unable to capture the gestalt of an image. Then we introduce two techniques for improving our networks' outof-sample performance. The first method is built on the reverse gradient method that pushes our model to learn representations from which the GLCM representation is not predictable. The second method is built on the independence introduced by projecting the model's representation onto the subspace orthogonal to GLCM representation's. We test our method on battery of standard domain generalization data sets and, interestingly, achieve comparable or better performance as compared to other domain generalization methods that explicitly require samples from the target distribution for training.
Summary:
The paper proposes an unsupervised approach to identify image features that are not meaningful for image classification tasks. The goal is to address the domain adaptation (DA)/domain generalization (DG) issue. The paper introduces a new learning task where the domain identity is unavailable during training, called unguided domain generalization (UDG). The proposed approach is based on an old method of using gray level co-occurence matrix, updated to allow for differentiable training. This new approach is used in two different ways to reduce the effect of background texture in a classification task. The paper introduces a new dataset, and shows extensive and carefully designed experiments using the new data as well as existing domain generalization datasets.
This paper revisits an old idea from image processing in a new way, and provides an interesting unsupervised method for identifying so called superficial features. The proposed block seems to be very modular in design, and can be plugged into other architectures. The main weakness is that it is a bit unclear exactly what is being assumed as "background texture" by the authors.
Overall comments:
- Some more clarity on what you mean by superficial statistics would be good. E.g. by drawing samples. Are you assuming the object is centered? Somehow filling the image?  Different patch statistics? How about a texture classification task?
- please derive why NGLCM reduces to GLCM in the appendix. Also show the effect of dropping the uniqueness constraint.
- Section 3.2: I assume you are referring to an autoencoder style architecture here. Please rewrite the first paragraph. The current setup seems to indicate that you doing supervised training, since you have labels y, but then you talk about decoder and encoder.
- Section 3.2: Please expand upon why you use F_L for training but F_P during testing
Minor typos/issues:
- Last bullet in Section 1: DG not yet defined, only defined in Section 2.
- page 2, Section 2, para 1: data collection conduct. Please reword.
- page 2, Section 2, para 2: Sentence: For a machine learning ... There is no object in this sentence. Not sure what you are trying to define.
- page 2, Section 2, para 2: Is $\mathcal{S}$ and $\mathcal{T}$ not intersecting?
- page 2, Section 2.1: Heckman (1977), use \citep
- page 2, Section 2.1: Manski, citep and missing year
- page 3, Section 2.1: Kumagai, use citet
- page 3, Section 3.1: We first expand ... --> We first flatten A into a row vector
- page 4, Section 3.1: b is undefined. I assume you mean d?
- page 4, Section 3.1: twice: contrain --> constraint
- page 4, Section 3.2: <X,y> --> {X,y} as used in Section 3.1.
- page 4, Section 3.2, just below equation: as is introduced in the previous section. New sentence about MLP please. And MLP not defined.
- page 4, Section 3.2, next paragraph: missing left bracket (
- page 4, Section 3.2: inferred from its context.
- page 5, Section 4: popular DG method (DANN)
- page 7: the rest one into --> the remaining one into
- page 8: rewrite: when the empirical performance interestingly preserves.
- page 8, last sentence: GD --> DG
- A2.2: can bare with. --> can deal with.
- A2.2: linear algebra and Kailath Variant. Unsure what you are trying to say.
- A2.2: sensitive to noises --> sensitive to noise.