Domain shifts are frequently encountered in real-world scenarios. In this paper, we consider the problem of domain-invariant deep learning by explicitly modeling domain shifts with only a small amount of domain-specific parameters in a Convolutional Neural Network (CNN). By exploiting the observation that a convolutional filter can be well approximated as a linear combination of a small set of basis elements, we show for the first time, both empirically and theoretically, that domain shifts can be effectively handled by decomposing a regular convolutional layer into a domain-specific basis layer and a domain-shared basis coefficient layer, while both remain convolutional. An input channel will now first convolve spatially only with each respective domain-specific basis to "absorb" domain variations, and then output channels are linearly combined using common basis coefficients trained to promote shared semantics across domains. We use toy examples, rigorous analysis, and real-world examples to show the framework's effectiveness in cross-domain performance and domain adaptation. With the proposed architecture, we need only a small set of basis elements to model each additional domain, which brings a negligible amount of additional parameters, typically a few hundred.
The paper proposes an approach to learning domain invariant representations using the adaptive decomposition of the convolutional filters. The approach is similar to methods that use multi-stream networks (a stream for each domain), but using the filer decomposition scheme, the authors avoid the issue of excessive increase in the number of parameters typical in fully multi-stream architectures. 
This is achieved by learning a separate basis for convolutional filters for each domain while sharing the basis coefficients across domains. This encourages shared semantics across domains while maintaining a balance between the network expressiveness and the computational complexity.
The authors argue that the basis learned for each domain can be understood as correction/alignment mappings to bring together the representations of each domain. A toy example presented is convincing and shows the correction basis learned in a simple synthetic case. Theoretical arguments are provided to show that the proposed correction scheme covers a large range of possible domain shifts.
The authors also show that plugging the decomposition scheme into existing CNN based unsupervised domain adaptation algorithms results in consistent improvements across methods and datasets. 
How does the method compare to approaches that learn to adapt the representations using conditional/adaptive batch norm [1,2]? 
Overall, the paper was well motivated and easy to read. The methods appear to be a useful addition to tools available for domain invariant learning.
[1] Chang, Woong-Gi, Tackgeun You, Seonguk Seo, Suha Kwak, and Bohyung Han. "Domain-Specific Batch Normalization for Unsupervised Domain Adaptation." In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7354-7362. 2019.
[2] Kumar, Abhishek, Prasanna Sattigeri, Kahini Wadhawan, Leonid Karlinsky, Rogerio Feris, Bill Freeman, and Gregory Wornell. "Co-regularized alignment for unsupervised domain adaptation." In Advances in Neural Information Processing Systems, pp. 9345-9356. 2018.