THIRD-PERSON IMITATION LEARNING
Reinforcement learning (RL) makes it possible to train agents capable of achieving sophisticated goals in complex and uncertain environments. A key difficulty in reinforcement learning is specifying a reward function for the agent to optimize. Traditionally, imitation learning in RL has been used to overcome this problem. Unfortunately, hitherto imitation learning methods tend to require that demonstrations are supplied in the first-person: the agent is provided with a sequence of states and a specification of the actions that it should have taken. While powerful, this kind of imitation learning is limited by the relatively hard problem of collecting first-person demonstrations. Humans address this problem by learning from third-person demonstrations: they observe other humans perform tasks, infer the task, and accomplish the same task themselves. In this paper, we present a method for unsupervised third-person imitation learning. Here third-person refers to training an agent to correctly achieve a simple goal in a simple environment when it is provided a demonstration of a teacher achieving the same goal but from a different viewpoint; and unsupervised refers to the fact that the agent receives only these third-person demonstrations, and is not provided a correspondence between teacher states and student states. Our methods primary insight is that recent advances from domain confusion can be utilized to yield domain agnostic features which are crucial during the training process. To validate our approach, we report successful experiments on learning from third-person demonstrations in a pointmass domain, a reacher domain, and inverted pendulum.
1 INTRODUCTION
Reinforcement learning (RL) is a framework for training agents to maximize rewards in large, unknown, stochastic environments. In recent years, combining techniques from deep learning with reinforcement learning has yielded a string of successful applications in game playing and robotics Mnih et al. (2015; 2016); Schulman et al. (2015a); Levine et al. (2016). These successful applications, and the speed at which the abilities of RL algorithms have been increasing, makes it an exciting area of research with significant potential for future applications.
One of the major weaknesses of RL is the need to manually specify a reward function. For each task we wish our agent to accomplish, we must provide it with a reward function whose maximizer will precisely recover the desired behavior. This weakness is addressed by the field of Inverse Reinforcement Learning (IRL). Given a set of expert trajectories, IRL algorithms produce a reward function under which these the expert trajectories enjoy the property of optimality. Recently, there has been a significant amount of work on IRL, and current algorithms can infer a reward function from a very modest number of demonstrations (e.g,. Abbeel & Ng (2004); Ratliff et al. (2006); Ziebart et al. (2008); Levine et al. (2011); Ho & Ermon (2016); Finn et al. (2016)).
While IRL algorithms are appealing, they impose the somewhat unrealistic requirement that the demonstrations should be provided from the first-person point of view with respect to the agent. Human beings learn to imitate entirely from third-person demonstrations – i.e., by observing other humans achieve goals. Indeed, in many situations, first-person demonstrations are outright impossible to obtain. Meanwhile, third-person demonstrations are often relatively easy to obtain.
The goal of this paper is to develop an algorithm for third-person imitation learning. Future advancements in this class of algorithms would significantly improve the state of robotics, because it will enable people to easily teach robots news skills and abilities. Importantly, we want our algorithm to be unsupervised: it should be able to observe another agent perform a task, infer that there is an underlying correspondence to itself, and find a way to accomplish the same task.
We offer an approach to this problem by borrowing ideas from domain confusion Tzeng et al. (2014) and generative adversarial networks (GANs) Goodfellow et al. (2014). The high-level idea is to introduce an optimizer under which we can recover both a domain-agnostic representation of the agent’s observations, and a cost function which utilizes this domain-agnostic representation to capture the essence of expert trajectories. We formulate this as a third-person RL-GAN problem, and our solution builds on the first-person RL-GAN formulation by Ho & Ermon (2016).
Surprisingly, we find that this simple approach has been able to solve the problems that are presented in this paper (illustrated in Figure 1), even though the student’s observations are related in a complicated way to the teacher’s demonstrations (given that the observations and the demonstrations are pixel-level). As techniques for training GANs become more stable and capable, we expect our algorithm to be able to infer solve harder third-person imitation tasks without any direct supervision.
2 RELATED WORK
Imitation learning (also learning from demonstrations or programming by demonstration) considers the problem of acquiring skills from observing demonstrations. Imitation learning has a long history, with several good survey articles, including (Schaal, 1999; Calinon, 2009; Argall et al., 2009). Two main lines of work within imitation learning are: 1) behavioral cloning, where the demonstrations are used to directly learn a mapping from observations to actions using supervised learning, potentially with interleaving learning and data collection (e.g., Pomerleau (1989); Ross et al. (2011)). 2) Inverse reinforcement learning (Ng et al., 2000), where a reward function is estimated that explains the demonstrations as (near) optimal behavior. This reward function could be represented as nearness to a trajectory (Calinon et al., 2007; Abbeel et al., 2010), as a weighted combination of features (Abbeel & Ng, 2004; Ratliff et al., 2006; Ramachandran & Amir, 2007; Ziebart et al., 2008; Boularias et al., 2011; Kalakrishnan et al., 2013; Doerr et al., 2015), or could also involve feature learning (Ratliff et al., 2007; Levine et al., 2011; Wulfmeier et al., 2015; Finn et al., 2016; Ho & Ermon, 2016).
This past work, however, is not directly applicable to the third person imitation learning setting. In third-person imitation learning, the observations and actions obtained from the demonstrations are not the same as what the imitator agent will be faced with. A typical scenario would be: the imitator agent watches a human perform a demonstration, and then has to execute that same task. As discussed in Nehaniv & Dautenhahn (2001) the ”what and how to imitate” questions become significantly more challenging in this setting. To directly apply existing behavioral cloning or inverse reinforcement learning techniques would require knowledge of a mapping between observations and actions in the demonstrator space to observations and actions in the imitator space. Such a mapping is often difficult to obtain, and it typically relies on providing feature representations that captures the invariance between both environments Carpenter et al. (2002); Shon et al. (2005); Calinon et al. (2007); Nehaniv (2007); Gioioso et al. (2013); Gupta et al. (2016). Contrary to prior work, we consider third-person imitation learning from raw sensory data, where no such features are made available.
The most closely related work to ours is by Finn et al. (2016); Ho & Ermon (2016); Wulfmeier et al. (2015), who also consider inverse reinforcement learning directly from raw sensory data. However, the applicability of their approaches is limited to the first-person setting. Indeed, matching raw sensory observations is impossible in the 3rd person setting.
Our work also closely builds on advances in generative adversarial networks Goodfellow et al. (2014), which are very closely related to imitation learning as explained in Finn et al. (2016); Ho & Ermon (2016). In our optimization formulation, we apply the gradient flipping technique from Ganin & Lempitsky (2014).
The problem of adapting what is learned in one domain to another domain has been studied extensively in computer vision in the supervised learning setting Yang et al. (2007); Mansour et al. (2009); Kulis et al. (2011); Aytar & Zisserman (2011); Duan et al. (2012); Hoffman et al. (2013); Long & Wang (2015). It has also been shown that features trained in one domain can often be relevant to other domains Donahue et al. (2014). The work most closely related to ours is Tzeng et al. (2014; 2015), who also consider an explicit domain confusion loss, forcing trained classifiers to rely on features that don’t allow to distinguish between two domains. This work in turn relates to earlier work by Bromley et al. (1993); Chopra et al. (2005), which also considers supervised training of deep feature embeddings.
Our approach to third-person imitation learning relies on reinforcement learning from raw sensory data in the imitator domain. Several recent advances in deep reinforcement learning have made this practical, including Deep Q-Networks (Mnih et al., 2015), Trust Region Policy Optimization (Schulman et al., 2015a), A3C Mnih et al. (2016), and Generalized Advantage Estimation (Schulman et al., 2015b). Our approach uses Trust Region Policy Optimization.
3 BACKGROUND AND PRELIMINARIES
A discrete-time finite-horizon discounted Markov decision process (MDP) is represented by a tuple M = (S,A,P, r, ρ0, γ, T ), in which S is a state set, A an action set, P : S × A × S → R+ a transition probability distribution, r : S × A → R a reward function, ρ0 : S → R+ an initial state distribution, γ ∈ [0, 1] a discount factor, and T the horizon. In the reinforcement learning setting, the goal is to find a policy πθ : S × A → R+ parametrized by θ that maximizes the expected discounted sum of rewards incurred, η(πθ) = Eπθ [ ∑T t=0 γ
tc(st)], where s0 ∼ ρ0(s0), at ∼ πθ(at|st), and st+1 ∼ P(st+1|st, at). In the (first-person) imitation learning setting, we are not given the reward function. Instead we are given traces (i.e., sequences of states traversed) by an expert who acts according to an unknown policy πE . The goal is to find a policy πθ that performs as well as the expert against the unknown reward function. It was shown in Abbeel & Ng (2004) that this can be achieved through inverse reinforcement learning by finding a policy πθ that matches the expert’s empirical expectation over discounted sum of all features that might contribute to the reward function. The work by Ho & Ermon (2016) generalizes this to the setting when no features are provided as follows: Find a policy πθ that makes it impossible for a discriminator (in their work a deep neural net) to distinguish states visited by the expert from states visited by the imitator agent. This can be formalized as follows:
max πθ min DR − Eπθ [logDR(s)]− EπE [log(1−DR(s))] (1)
Here, the expectations are over the states experienced by the policy of the imitator agent, πθ, and by the policy of the expert, πE , respectively. DR is the discriminator, which outputs the probability of a state having originated from a trace from the imitator policy πθ. If the discriminator is perfectly able to distinguish which policy originated state-action pairs, then DR will consistently output a probability of 1 in the first term, and a probability of 0 in the second term, making the objective its lowest possible value of zero. It is the role of the imitator agent πθ to find a policy that makes it difficult for the discriminator to make that distinction. The desired equilibrium has the imitator agent making it impractical for the discriminator to distinguish, hence forcing the discriminator to assign probability 0.5 in all cases. Ho & Ermon (2016) present a practical approach for solving this type of game when representing both πθ and DR as deep neural networks. Their approach repeatedly performs gradient updates on each of them. Concretely, for a current policy πθ traces can be collected, which together with the expert traces form a data-set on which DR can be trained with supervised learning minimizing the negative log-likelihood (in practice only performing a modest number of updates). For a fixed DR, this is a policy optimization problem where − logDR(s, a) is the reward, and policy gradients can be computed from those same traces. Their approach uses trust region policy optimization (Schulman et al., 2015a) to update the imitator policy πθ from those gradients.
In our work we will have more terms in the objective, so for compactness of notation, we will realize the discriminative minimization from Eqn. (1) as follows:
max πθ min DR LR = ∑ i CE(DR(si), c`i) (2)
Where si is state i, c`i is the correct class label (was the state si obtained from an expert vs. from a non-expert), and CE is the standard cross entropy loss.
4 A FORMAL DEFINITION OF THE THIRD-PERSON IMITATION LEARNING PROBLEM
Formally, the third-person imitation learning problem can be stated as follows. Suppose we are given two Markov Decision Processes MπE and Mπθ . Suppose further there exists a set of traces ρ = {(s1, . . . , sn)}ni=0 which were generated under a policy πE acting optimally under some unknown reward RπE . In third-person imitation learning, one attempts to recover by proxy through ρ a policy πθ = f(ρ) which acts optimally with respect to Rπθ .
5 A THIRD-PERSON IMITATION LEARNING ALGORITHM
5.1 GAME FORMULATION
In this section, we discuss a simple algorithm for third-person imitation learning. This algorithm is able to successfully discriminate between expert and novice policies, even when the policies are executed under different environments. Subsequently, this discrimination signal can be used to train expert policies in new domains via RL by training the novice policy to fool the discriminator, thus forcing it to match the expert policy.
In third-person learning, observations are more typically available rather than direct state access, so going forward we will work with observations ot instead of states st as representing the expert traces. The top row of Figure 8 illustrates what these observations are like in our experiments.
We begin by recalling that in the algorithm proposed by Ho & Ermon (2016) the loss in Equation 2 is utilized to train a discriminator DR capable of distinguishing expert vs non-expert policies. Unfortunately, (2) will likely fail in cases when the expert and non-expert act in different environments, since DR will quickly learn these differences and use them as a strong classification signal. To handle the third-person setting, where expert and novice are in different environments, we consider that DR works by first extracting features from ot, and then using these features to make a
classification. Suppose then that we partition DR into a feature extractor DF and the actual classifier which assigns probabilities to the outputs of DF . Overloading notation, we will refer to the classifier as DR going forward. For example, in case of a deep neural net representation, DF would correspond to the earlier layers, and DR to the later layers. The problem is then to ensure that DF contains no information regarding the rollout’s domain label d` (i.e., expert vs. novice domain). This can be realized as
max πθ minLR = ∑ i CE(DR(DF (oi)), c`i)
s.t. MI(DF (oi); dl) = 0
Where MI is mutual information and hence we have abused notation by using DR, DF , and d` to mean the classifier, feature extractor, and the domain label respectively as well as distributions over these objects.
The mutual information term can be instantiated by introducing another classifier DD, which takes features produced by DF and outputs the probability that those features were produced by in the expert vs. non-expert environment. (See Bridle et al. (1992); Barber & Agakov (2005); Krause et al. (2010); Chen et al. (2016) for further discussion on instantiating the information term by introducing another classifier.) If σi = DF (oi), then the problem can be written as
max πθ min DR max DD LR + LD = ∑ i CE(DR(σi), c`i) + CE(DD(σi), d`i) (3)
In words, we wish to minimize class loss while maximizing domain confusion.
Often, it can be difficult for even humans to judge a static image as expert vs. non-expert because it does not convey any information about the environmental change affected by the agent’s actions. For example, if a pointmass is attempting to move to a target location and starts far away from its goal state, it can be difficult to judge if the policy itself is bad or the initialization was simply unlucky. In response to this difficulty, we give DR access to not only the image at time t, but also at some future time t + n. Define σt = DF (ot) and σt+n = DF (ot+n). The classifier then makes a prediction DR(σt, σt+n) = ĉ`. This renders the following formulation:
max πθ min DR max DD LR + LD = ∑ i CE(DR(σi, σi+n), c`i) + CE(DD(σi), d`i) (4)
Note we also want to optimize overDF , the feature extractor, but it feeds both intoDR and intoDD, which are competing (hidden under σ), which we will address now.
To deal with the competition over DF , we introduce a function G that acts as the identity when moving forward through a directed acyclic graph and flips the sign when backpropagating through the graph. This technique has enjoyed recent success in computer vision. See, for example, (Ganin & Lempitsky, 2014). With this trick, the problem reduces to its final form
max πθ min DR,DD,DF LR + LD = ∑ i CE(DR(σi, σi+n), c`i) + λ CE(DD(G(σi), d`i) (5)
In Equation (5), we flip the gradient’s sign during backpropagation ofDF with respect to the domain classification loss. This corresponds to stochastic gradient ascent away from features that are useful for domain classification, thus ensuring that DF produces domain agnostic features. Equation 5 can be solved efficiently with stochastic gradient descent. Here λ is a hyperparameter that determines the trade-off made between the objectives that are competing over DF . To ensure sufficient signal for discrimination between expert and non-expert, we collect third-person demonstrations in the expert domain from both an expert and from a non-expert.
Our complete formulation is graphically summarized in Figure 2.
5.2 ALGORITHM
To solve the game formulation in Equation (5), we perform alternating (partial) optimization over the policy πθ and the reward function and domain confusion encoded through DR,DD,DF . The optimization over DR,DD,DF is done through stochastic gradient descent with ADAM Kingma & Ba (2014).
Our generator (πθ) step is similar to the generator step in the algorithm by (Ho & Ermon, 2016). We simply use − logDR as the reward. Using policy gradient methods (TRPO), we train the generator to minimize this cost and thus push the policy further towards replicating expert behavior. Once the generator step is done, we start again with the discriminator step. The entire process is summarized in algorithm 1.
6 EXPERIMENTS
We seek to answer the following questions through experiments:
1. Is it possible to solve the third-person imitation learning problem in simple settings? I.e., given a collection of expert image-based rollouts in one domain, is it possible to train a policy in a different domain that replicates the essence of the original behavior?
2. Does the algorithm we propose benefit from both domain confusion and velocity?
3. How sensitive is our proposed algorithm to the selection of hyper-parameters used in deployment?
4. How sensitive is our proposed algorithm to changes in camera angle?
5. How does our method compare against some reasonable baselines?
Algorithm 1 A third-person imitation learning algorithm. 1: Let CE be the standard cross entropy loss. 2: Let G be a function that flips the gradient sign during backpropogation and acts as the identity
map otherwise. 3: Initialize two domains, E and N for the expert and novice. 4: Initialize a memory bank Ω of expert success and of failure in domainE. Each trajectory ω ∈ Ω
comprises a rollout of images o = o1, . . . , ot, . . . on, a class label c`, and a domain label d`. 5: Initialize D = DF ,DR,DD, a domain invariant discriminator. 6: Initialize a novice policy πθ. 7: Initialize numiters, the number of inner policy optimization iterations we wish to run. 8: for iter in numiters do 9: Sample a set of successes and failures ωE from Ω.
10: Collect on policy samples ωN 11: Set ω = ωE ∪ ωN . 12: Shuffle ω 13: for o, c`, d` in ω do 14: for ot in o do 15: σt = DF (ot) 16: σt+4 = DF (ot+4) 17: LR = CE(DR(σt, σt+4), c`) 18: Ld = CE(DD(G(σt)), d`) 19: L = λ · Ld + LR 1 20: minimize L with ADAM. 21: end for 22: end for 23: Collect on policy samples ωN from πθ. 24: for ω in ωN do 25: for ωt in ω do 26: σt = DF (ot) 27: σt+4 = DF (ot+4) 28: ĉ` = DR(σt, σt+4) 29: r = ĉ`[0], the probability that ot, ot+4 were generated via expert rollouts. 30: Use r to train πθ with via policy gradients (TRPO). 31: end for 32: end for 33: end for 34: return optimized policy πθ
6.1 ENVIRONMENTS
To evaluate our algorithm, we consider three environments in the MuJoCo physics simulator. There are two different versions of each environment, an expert variant and a novice variant. Our goal is to train a cost function that is domain agnostic, and hence can be trained with images on the expert domain but nevertheless produce a reasonable cost on the novice domain. See Figure 1 for a visualization of the differences between expert and novice environments for the three tasks.
Point: A pointmass attempts to reach a point in a plane. The color of the target and the camera angle change between domains.
Reacher: A two DOF arm attempts to reach a designated point in the plane. The camera angle, the length of the arms, and the color of the target point are changed between domains. Note that changing the camera angle significantly alters the image background color from largely gray to roughly 30 percent black. This presents a significant challenge for our method.
Inverted Pendulum: A classic RL task wherein a pendulum must be made to balance via control. For this domain, We only change the color of the pendulum and not the camera angle. Since there is no target point, we found that changing the camera angle left the domain invariant representations with too little information and resulted in a failure case. In contrast to some traditional renderings
of this problem, we do not terminate an episode when the agent falls but rather allow data collection to continue for a fixed horizon.
6.2 EVALUATIONS
Is it possible to solve the third-person imitation learning problem in simple settings? In Figure 3, we see that our proposed algorithm is indeed able to recover reasonable policies for all three tasks we examined. Initially, the training is quite unstable due to the domain confusion wreaking havoc on the learned cost. However, after several iterations the policies eventually head towards reasonable local minima and the standard deviation over the reward distribution shrinks substantially. Finally, we note that the extracted feature representations used to complete this task are in fact domain-agnostic, as seen in Figure 9. Hence, the learning is properly taking place from a third-person perspective.
Does the algorithm we propose benefit from both domain confusion and the multi-time step input? We answer this question with the experiments summarized in Figure 5. This experiment compares our approach with: (i) our approach without the domain confusion loss; (ii) our approach without the multi-time step input; (iii) our approach without the domain confusion loss and without the multitime step input (which is very similar to the approach in Ho & Ermon (2016)). We see that adding domain confusion is essential for getting strong performance in all three experiments. Meanwhile, adding multi-time step input marginally improves the results. See also Figure 7 for an analysis of the effects of multi-time step input on the final results.
How sensitive is our proposed algorithm to the selection of hyper-parameters used in deployment? Figure 6 shows the effect of the domain confusion coefficient λ, which trades off how much we should weight the domain confusion objective vs. the standard cost-recovery objective, on the final performance of the algorithm. Setting λ too low results in slower learning and features that are not domain-invariant. Setting λ too high results in an objective that is too quick to destroy information, which makes it impossible to recover an accurate cost.
For multi-time step input, one must choose the number of look-ahead frames that are utilized. If too small a window is chosen, the agent’s actions have not affected a large amount of change in the environment and it is difficult to discern any additional class signal over static images. If too large a time-frame passes, causality becomes difficult to interpolate and the agent does worse than simply being trained on static frames. Figure 7 illustrates that no number of look-ahead frames is consistently optimal across tasks. However, a value of 4 showed good performance over all tasks, and so this value was utilized in all other experiments.
How sensitive is our algorithm to changes in camera angle? We present graphs for the reacher and point experiments wherein we exam the final reward obtained by a policy trained with thirdperson imitation learning vs the camera angle difference between the first-person and third-person perspective. We omit the inverted double pendulum experiment, as the color and not the camera angle changes in that setting and we found the case of slowly transitioning the color to be the definition of uninteresting science.
How does our method compare against reasonable baselines? We consider the following baselines for comparisons against third-person imitation learning. 1) Standard reinforcement learning with using full state information and the true reward signal. This agent is trained via TRPO. 2)
Standard GAIL (first-person imitation learning). Here, the agent receives first-person demonstration and attempts to imitate the correct behavior. This is an upper bound on how well we can expect to do, since we have the correct perspective. 3) Training a policy using first-person data and applying it to the third-person environment.
We compare all three of these baselines to third-person imitation learning. As we see in figure 9: 1) Standard RL, which (unlike the imitation learning approaches) has access to full state and true reward, helps calibrate performance of the other approaches. 2) First-person imitation learning is faced with a simpler imitation problem and accordingly outperforms third-person imitation, yet third-person imitation learning is nevertheless competitive. 3) Applying the first-person policy to the third-person agent fails miserably, illustrating that explicitly considering third-person imitation is important in these settings.
Somewhat unfortunately, the different reward function scales make it difficult to capture information on the variance of each learning curve. Consequently, in Appendix A we have included the full learning curves for these experiments with variance bars, each plotted with an appropriate scale to examine the variance of the individual curves.
7 DISCUSSION AND FUTURE WORK
In this paper, we presented the problem of third-person imitation learning. We argue that this problem will be important going forward, as techniques in reinforcement learning and generative adversarial learning improve and the cost of collecting first-person samples remains high. We presented an algorithm which builds on Generative Adversarial Imitation Learning and is capable of solving simple third-person imitation tasks.
One promising direction of future work in this area is to jointly train policy features and cost features at the pixel level, allowing the reuse of image features. Code to train a third person imitation learning agent on the domains from this paper is presented here: https://github.com/bstadie/ third_person_im
ACKNOWLEDGEMENTS
This work was done partially at OpenAI and partially at Berkeley. Work done at Berkeley was supported in part by Darpa under the Simplex program and the FunLoL program.
8 APPENDIX A: LEARNING CURVES FOR BASELINES
Here, we plot the learning curves for each of the baselines mentioned in the experiments section as a standalone plot. This allows one to better examine the variance of each individual learning curve.
9 APPENDIX B: ARCHITECTURE PARAMETERS
Joint Feature Extractor: Input is images are size 50 x 50 with 3 channels, RGB. Layers are 2 convolutional layers each followed by a max pooling layer of size 2. Layers use 5 filters of size 3 each.
Domain Discriminator and the Class Discriminator: Input is domain agnostic output of convolutional layers. Layers are two feed forward layers of size 128 followed by a final feed forward layer of size 2 and a soft-max layer to get the log probabilities.
ADAM is used for discriminator training with a learning rate of 0.001. The RL generator uses the off-the-shelf TRPO implementation available in RLLab.

The paper presents an interesting new problem setup for imitation learning: an agent tries to imitate a trajectory demonstrated by an expert but said trajectory is demonstrated in a different state or observation space than the one accessible by the agent (although the dynamics of the underlying MDP are shared). The paper proposes a solution strategy that combines recent work on domain confusion losses with a recent IRL method based on generative adversarial networks.
I believe the general problem to be relevant and agree with the authors that it results in a more natural formulation for imitation learning that might be more widely applicable.
There are however a few issues with the paper in its current state that make the paper fall short of being a great exploration of a novel idea. I will list these concerns in the following (in arbitrary order)
- The paper feels at times to be a bit hurriedly written (this also mainly manifests itself in the experiments, see comment below) and makes a few fairly strong claims in the introduction that in my opinion are not backed up by their approach. For example: "Advancements in this class of algorithms would significantly improve the state of robotics, because it will enable anyone to easily teach robots new skills"; given that the current method to my understanding has the same issues that come with standard GAN training (e.g. instability etc.) and requires a very accurate simulator to work well (since TRPO will require a large number of simulated trajectories in each step) this seems like an overstatement.
  There are some sentences that are ungrammatical or switch tense in the middle of the sentence making the paper harder to read than necessary, e.g. Page 2: "we find that this simple approach has been able to solve the problems"
- The general idea of third person imitation learning is nice, clear and (at least to my understanding) also novel. However, instead of exploring how to generally adapt current IRL algorithms to this setting the authors pick a specific approach that they find promising (using GANs for IRL) and extend it. A significant amount of time is then spent on explaining why current IRL algorithms will fail in the third-person setting. I fail to see why the situation for the GAN based approach is any different than that of any other existing IRL algorithm. To be more clear: I see no reason why e.g. behavioral cloning could not be extended with a domain confusion loss in exactly the same way as the approach presented. To this end it would have been nice to rather discuss which algorithms can be adapted in the same way (and also test them) and which ones cannot. One straightforward approach to apply any IRL algorithm would for example be to train two autoencoders for both domains that share higher layers + a domain confusion loss on the highest layer, should that not result in features that are directly usable? If not, why?
- While the general argument that existing IRL algorithms will fail in the proposed setting seems reasonable it is still unfortunate that no attempts have been made to validate this empirically. No comparison is made regarding what happens when one e.g. performs supervised learning (behavioral cloning) using the expert observations and then transfers to the changed domain. How well would this work in practice ? Also, how fast can different IRL algorithms solve the target task in general (assuming a first person perspective) ?
- Although I like the idea of presenting the experiments as being directed towards answering a specific set of questions I feel like the posed questions somewhat distract from the main theme of the paper. Question 2 suddenly makes the use of additional velocity information to be a main point of importance and the experiments regarding Question 3 in the end only contain evaluations regarding two hyperparameters (ignoring all other parameters such as the parameters for TRPO, the number of rollouts per iteration, the number of presented expert episodes and  the design choices for the GAN). I understand that not all of these can be evaluated thoroughly in a conference paper but I feel like some more experiments or at least some discussion would have helped here.
- The presented experimental evaluation somewhat hides the cost of TRPO training with the obtained reward function. How many roll-outs are necessary in each step?
- The experiments lack some details: How are the expert trajectories obtained? The domains for the pendulum experiment seem identical except for coloring of the pole, is that correct (I am surprised this small change seems to have such a detrimental effect)? Figure 3 shows average performance over 5 trials, what about Figure 5 (if this is also average performance, what is the variance here)? Given that GANs are not easy to train, how often does the training fail/were you able to re-use the hyperparameters across all experiments?
UPDATE:
I updated the score. Please see my response to the rebuttal below.


We study the problem of transferring a sample in one domain to an analog sample in another domain. Given two related domains, S and T , we would like to learn a generative function G that maps an input sample from S to the domain T , such that the output of a given representation function f , which accepts inputs in either domains, would remain unchanged. Other than f , the training data is unsupervised and consist of a set of samples from each domain, without any mapping between them. The Domain Transfer Network (DTN) we present employs a compound loss function that includes a multiclass GAN loss, an f preserving component, and a regularizing component that encourages G to map samples from T to themselves. We apply our method to visual domains including digits and face images and demonstrate its ability to generate convincing novel images of previously unseen entities, while preserving their identity.
We study the problem of transferring a sample in one domain to an analog sample in another domain. Given two related domains, S and T , we would like to learn a generative function G that maps an input sample from S to the domain T , such that the output of a given representation function f , which accepts inputs in either domains, would remain unchanged. Other than f , the training data is unsupervised and consist of a set of samples from each domain, without any mapping between them. The Domain Transfer Network (DTN) we present employs a compound loss function that includes a multiclass GAN loss, an f preserving component, and a regularizing component that encourages G to map samples from T to themselves. We apply our method to visual domains including digits and face images and demonstrate its ability to generate convincing novel images of previously unseen entities, while preserving their identity.
1 INTRODUCTION
Humans excel in tasks that require making analogies between distinct domains, transferring elements from one domain to another, and using these capabilities in order to blend concepts that originated from multiple source domains. Our experience tells us that these remarkable capabilities are developed with very little, if any, supervision that is given in the form of explicit analogies.
Recent achievements replicate some of these capabilities to some degree: Generative Adversarial Networks (GANs) are able to convincingly generate novel samples that match that of a given training set; style transfer methods are able to alter the visual style of images; domain adaptation methods are able to generalize learned functions to new domains even without labeled samples in the target domain and transfer learning is now commonly used to import existing knowledge and to make learning much more efficient.
These capabilities, however, do not address the general analogy synthesis problem that we tackle in this work. Namely, given separated but otherwise unlabeled samples from domains S and T and a perceptual function f , learn a mapping G : S → T such that f(x) ∼ f(G(x)). In order to solve this problem, we make use of deep neural networks of a specific structure in which the function G is a composition of the input function f and a learned function g. A compound loss that integrates multiple terms is used. One term is a Generative Adversarial Network (GAN) term that encourages the creation of samples G(x) that are indistinguishable from the training samples of the target domain, regardless of x ∈ S or x ∈ T . The second loss term enforces that for every x in the source domain training set, ||f(x)− f(G(x))|| is small. The third loss term is a regularizer that encourages G to be the identity mapping for all x ∈ T . The type of problems we focus on in our experiments are visual, although our methods are not limited to visual or even to perceptual tasks. Typically, f would be a neural network representation that is taken as the activations of a network that was trained, e.g., by using the cross entropy loss, in order to classify or to capture identity.
As a main application challenge, we tackle the problem of emoji generation for a given facial image. Despite a growing interest in emoji and the hurdle of creating such personal emoji manually, no system has been proposed, to our knowledge, that can solve this problem. Our method is able to
produce face emoji that are visually appealing and capture much more of the facial characteristics than the emoji created by well-trained human annotators who use the conventional tools.
2 RELATED WORK
As far as we know, the domain transfer problem we formulate is novel despite being ecological (i.e., appearing naturally in the real-world), widely applicable, and related to cognitive reasoning (Fauconnier & Turner, 2003). In the discussion below, we survey recent GAN work, compare our work to the recent image synthesis work and make links to unsupervised domain adaptation.
GAN (Goodfellow et al., 2014) methods train a generator network G that synthesizes samples from a target distribution given noise vectors. G is trained jointly with a discriminator network D, which distinguishes between samples generated by G and a training set from the target distribution. The goal of G is to create samples that are classified by D as real samples.
While originally proposed for generating random samples, GANs can be used as a general tool to measure equivalence between distributions. Specifically, the optimization of D corresponds to taking the most discriminative D achievable, which in turn implies that the indistinguishability is true for every D. Formally, Ganin et al. (2016) linked the GAN loss to the H-divergence between two distributions of Ben-david et al. (2006).
The generative architecture that we employ is based on the successful architecture of Radford et al. (2015). There has recently been a growing concern about the uneven distribution of the samples generated by G – that they tend to cluster around a set of modes in the target domain (Salimans et al., 2016). In general, we do not observe such an effect in our results, due to the requirement to generate samples that satisfy specific f -constancy criteria.
A few contributions (“Conditional GANs”) have employed GANs in order to generate samples from a specific class (Mirza & Osindero, 2014), or even based on a textual description (Reed et al., 2016). When performing such conditioning, one can distinguish between samples that were correctly generated but fail to match the conditional constraint and samples that were not correctly generated. This is modeled as a ternary discriminative function D (Reed et al., 2016; Brock et al., 2016).
The recent work by Dosovitskiy & Brox (2016), has shown promising results for learning to map embeddings to their pre-images, given input-target pairs. Like us, they employ a GAN as well as additional losses in the feature- and the pixel-space. Their method is able to invert the midlevel activations of AlexNet and reconstruct the input image. In contrast, we solve the problem of unsupervised domain transfer and apply the loss terms in different domains: pixel loss in the target domain, and feature loss in the source domain.
Another class of very promising generative techniques that has recently gained traction is neural style transfer. In these methods, new images are synthesized by minimizing the content loss with respect to one input sample and the style loss with respect to one or more input samples. The content loss is typically the encoding of the image by a network training for an image categorization task, similar to our work. The style loss compares the statistics of the activations in various layers of the neural network. We do not employ style losses in our method. While initially style transfer was obtained by a slow optimization process (Gatys et al., 2016), recently, the emphasis was put on feed-forward methods (Ulyanov et al., 2016; Johnson et al., 2016).
There are many links between style transfer and our work: both are unsupervised and generate a sample under f constancy given an input sample. However, our work is much more general in its scope and does not rely on a predefined family of perceptual losses. Our method can be used in order to perform style transfer, but not the other way around. Another key difference is that the current style transfer methods are aimed at replicating the style of one or several images, while our work considers a distribution in the target space. In many applications, there is an abundance of unlabeled data in the target domain T , which can be modeled accurately in an unsupervised manner.
Given the impressive results of recent style transfer work, in particular for face images, one might get the false impression that emoji are just a different style of drawing faces. By way of analogy, this claim is similar to stating that a Siamese cat is a Labrador in a different style. Emoji differ from facial photographs in both content and style. Style transfer can create visually appealing face images; However, the properties of the target domain are compromised.
In the computer vision literature, work has been done to automatically generate sketches from images, see Kyprianidis et al. (2013) for a survey. These systems are able to emphasize image edges and facial features in a convincing way. However, unlike our method, they require matching pairs of samples, and were not shown to work across two distant domains as in our method. Due to the lack of supervised training data, we did not try to apply such methods to our problems. However, one can assume that if such methods were appropriate for emoji synthesis, automatic face emoji services would be available.
Unsupervised domain adaptation addresses the following problem: given a labeled training set in S × Y , for some target space Y , and an unlabeled set of samples from domain T , learn a function h : T → Y (Chen et al., 2012; Ganin et al., 2016). One can solve the sample transfer problem (our problem) using domain adaptation and vice versa. In both cases, the solution is indirect. In order to solve domain adaptation using domain transfer, one would learn a function from S to Y and use it as the input method of the domain transfer algorithm in order to obtain a map from S to T 1. The training samples could then be transferred to T and used to learn a classifier there.
In the other direction, given the function f , one can invert f in the domain T by generating training samples (f(x), x) for x ∈ T and learn from them a function h from f(T ) = {f(x)|x ∈ T} to T . Domain adaptation can then be used in order to map f(S) = {f(x)|x ∈ S} to T , thus achieving domain transfer. Based on the work by Zhmoginov & Sandler (2016), we expect that h, even in the target domain of emoji, will be hard to learn, making this solution hypothetical at this point.
3 A BASELINE PROBLEM FORMULATION
Given a set s of unlabeled samples in a source domain S sampled i.i.d according to some distribution DS , a set of samples in the target domain t ⊂ T sampled i.i.d from distribution DT , a function f from the domain S ∪ T , some metric d, and a weight α, we wish to learn a function G : S → T that minimizes the combined risk R = RGAN + αRCONST, which is comprised of
RGAN = max D Ex∼DS log[1−D(G(x))] + Ex∼DT log[D(x)], (1)
where D is a binary classification function from T , D(x) the probability of the class 1 it assigns for a sample x ∈ T , and
RCONST = Ex∼DS d(f(x), f(G(x))) (2)
The first term is the adversarial risk, which requires that for every discriminative function D, the samples from the target domain would be indistinguishable from the samples generated by G for samples in the source domain. An adversarial risk is not the only option. An alternative term that does not employ GANs would directly compare the distribution DT to the distribution of G(x) where x ∼ DS , e.g., by using KL-divergence. The second term is the f -constancy term, which requires that f is invariant under G. In practice, we have experimented with multiple forms of d including Mean Squared Error (MSE) and cosine distance, as well as other variants including metric learning losses (hinge) and triplet losses. The performance is mostly unchanged, and we report results using the simplest MSE solution.
Similarly to other GAN formulations, one can minimize the loss associated with the risk R over G, while maximizing it over D, where G and D are deep neural networks, and the expectations in R are replaced by summations over the corresponding training sets. However, this baseline solution, as we will show experimentally, does not produce desirable results.
1The function trained this way would be more accurate on S than on T . This asymmetry is shared with all experiments done in this work.
4 THE DOMAIN TRANSFER NETWORK
We suggest to employ a more elaborate architecture that contains two high level modifications. First, we employ f(x) as the baseline representation to the function G. Second, we consider, during training, the generated samples G(x) for x ∈ t. The first change is stated as G = g ◦ f , for some learned function g. By applying this, we focus the learning effort ofG on the aspects that are most relevant toRCONST. In addition, in most applications, f is not as accurate on T as it on S. The composed function, which is trained on samples from both S and T , adds layers on top of f , which adapt it.
The second change alters the form of LGAN, making it multiclass instead of binary. It also introduces a new term LTID that requires G to be the identity matrix on samples from T . Taken together and written in terms of training loss, we now have two losses LD and LG = LGANG + αLCONST + βLTID + γLTV, for some weights α, β, γ, where
LD = − ∑ x∈s logD1(g(f(x)))− ∑ x∈t logD2(g(f(x)))− ∑ x∈t logD3(x) (3)
LGANG = − ∑ x∈s logD3(g(f(x)))− ∑ x∈t logD3(g(f(x))) (4)
LCONST = ∑ x∈s d(f(x), f(g(f(x)))) (5)
LTID = ∑ x∈t d2(x,G(x)) (6)
and where D is a ternary classification function from the domain T to 1, 2, 3, and Di(x) is the probability it assigns to class i = 1, 2, 3 for an input sample x, and d2 is a distance function in T . During optimization, LG is minimized over g and LD is minimized over D. See Fig. 1 for an illustration of our method.
Eq. 3 and 4 make sure that the generated analogy, i.e., the output ofG, is in the target space T . Since D is ternary and can therefore confuse classes in more than one way, this role, which is captured by Eq. 1 in the baseline formulation, is split into two. However, the two equations do not enforce any similarity between the source sample x and the generated G(x). This is done by Eq. 5 and 6: Eq. 5 enforces f -constancy for x ∈ S, while Eq. 6 enforces that for samples x ∈ T , which are already in the target space, G is the identity mapping. The latter is a desirable behavior, e.g., for the cartooning task, given an input emoji, one would like it to remain constant under the mapping of G. It can also be seen as an autoencoder type of loss, applied only to samples from T . The experiments reported in Sec. 5 evaluate the contributions of LCONST and LTID and reveal that at least one of these is required, and that when employing only one loss, LCONST leads to a better performance than LTID.
The last loss, LTV is an anisotropic total variation loss (Rudin et al., 1992; Mahendran & Vedaldi, 2015), which is added in order to slightly smooth the resulting image. The loss is defined on the generated image z = [zij ] = G(x) as
LTV (z) = ∑ i,j ( (zi,j+1 − zij)2 + (zi+1,j − zij)2 )B 2 , (7)
where we employ B = 1.
In our work, MSE is used for both d and d2. We also experimented with replacing d2, which, in visual domains, compares images, with a second GAN. No noticeable improvement was observed. Throughout the experiments, the adaptive learning rate method Adam by Kingma & Ba (2016) is used as the optimization algorithm.
5 EXPERIMENTS
The Domain Transfer Network (DTN) is evaluated in two application domains: digits and face images. In the first domain, we transfer images from the Street View House Number (SVHN) dataset of Netzer et al. (2011) to the domain of the MNIST dataset by LeCun & Cortes (2010). In
the face domain, we transfer a set of random and unlabeled face images to a space of emoji images. In both cases, the source and target domains differ considerably.
5.1 DIGITS: FROM SVHN TO MNIST
For working with digits, we employ the extra training split of SVHN, which contains 531,131 images for two purposes: learning the function f and as an unsupervised training set s for the domain transfer method. The evaluation is done on the test split of SVHN, comprised of 26,032 images. The architecture of f consists of four convolutional layers with 64, 128, 256, 128 filters respectively, each followed by max pooling and ReLU non-linearity. The error on the test split is 4.95%. Even tough this accuracy is far from the best reported results, it seems to be sufficient for the purpose of domain transfer. Within the DTN, f maps a 32 × 32 RGB image to the activations of the last convolutional layer of size 128× 1× 1 (post a 4× 4 max pooling and before the ReLU). In order to apply f on MNIST images, we replicate the grayscale image three times.
The set t contains the test set of the MNIST dataset. For supporting quantitative evaluation, we have trained a classifier on the train set of the MNIST dataset, consisting of the same architecture as f . The accuracy of this classifier on the test set approaches perfect performance at 99.4% accuracy, and is, therefore, trustworthy as an evaluation metric. In comparison, the network f , achieves 76.08% accuracy on t.
Network g, inspired by Radford et al. (2015), maps SVHN-trained f ’s 128D representations to 32× 32 grayscale images. g employs four blocks of deconvolution, batch-normalization, and ReLU, with a hyperbolic tangent terminal. The architecture ofD consists of four batch-normalized convolutional layers and employs ReLU. See Radford et al. (2015) for more details on the networks architecture. In the digit experiments, the results were obtained with the tradeoff hyperparamemters α = β = 15. We did not observe a need to add a smoothness term and the weight of LTV was set to γ = 0.
Despite not being very accurate on both domains (and also considerably worse than the SVHN state of the art), we were able to achieve visually appealing domain transfer, as shown in Fig. 2(a). In order to evaluate the contribution of each of the method’s components, we have employed the MNIST network on the set of samples G(sTEST ) = {G(x)|x ∈ sTEST }, using the true SVHN labels of the test set.
We first compare to the baseline method of Sec. 3, where the generative function, which works directly with samples in S, is composed out of a few additional layers at the bottom of G. The results, shown in Tab. 1, demonstrate that DTN has a clear advantage over the baseline method. In addition, the contribution of each one of the terms in the loss function is shown in the table. The regularization term LTID seems less crucial than the constancy term. However, at least one of them is required in order to obtain good performance. The GAN constraints are also important. Finally, the inclusion of f within the generator function G has a dramatic influence on the results.
As explained in Sec. 2, domain transfer can be used in order to perform unsupervised domain adaptation. For this purposes, we transformed the set s to the MNIST domain (as above), and using the true labels of s employed a simple nearest neighbor classifier there. The choice of classifier was
to emphasize the simplicity of the approach; However, the constraints of the unsupervised domain transfer problem would be respected for any classifier trained on G(s). The results of this experiment are reported in Tab. 2, which shows a clear advantage over the state of the art method of Ganin et al. (2016). This is true both when transferring the samples of the set s and when transferring the test set of SVHN, which is much smaller and was not seen during the training of the DTN.
5.1.1 UNSEEN DIGITS
Another set of experiments was performed in order to study the ability of the domain transfer network to overcome the omission of a class of samples. This type of ablation can occur in the source or the target domain, or during the training of f and can help us understand the importance of each of these inputs. The results are shown visually in Fig. 3, and qualitatively in Tab. 3, based on the accuracy of the MNIST classifier only on the transferred samples from the test set of SVHN that belong to class ‘3’.
It is evident that not including the class in the source domain is much less detrimental than eliminating it from the target domain. This is the desirable behavior: never seeing any ‘3’-like shapes in t, the generator should not generate such samples. Results are better when not observing ‘3’ in both s, t than when not seeing it only in t since in the latter case, G learns to map source samples of ‘3’ to target images of other classes.
5.2 FACES: FROM PHOTOS TO EMOJI
For face images, we use a set s of one million random images without identity information. The set t consists of assorted facial avatars (emoji) created by an online service (bitmoji.com). The emoji images were processed by a fully automatic process that localizes, based on a set of heuristics, the center of the irides and the tip of the nose. Based on these coordinates, the emoji were centered and scaled into 152× 152 RGB images. As the function f , we employ the representation layer of the DeepFace network Taigman et al. (2014). This representation is 256-dimensional and was trained on a labeled set of four million images that does not intersect the set s. Network D takes 152 × 152 RGB images (either natural or scaled-up emoji) and consists of 6 blocks, each containing a convolution with stride 2, batch normalization, and a leaky ReLU with a parameter of 0.2. Network g maps f ’s 256D representations to 64× 64 RGB images through a network with 5 blocks, each consisting of an upscaling convolution, batch-normalization and ReLU. Adding 1 × 1 convolution to each block resulted in lower LCONST training errors, and made g 9-layers deep. We set α = 100, β = 1, γ = 0.05 as the tradeoff hyperparameters within LG via validation. As expected, higher values of α resulted in better f -constancy, however introduced artifacts such as general noise or distortions. The network was trained for 3 epochs, the point where no further reduction of validation error was observed on LCONST.
In order to upscale the 64 × 64 output to print quality, we used the method of Dong et al. (2015), which was shown to work well on art. We did not retrain this network for our application, and apply the published one to the final output of our method after its training was finished. Results without this upscale are shown, for comparison, in Appendix C.
Comparison With Human Annotators For evaluation purposes only, a team of professional annotators manually created an emoji, using a web service, for 118 random images from the CelebA dataset (Yang et al., 2015). Fig. 4 shows side by side samples of the original image, the human generated emoji and the emoji generated by the learned generator function G. As can be seen, the automatically generated emoji tend to be more informative, albeit less restrictive than the ones created manually.
In order to evaluate the identifiability of the resulting emoji, we have collected a second example for each identity in the set of 118 CelebA images and a set s′ of 100,000 random face images, which were not included in s. We then employed the VGG face CNN descriptor of Parkhi et al. (2015) in order to perform retrieval as follows. For each image x in our manually annotated set, we create a gallery s′ ∪ x′, where x′ is the other image of the person in x. We then perform retrieval using the VGG face descriptor using either the manually created emoji or G(x) as probe.
The VGG network is used in order to avoid a bias that might be caused by using f both for training the DTN and for evaluation. The results are reported in Tab. 4. As can be seen, the emoji generated by G are much more discriminative than the emoji created manually and obtain a median rank of 16 in cross-domain identification out of 105 distractors.
Multiple Images Per Person We evaluate the visual quality that is obtained per person and not just per image, by testing DTN on the Facescrub dataset (Ng & Winkler, 2014). For each person p, we considered the set of their images Xp, and selected the emoji that was most similar to their
source image: argmin x∈Xp ||f(x)− f(G(x))|| (8)
This simple heuristic seems to work well in practice; The general problem of mapping a set X ⊂ S to a single output in T is left for future work. Fig. 2(b) contains several examples from the Facescrub dataset. For the complete set of identities, see Appendix A.
Transferring both identity and expression We also experimented with multiple expressions. As it turns out the face identification network f encodes enough expression information to support a successful transfer of both identity as well as expression, see Appendix B.
Network Visualization The obtained mapping g can serve as a visualization tool for studying the properties of the face representation. This is studied in Appendix D by computing the emoji generated for the standard basis of R256. The resulting images present a large amount of variability, indicating that g does not present a significant mode effect.
5.3 STYLE TRANSFER AS A SPECIFIC DOMAIN TRANSFER TASK
Fig. 5(a-c) demonstrates that neural style transfer Gatys et al. (2016) cannot solve the photo to emoji transfer task in a convincing way. The output image is perhaps visually appealing; However, it does not belong to the space t of emoji. Our result are given in Fig. 5(d) for comparison. Note that DTN is able to fix the missing hair in the image.
Domain transfer is more general than style transfer in the sense that we can perform style transfer using a DTN. In order to show this, we have transformed, using the method of Johnson et al. (2016), the training images of CelebA based on the style of a single image (shown in Fig. 5(e)). The original photos were used as the set s, and the transformed images were used as t. Applying DTN, using face representation f , we obtained styled face images such as the one shown in the figure 5(f).
6 DISCUSSION AND LIMITATIONS
Asymmetry is central to our work. Not only does our solution handle the two domains S and T differently, the function f is unlikely to be equally effective in both domains since in most practical cases, f would be trained on samples from one domain. While an explicit domain adaptation step can be added in order to make f more effective on the second domain, we found it to be unnecessary. Adaptation of f occurs implicitly due to the application of D downstream.
Using the same function f , we can replace the roles of the two domains, S and T . For example, we can synthesize an SVHN image that resembles a given MNIST image, or synthesize a face that matches an emoji. As expected, this yields less appealing results due to the asymmetric nature of f and the lower information content in these new source domains, see Appendix E.
Domain transfer, as an unsupervised method, could prove useful across a wide variety of computational tasks. Here, we demonstrate the ability to use domain transfer in order to perform unsupervised domain adaptation. While this is currently only shown in a single experiment, the simplicity of performing domain adaptation and the fact that state of the art results were obtained effortlessly with a simple nearest neighbor classifier suggest it to be a promising direction for future research.
A FACESCRUB DATASET GENERATIONS
In Fig. 6 we show the full set of identities of the Facescrub dataset, and their corresponding generated emoji.
B TRANSFERRING NON-IDENTITY DATA
f may encode, in addition to identity, other data that is desirable to transfer. In the example of faces, this information might include expression, facial hair, glasses, pose, etc. In order to transfer such information, it is important that the set of samples in the target domain t present variability along the desirable dimensions. Otherwise, the GAN applied in the target domain (Eq. 4) would maintain these dimensions fixed. The set t employed throughout our experiments in Sec. 5.2 was constructed by sampling emoji of neutral expression. To support a smiling expression for example, we simply added to set t random smiling emoji and re-trained the DTN. The results, presented in Fig. 7, demonstrate that f contains expression information in addition to identity information, and that this information is enough in order to transfer smiling photos to smiling emoji.
C THE EFFECT OF SUPER-RESOLUTION
As mentioned in Sec. 5, in order to upscale the 64× 64 output to print quality, the method of Dong et al. (2015) is used. Fig. 8 shows the effect of applying this postprocessing step.
D THE BASIS ELEMENTS OF THE FACE REPRESENTATION
Fig. 9 depicts the face emoji generated by g for the standard basis of the face representation (Taigman et al., 2014), viewed as the vector space R256.
E DOMAIN TRANSFER IN THE REVERSE DIRECTION
For completion, we present, in Fig. 10 results obtained by performing domain transfer using DTNs in the reverse direction of the one reported in Sec. 5.

Update: thank you for running more experiments, and add more explanations in the manuscript. They addressed most of my concerns, so I updated the score accordingly.
The work aims at learning a generative function G that can maps input from source domain to the target domain, such that a given representation function f remain unchanged accepting inputs from either domain. The criteria is termed f-constancy. The proposed method is evaluated on two visual domain adaptation tasks.
The paper is relatively easy to follow, and the authors provided quite extensive experimental results on the two datasets. f-constancy is the main novelty of the work.
It seems counter-intuitive to force the function G to be of g o f, i.e., starting from a restricted function f which might have already lost information. As in the face dataset, f is learned to optimize the performance of certain task on some external dataset. It is not clear if an input from the source or target domain can be recovered from applying G as in equation (5) and (6). Also, the f function is learned with a particular task in mind. As in the two experiments, the representation function f is learned to identify the digits in the source SVHN dataset or the identity of some face dataset.  As a result, the procedure has to be repeated if we were to perform domain adaptation for the same domains but for different tasks, such as recognizing expressions instead of identity.
Do the authors have insight on why the baseline method proposed in equation (1) and (2) perform so poorly?
Figure 5 shows some visual comparison between style transfer and the proposed method. It is not clear though which method is better. Will it be possible to apply style transfer to generate emojis from photos and repeat the experiments shown in table 4?