People can learn a wide range of tasks from their own experience, but can also learn from observing other creatures. This can accelerate acquisition of new skills even when the observed agent differs substantially from the learning agent in terms of morphology. In this paper, we examine how reinforcement learning algorithms can transfer knowledge between morphologically different agents (e.g., different robots). We introduce a problem formulation where two agents are tasked with learning multiple skills by sharing information. Our method uses the skills that were learned by both agents to train invariant feature spaces that can then be used to transfer other skills from one agent to another. The process of learning these invariant feature spaces can be viewed as a kind of “analogy making,” or implicit learning of partial correspondences between two distinct domains. We evaluate our transfer learning algorithm in two simulated robotic manipulation skills, and illustrate that we can transfer knowledge between simulated robotic arms with different numbers of links, as well as simulated arms with different actuation mechanisms, where one robot is torque-driven while the other is tendon-driven.
1 INTRODUCTION
People can learn large repertoires of motor skills autonomously from their own experience. However, learning is accelerated substantially when the learner is allowed to observe another person performing the same skill. In fact, human infants learn faster when they observe adults performing a task, even when the adult performs the task differently from the child, and even when the adult performs the task incorrectly (Meltzoff, 1999). Clearly, we can accelerate our own skill learning by observing a novel behavior, even when that behavior is performed by an agent with different physical capabilities or differences in morphology. Furthermore, evidence in neuroscience suggests that the parts of the brain in monkeys that respond to the pose of the hand can quickly adapt to instead respond to the pose of the end-effector of a tool held in the hand (Umilta et al., 2008). This suggests that the brain learns an invariant feature space for the task (e.g., reaching with a tool) that is independent of the morphology of the limb performing that task. Mirror neurons also fire both when the animal performs a task and when it observes another animal performing it (Rizzolatti & Craighero, 2004; Ferrari et al., 2005). Can we enable robots and other autonomous agents to transfer knowledge from other agents with different morphologies by learning such invariant representations?
In robotics and reinforcement learning, prior works have considered building direct isomorphisms between state spaces, as discussed in Section 2. However, most of these methods require specific domain knowledge to determine how to form the mapping, or operate on simple, low-dimensional environments. For instance, Taylor et al. (2008) find a mapping between state spaces by searching through all possible pairings. Learning state-to-state isomorphisms involves an assumption that the two domains can be brought into correspondence, which may not be the case for morphologically
∗These authors contributed equally to this work.
different agents. Some aspects of the skill may not be transferable at all, in which case they must be learned from scratch, but we would like to maximize the information transferred between the agents.
In this paper, we formulate this multi-agent transfer learning problem in a setting where two agents are learning multiple skills. Using the skills that have been already acquired by both agents, each agent can construct a mapping from their states into an invariant feature space. Each agent can then transfer a new skill from the other agent by projecting the executions of that skill into the invariant space, and tracking the corresponding features through its own actions. This provides a well-shaped reward function to the learner that allows it to imitate those aspects of the “teacher” agent that are invariant to differences in their morphology, while ignoring the parts of the state that cannot be imitated. Since the mapping from the state spaces of each agent into the invariant feature space might be complex and nonlinear, we use deep neural networks to represent the mappings, and we present an algorithm that can learn these mappings from the shared previously acquired skills.
The main contributions of our work are a formulation of the multi-skill transfer problem, a definition of the common feature space, and an algorithm that can be used to learn the maximally informative feature space for transfer between two agents (e.g., two robots with different morphologies). To evaluate the efficiency of this transfer process, we use a reinforcement learning algorithm to transfer skills from one agent to another through the invariant feature space. The agents we consider may differ in state-space, action-space, and dynamics. We evaluate our transfer learning method in two simulated robotic manipulation tasks, and illustrate that we can transfer knowledge between simulated robotic arms with different numbers of links, as well as simulated arms with different actuation mechanisms, where one robot is torque-driven while the other is tendon-driven.
2 RELATED WORK
Transfer learning has long been recognized as an important direction in robotics and reinforcement learning (Taylor & Stone (2009)). Konidaris & Barto (2006) learned value functions on subsets of the state representation that were shared between tasks, providing a shaping reward in the target task. Taylor et al. (2007) manually construct a function to map a Q-function from one Markov decision process (MDP) to another. Ammar & Taylor (2012) manually define a common feature space between the states of two MDPs, and use this feature space to learn a mapping between states.
Later work by Ammar et al. (2015a) uses unsupervised manifold alignment to assign pairings between states for transfer. Like in our method, they aim to transfer skills between robots with different configurations and action spaces by guiding exploration in the target domain. The main difference from our work is that Ammar et al. (2015a) assume the presence of a feature mapping that provides distances between states, and use these (hand designed) features to assign correspondences between states in the different domains. In contrast, we assume that good correspondences in episodic tasks can be extracted through time alignment, and focus on learning the feature mapping itself. Additionally, we do not try to learn a direct mapping between state spaces but instead try to learn nonlinear embedding functions into a common feature space, as compared to linear mappings between state spaces learned in Ammar et al. (2015a). In a similar vein, Raimalwala et al. (2016) consider transfer learning across linear time-invariant (LTI) systems through simple alignment based methods. Although this method is quite effective in enabling transfer in these systems, it does not apply to the higher dimensional continuous control tasks we consider which may have non-linear dynamics, and may not be LTI.
In machine learning, Pan & Yang (2010) provide an extensive survey on transfer learning which addresses the case of train and test data being drawn from different distributions, as well as learning models that succeed on multiple, related tasks. Ben-David & Schuller (2003) derive theoretical guarantees on this sort of multitask learning and provide a formal framework for defining task relatedness. In deep learning, Caruana (1997) show that a multitask network can leverage a shared representation of the input to learn multiple tasks more quickly together than separately.
More recent work in deep learning has also looked at transferring policies by reusing policy parameters between environments (Rusu et al., 2016a;b; Braylan et al., 2015; Daftry et al., 2016), using either regularization or novel neural network architectures, though this work has not looked at transfer between agents with structural differences in state, such as different dimensionalities. Our approach is largely orthogonal to policy transfer methods, since our aim is not to directly transfer a
skill policy, which is typically impossible in the presence of substantial morphological differences, but rather to learn a shared feature space that can be used to transfer information about a skill that is shared across robots, while ignoring those aspects that are not shared. Our own recent work has looked at morphological differences in the context of multi-agent and multi-task learning (Devin et al., 2016), by reusing neural network components across agent/task combinations. In contrast to that work, which transferred components of policies, our present work aims to learn common feature spaces in situations where we have just two agents. We do not aim to transfer parts of policies themselves, but instead look at shared structure in the states visited by optimal policies, which can be viewed as a kind of analogy making across domains.
Learning feature spaces has also been studied in the domain of computer vision as a mechanism for domain adaptation and metric learning. Xing et al. (2002) finds a linear transformation of the input data to satisfy pairwise similarity contraints, while past work by Chopra et al. (2005) used Siamese networks to learn a feature space where paired images are brought close together and unpaired images are pushed apart. This enables a semantically meaningful metric space to be learned with only pairs as labels. Later work on domain adaptation by Tzeng et al. (2015) and Ganin et al. (2016) use an adversarial approach to learn an image embedding that is useful for classification and invariant to the input image’s domain. We use the idea of learning a metric space from paired states, though the adversarial approach could also be used with our method as an alternative objective function in future work.
3 PROBLEM FORMULATION AND ASSUMPTIONS
We formalize our transfer problem in a general way by considering a source domain and a target domain, denoted DS and DT , which each correspond to Markov decision processes (MDPs) DS = (SS,AS,TS,RS) and DT = (ST ,AT ,TT ,RT ), each with its own state space S , action space A , dynamics or transition function T , and reward function R. In general, the state and action spaces in the two domains might be completely different. Correspondingly, the dynamics TS and TT also differ, often dramatically. However, we assume that the reward functions share some structural similarity, in that the state distribution of an optimal policy in the source domain will resemble the state distribution of an optimal policy in the target domain when projected into some common feature space. For example, in one of our experimental tasks, DS corresponds to a robotic arm with 3 links, while DT is an arm with 4 links. While the dimensionalities of the states and action are completely different, the two arms are performing the same task, with a reward that depends on the position of the end-effector. Although this end-effector is a complex nonlinear function of the state, the reward is structurally similar for both agents.
3.1 COMMON FEATURE SPACES
We can formalize this common feature space assumption as following: if πS(sS) denotes the state distribution of the optimal policy in DS, and πT (sT ) denotes the state distribution of the optimal policy in DT , it is possible to learn two functions, f and g, such that p( f (sS)) = p(g(sT )) for sS ∼ πS and sT ∼ πT . That is, the images of πS under f and πT under g correspond to the same distribution. This assumption is trivially true if we allow lossy mappings f and g (e.g. if f (sS) = g(sT ) = 0 for all sS and sT ). However, the less information we lose in f and g, the more informative the shared feature will be for the purpose of transfer. So while we might not in general be able to fully recover πT from the image of πS under f , we can attempt to learn f and g to maximize the amount of information contained in the shared space.
3.2 LEARNING WITH MULTIPLE SKILLS
In order to learn the common feature space, we need examples from both domains. While both agents could in principle learn a common feature space through direct exploration, in this work we instead assume that the agents have prior knowledge about each other, in the form of other skills that they have both learned. This assumption is reasonable, since many practical use-cases of transfer involve two agents that already have competence in a range of simple settings, and wish to transfer the competence of one agent in a new setting to another one. For example, we might wish to transfer a particular cooking skill from one home robot to another one, in a setting where both robots have
already learned some basic manipulation behaviors that can allow us to build a common feature space between the two robots. Humans similarly leverage their extensive prior knowledge to aid in transfer, by recognizing limbs and hands and understanding their function.
To formalize the setting where the two agents can perform multiple tasks, we divide the state space in each of the two domains into an agent-specific state sr and a task-specific state senv. A similar partitioning of the state variables was previously discussed by Devin et al. (2016), and is closely related to the agent-space proposed by Konidaris (2006). For simplicity, we will consider a case where there are just two skills: one proxy skill that has been learned by both agents, and one test skill that has been learned by the source agent in the domain DS and is currently being transferred to the target agent in domain DT . We will use DSp and DT p to denote the proxy task domains for the source and target agents. We assume that DS and DSp (and similarly DT and DT p) differ only in their reward functions and task-specific states, with the agent-specific state spaces Sr and action spaces being the same between the proxy and test domains. For example DSp might correspond to a 3-link robot pushing an object, while DS might correspond to the same robot opening a drawer, and DT p and DT correspond to a completely different robot performing those tasks. Then, we can learn functions f and g on the robot-specific states of the proxy domains, and use them to transfer knowledge from DS to DT .
The idea in this setup is that both agents will have already learned the proxy task, and we can compare how they perform this task in order to determine the common feature space. This is a natural problem setup for many robotic transfer learning problems, as well as other domains where multiple distinct agents might need to each learn a large collection of skills, exchanging their experience and learning which information they can and cannot transfer from each other. In a practical scenario, each robot might have already learned a large number of basic skills, some of which were learned by both robots. These skills are candidate proxy tasks that the robots can use to learn their shared space, which one robot can then use to transfer knowledge from the other one and more quickly learn skills that it does not yet possess.
3.3 ESTIMATING CORRESPONDENCES FROM PROXY SKILL
The proxy skill is useful for learning which pairs of agent-specific states correspond across both domains. We want to learn a pairing P, which is a list of pairs of states in both domains which are corresponding. This is then used for the contrastive loss as described in Section 4. These correspondences could be obtained through an unsupervised alignment procedure but in our method we explore two simpler approaches exploiting the fact that the skills we consider are episodic.
3.3.1 TIME-BASED ALIGNMENT
The first extremely simple approach we consider is to say that in such episodic skills, a reasonable approximate alignment can be obtained by assuming that the two agents will perform each task at roughly the same rate, and we can therefore simply pair the states that are visited in the same time step in the two proxy domains.
3.3.2 ALTERNATING OPTIMIZATION USING DYNAMIC TIME WARPING
However, this alignment is sensitive to time based alignment and may not be very robust if the agents are performing the task at somewhat different rates. In order to address this, we formulate an alternating optimization procedure to be more robust than time-based alignment. This optimization alternates between learning a common feature space using currently estimated correspondences, and re-estimating correspondences using the currently learned feature space. We make use of Dynamic Time Warping (DTW) as described in Müller (2007), a well known method for learning correspondences across sequences which may vary in speed. Dynamic time warping requires a metric space to compare elements in the sequences to compute an optimal alignment between the sequences. In this method, we initialize the weak time-based alignment described in the previous paragraph and use it to learn a common feature space. This feature space serves as a metric space for DTW to re-estimate correspondences across domains. The new correspondences are then used as pairs for learning a better feature space, and so on. This forms an Expectation-Maximization style approach which can help estimate better correspondences than naive time-alignment.
4 LEARNING COMMON FEATURE SPACES FOR SKILL TRANSFER
In this section, we will discuss how the shared space can be learned by means of the proxy task. We will then describe how this shared space can be used for knowledge transfer for a new task, and finally present results that evaluate transfer on a set of simulated robotic control domains.
We wish to find functions f and g such that, for states sS p and sT p along the optimal policies πS p∗ and πT p∗, f and g approximately satisfy p( f (sSp,r)) = p(g(sT p,r)). If we can find the common feature space by learning f and g, we can optimize πT by directly mimicking the distribution over f (sSp,r), where sSp,r ∼ πS.
4.1 LEARNING THE EMBEDDING FUNCTIONS FROM A PROXY TASK
To approximate the requirement that p( f (sSp,r)) = p(g(sT p,r)), we assume a pairing P of states in the proxy domains as described in 3.3. The pairing P is a list of pairs of states (sSp,sT p) which are corresponding across domains. As f and g are parametrized as neural networks, we can optimize them using the similarity loss metric introduced by Chopra et al. (2005):
Lsim(sSp,sT p;θ f ,θg) = || f (sSp,r;θ f )−g(sT p,r;θg)||2.
Where θ f and θg are the function parameters, (sSp,r,sT p,r) ∈ P.
Figure 1: The two embedding functions f and g are trained with a contrastive loss between the domains, along with decoders that optimize autoencoder losses.
However, as described in Section 3, if this is the only objective for learning f and g, we can easily end up with uninformative degenerate mappings, such as the one where f (sSp,r) = g(sT p,r) = 0. Intuitively, a good pair of mappings f and g would be as close as possible to being invertible, so as to preserve as much of the information about the source domain as possible. We therefore train a second pair of decoder networks with the goal of optimizing the quality of the reconstruction of sSp,r and sT p,r from the shared feature space, which encourages f and g to preserve the maximum amount of domaininvariant information. We define decoders DecS( f (sSp,r)) and DecT (g(sT p,r)) that map from the feature space back to their respective states. Note that, compared to conven-
tional Siamese network methods, the weights between f and g are not tied, and in general the networks have different dimensional inputs. The objectives for these are:
LAES(sSp,r;θ f ,θDecS) = ||sSp,r−DecS( f (sSp,r;θ f );θDecS)||2,
LAET(sT p,r;θg,θDecT ) = ||sT p,r−DecT (g(sT p,r;θg);θDecT )||2, where θDecS and θDecT are the decoder weights. We train the entire network end-to-end using backpropagation, where the full objective is
min θ f ,θg,θDecS ,θDecT ∑ (sSp,sT p)∈P LAES(sSp,r;θ f ,θDecS)+LAET(sT p,r;θg,θDecT )+Lsim(sSp,r,sT p,r;θ f ,θg)
A diagram of this learning approach is shown in Figure 1.
4.1.1 USING THE COMMON EMBEDDING FOR KNOWLEDGE TRANSFER
The functions f and g learned using the approach described above establish an invariant space across the two domains. However, because these functions need not be invertible, directly mapping from a state in the source domain to a state in the target domain is not feasible.
Instead of attempting direct policy transfer, we match the distributions of optimal trajectories across the domains. Given f and g learned from the network described in Section 4, and the distribution π∗S of optimal trajectories in the source domain, we can incentivize the distribution of trajectories in the target domain to be similar to the source domains under the mappings f and g. Ideally, we would like the distributions p( f (sS,r)) and p(g(sT,r)) to match as closely as possible. However, it may still be necessary for the target agent to learn some aspects of the skill from scratch, since not
all intricacies will transfer in the presence of morphological differences. We therefore use a reinforcement learning algorithm to learn πT , but with an additional term added to the reward function that provides guidance via f (sS,r). This term has following form:
rtransfer(s (t) T,r) = α|| f (s (t) S,r;θ f )−g(s (t) T,r;θg)||2,
where s(t)S,r is the agent-specific state along the optimal policy in the source domain at time step t, and s(t)T,r is the agent-specific state along the current policy that is being learned in the target domain at time step t, and α is a weight on the transfer reward that controls its importance relative to the overall task goal. In essence, this additional reward provides a form of reward shaping, which gives additional learning guidance in the target domain. In sparse reward environments, task performance is highly dependent on directed exploration, and this additional incentive to match trajectory distributions in the embedding space provides strong guidance for task performance.
In tasks where the pairs mapping P is imperfect, the transfer reward may sometimes interfere with learning when the target domain policy is already very good, though it is usually very helpful in the early stages of learning. We therefore might consider gradually reducing the weight α as learning progresses in the target domain. We use this technique for our second experiment, which learns a policy for a tendon-driven arm.
5 EXPERIMENTS
Our experiments aim to evaluate how well common feature space learning can transfer skills between morphologically different agents. The experiments were performed in simulation using the MuJoCo physics simulator (Todorov et al., 2012), in order to explore a variety of different robots and actuation mechanisms. The embedding functions f and g in our experiments are 3 layer neural networks with 60 hidden units each and ReLu non-linearities. They are trained end-to-end with standard backpropagation using the ADAM optimizer (Kingma & Ba, 2015). Videos of our experiment will be available at https://sites.google.com/ site/invariantfeaturetransfer/ For details of the reinforcement learning algorithm used, refer to Appendix A.
5.1 METHODS USED FOR COMPARISON
In the following experiments, we compare our method with other methods. The simplest one, referred to as “no transfer”, aims to learn the target task from scratch. This method generally cannot succeed in sparse reward environments without a large number of episodes. Table 1 shows that, without transfer, the tasks are not learned even with 3-4 times more experience.
We also compare to several linear methods, including random projections, canonical correlation analysis (CCA), and unsupervised manifold alignment (UMA). Random projections of data have been found to provide meaningful dimensionality reduction (Hegde et al., 2008). We assign f and g be random projections into spaces of the same dimension, and transfer as described in Section 4.1.1. CCA (Hotelling, 1936) aims to find a basis for the data in which the source data and target data are maximally correlated. We use the matrices that map from state space to the learned basis as f and g. UMA (Wang & Mahadevan (2009), Ammar et al. (2015b)) uses pairwise distances between states to align the manifolds of the two domains. These methods impose a linearity constraint on f and g which proves to limit the expressiveness of the embeddings. We find that using CCA to learn the embedding allows for transfer between robots, albeit without as much performance gained than if f and g are neural networks.
We also compare to kernel-CCA (KCCA) which uses a kernel matrix to perform CCA, allowing the method to use an implied non-linear feature mapping of the data. We test on several different kernels, including polynomial (quad), radial basis (rbf), and linear. These methods perform especially well on transfer between different actuation methods, but which kernel to use for best performance is not consistent between experiments. For example, although the quadratic kernel performs competitively with our method for the tendon experiment, it does not work at all for our button pushing experiment.
The last method we compare with is “direct mapping” which learns to directly predict sT,r from sS,r instead of mapping both into a common space. This is representative of a number of prior techniques that attempt to put source and target domains into direct correspondence such as Taylor et al. (2008). In this method, we use the same pairs as we do for our method, estimated from prior experience, but try to map directly from the source domain to the target domain. In order to guide learning using this method, we pass optimal source trajectories through the learned mapping, and then penalize the target robot for deviating from these predicted trajectories. As seen in Figures 5 and 8 this method does not succeed, probably because mapping from one state space to another is more difficult than mapping both state spaces into similar embeddings. The key difference between this method and ours is that we map both domains into a common space, which allows us to put only the common parts of the state spaces in correspondence instead of trying to map between entire states across domains.
We have also included a comparison between using time-based alignment across domains versus using a more elaborate EM-style procedure as described in 3.3.2.
5.2 TRANSFER BETWEEN ROBOTS WITH DIFFERENT NUMBERS OF LINKS
In our first experiment, we evaluate our method on transferring information from a 3-link robot to a 4-link robot. These robots have similar size but different numbers of links and actuators, making the representation needed for transfer non-trivial to learn. In order to evaluate the effectiveness of our method, we consider tasks with sparse or delayed rewards, which are difficult to learn quickly without the use of prior knowledge, large amounts of experience, or a detailed shaping function to guide exploration. For transfer between the 3 link and 4 link robots, we evaluate our method on a button pressing task as shown in Figures 2 and 3. The goal of this task is to reach through a narrow opening and press the white button to the red goal marker indicated in the figure. The caveat is that the reward signal tells the arms nothing about where the button is, but only penalizes distance between the white button and the red goal. Prior work has generally used well-shaped reward functions for tasks of this type, with terms that reward the arm for approaching the object of interest (Lillicrap et al., 2015; Devin et al., 2016). Without the presence of a directed reward shaping guiding the arm towards the button, it is very difficult for the task to be performed at all in the target domain, as seen from the performance of learning from scratch with no transfer (“baseline”) in the target domain in Figure 5. This is indicative of how such a task might be learned in the real world, where it is hard to provide anything but very sparse feedback by using a sensor on the button.
For this experiment, we compare the quality of transfer when using different proxy tasks: reaching a target, moving a white block to the red goal, and inserting a peg into a slot near the robot, as shown in Figure 4. These tasks are significantly easier than the sparse reward button pressing task. Collecting successful trajectories from the proxy task, we train the functions f and g as described in
Section 4. Note that the state in both robots is just the joint angles and joint velocities. Learning a suitable common feature space therefore requires the networks to understand how to map from joint angles to end-effectors for both robots.
We consider the 3-link robot pressing the button as the source domain and the 4-link robot pressing the button as the target domain. We allow the domain with the 3-link robot to have a well shaped cost function which has 2 terms: one for bringing the arm close to the button, and one for the distance of the button from the red goal position. The performance of our method is shown in Figure 5. The agent trained with our method performs more directed exploration and achieves an almost perfect success rate in 7 iterations. The CCA method requires about 4 times more experience to reach 60% success than our method, indicating that using deep function approximators for the functions f and g which allows for a more expressive mapping than CCA. Even with kernel CCA, the task is not able to be performed as well as our method. Additionally the UMA and random projections baselines perform much worse than our method. We additionally find that using the EM style alignment procedure described in 3.3.2 also allows us to reach perfect formance as shown in Figure 5. Investigating this method further will be the subject of future work.
Learning a direct mapping between states in both domains only provides limited transfer because this approach is forced to learn a mapping directly from one state space to the other, even though there is often no complete correspondence between two morphologically different robots. For example there may be some parts of the state which can be put in correspondence, but others which cannot. Our method of learning a common space between robots allows the embedding functions to only retain transferable information.
5.3 TRANSFER BETWEEN TORQUE CONTROLLED AND TENDON CONTROLLED MANIPULATORS
In order to illustrate the ability of our method to transfer across vastly different actuation mechanisms and learn representations that are hard to specify by hand, we consider transfer between a torque driven arm and a tendon driven arm, both with 3 links. These arms are pictured in Figure 6. The torque driven arm has motors at each of its joints that directly control its motion, and the state includes joint angles and joint velocities. The tendon driven arm, illustrated in Figure 6, uses three tendons to actuate the joints. The first tendon spans both the shoulder and the elbow, while the second and third control the elbow and wrist individually. The last tendon has a variable-length
lever arm, while the first two have fixed-length lever arms, corresponding to tendons that conform to the arm as it bends. This coupled system uses tendon lengths and tendon velocities as the state representation, without direct access to joint angles or end-effector positions.
The state representations of the two robots are dramatically different, both in terms of units, dimensionality, and semantics. Therefore, learning a suitable common feature space represents a considerable challenge. In our evaluation, the torque driven arm is the source robot, and the tendon driven arm is the target robot. The task we require both robots to perform is a block pulling task indicated in Figure 7. This involves pulling a block in the direction indicated, which is nontrivial because it requires moving the arm under and around the block, which is restricted to only move in the directions indicated in Figure 6. With random exploration, the target robot is unable to perform directed exploration to get the arm to actually pull the block in the desired direction, as shown in Figure 8.
We use one proxy task in the experiment, which involves both arms reaching to various locations. With embedding functions f and g trained on optimal trajectories from the proxy task, we see that the transfer reward from our method enables the task to actually be performed with a tendon driven arm. The baseline of learning from scratch, which again corresponds to attempting to learn the task with the target tendon-driven arm from scratch, fails completely. The other methods of using CCA, and learning a direct mapping are able to achieve better performance than learning from scratch but learn slower. Kernel CCA with the quadratic kernel does competitively with our method but in turn performed very poorly on the button task so is not very consistent. Additionally, the random projection and UMA baselines perform quite poorly. The performance of the EM style alignment procedure is very similar to the standard time based alignment as seen in Figure 8, likely because the data is already quite time aligned across the domains. These results indicate that learning the common feature subspace can enable substantially accelerated learning in the target domain, and in fact can allow the target agent to learn a task that it fails to learn without any transfer rewards, and performs better than alternative methods.
5.4 TRANSFER THROUGH IMAGE FEATURES
A compelling use-case for learned common embeddings is in learning vision-based policies. In this experimental setup, we evaluate our method on learning embeddings from raw pixels instead of from robot state. Enabling transfer from extra high dimensional inputs like images would allow significantly more natural transfer across a variety of robots without restrictive assumptions about full state information.
We evaluate our method on transfer across a 3-link and a 4-link robot as in Section 5.2, but use images instead of state. Because images from the source and target domains are the same size and the same ”type”, we let g = f . We parametrize f as 3 convolutional layers with 5x5 filters and no pooling. A spatial softmax (Levine et al., 2016) is applied to the output of the third layer such that f outputs normalized pixel indices of feature points on the image. These “feature points” form the latent representation that we compare across domains. Intuitively the common “feature points” embeddings should represent parts of the robots which are common across different robots.
Embeddings between the domains are built using a proxy task of reaching to a point, similar to the one described in the previous experiments. The test task in this case is to push a white block
to a red target as shown in Figure 9a, which suffers from sparse rewards because the reward only accounts for the distance of the block from the goal. Unless the robot knows that it has to touch the block, it receives no reward and has unguided exploration. As shown in Figure 9b, our method is able to transfer meaningful information from source to target robot directly from raw images and successfully perform the task even in the presence of sparse rewards.
(a) The 3-link robot demonstrating the task. The yellow triangles mark the locations of the feature points output by f applied to the image pixels. We then use the feature points to transfer the skill to the 4-link robot.
(b) Performance of 4-link robot on block pushing task for transfer using raw images. We transfer from the 3-link robot by learning a feature space from raw pixels of both domains, enabling effective faster learning. Random projections and linear kernel-CCA have some success in transfer. The baseline is unable to succeed because of the reward signal is too sparse without transfer.
6 DISCUSSION AND FUTURE WORK
We presented a method for transferring skills between morphologically different agents using invariant feature spaces. The formulation of our transfer problem corresponds to a setting where two agents (e.g. two different robots) have each learned a collection of skills, with some skills known to just one of the agents, and some shared by both. A shared skill can be used to learn a space that implicitly brings the agents into correspondence, without assuming that an explicit state space isomorphism can be constructed. By then mapping into this space a skill that is known to only one
of the agents, the other agent can substantially accelerate its learning of this skill by transferring the shared structure. We present an algorithm for learning the shared feature spaces using a shared proxy task, and experimentally illustrate that we can use this method to transfer manipulation skills between different simulated robotic arms. Our experiments include transfer between arms with different numbers of links, as well as transfer from a torque-driven arm to a tendon-driven arm.
A promising direction for future work is to explicitly handle situations where the two (or more) agents must transfer new skills by using a large collection of prior behaviors, with different degrees of similarity between the agents. In this case, constructing a shared feature space involves not only mapping the skills into a single space, but deciding which skills should or should not be combined. For example, a wheeled robot might share manipulation strategies with a legged robot, but should not attempt to share locomotion behaviors.
In a large-scale lifelong learning domain with many agent and many skills, we could also consider using our approach to gradually construct more and more detailed common feature spaces by transferring a skill from one agent to another, using that new skill to build a better common feature space, and then using this improved feature space to transfer more skills. Automatically choosing which skills to transfer when in order to minimize the training time of an entire skill repertoire is an interesting and exciting direction for future work.
7 APPENDIX
7.1 REINFORCEMENT LEARNING WITH LOCAL MODELS
Although we can use any suitable reinforcement learning algorithm for learning policies, in this work, we use a simple trajectory-centric reinforcement learning method that trains time-varying linear-Gaussian policies (Levine & Abbeel, 2014). While this method produces simple policies, it is very efficient, making it well suited for robotic learning. To obtain robot trajectories for training tasks and source robots, we optimize time-varying linear-Gaussian policies through a trajectorycentric reinforcement learning algorithm that alternates between fitting local time-varying linear dynamics models, and updating the time-varying linear-Gaussian policies using the iterative linearquadratic Gaussian regulator algorithm (iLQG) (Li & Todorov, 2004). This approach is simple and efficient, and is typically able to learn complex high-dimensional skills using just tens of trials, making it well suited for rapid transfer. The resulting time-varying linear-Gaussian policies are parametrized as p(ut |xt) = N (Ktxt + kt ,Ct) where Kt , kt , and Ct are learned parameters. Further details of this method are presented in prior work (Levine & Abbeel, 2014).
We use the same reinforcement learning algorithm to provide solutions in the source domain DS, though again any suitable reinforcement learning method (or even human demonstrations) could be used instead. To evaluate the ability of our method to provide detailed guidance through the transfer reward rtransfer, we use relatively sparse reward functions in the target domain DT , as discussed below. To generate the original skills in the source domain DS and in the proxy domains DSp and DT p, we manually designed the appropriate shaped costs to enable learning from scratch to succeed, though we note again that our method is agnostic to how the source domain and proxy domain skills are acquired.

This paper presents an approach for skills transfer from one task to another in a control setting (trained by RL) by forcing the embeddings learned on two different tasks to be close (L2 penalty). The experiments are conducted in MuJoCo, with a set of experiments being from the state of the joints/links (5.2/5.3) and a set of experiments on the pixels (5.4). They exhibit transfer from arms with different number of links, and from a torque-driven arm to a tendon-driven arm.
One limitation of the paper is that the authors suppose that time alignment is trivial, because the tasks are all episodic and in the same domain. Time alignment is one form of domain adaptation / transfer that is not dealt with in the paper, that could be dealt with through subsampling, dynamic time warping, or learning a matching function (e.g. neural network).
General remarks: The approach is compared to CCA, which is a relevant baseline. However, as the paper is purely experimental, another baseline (worse than CCA) would be to just have the random projections for "f" and "g" (the embedding functions on the two domains), to check that the bad performance of the "no transfer" version of the model is due to over-specialisation of these embeddings. I would also add (for information) that the problem of learning invariant feature spaces is also linked to metric learning (e.g. [Xing et al. 2002]). More generally, no parallel is drawn with multi-task learning in ML. In the case of knowledge transfer (4.1.1), it may make sense to anneal \alpha.
The experiments feel a bit rushed. In particular, the performance of the baseline being always 0 (no transfer at all) is uninformative, at least a much bigger sample budget should be tested. Also, why does Figure 7.b contain no "CCA" nor "direct mapping" results? Another concern that I have with the experiments: (if/how) did the author control for the fact that the embeddings were trained with more iterations in the case of doing transfer?
Overall, the study of transfer is most welcomed in RL. The experiments in this paper are interesting enough for publication, but the paper could have been more thorough.


MENTS THROUGH CORRUPTION
We present an efficient document representation learning framework, Document Vector through Corruption (Doc2VecC). Doc2VecC represents each document as a simple average of word embeddings. It ensures a representation generated as such captures the semantic meanings of the document during learning. A corruption model is included, which introduces a data-dependent regularization that favors informative or rare words while forcing the embeddings of common and non-discriminative ones to be close to zero. Doc2VecC produces significantly better word embeddings than Word2Vec. We compare Doc2VecC with several state-of-the-art document representation learning algorithms. The simple model architecture introduced by Doc2VecC matches or out-performs the state-of-the-art in generating high-quality document representations for sentiment analysis, document classification as well as semantic relatedness tasks. The simplicity of the model enables training on billions of words per hour on a single machine. At the same time, the model is very efficient in generating representations of unseen documents at test time.
1 INTRODUCTION
Text understanding starts with the challenge of finding machine-understandable representation that captures the semantics of texts. Bag-of-words (BoW) and its N-gram extensions are arguably the most commonly used document representations. Despite its simplicity, BoW works surprisingly well for many tasks (Wang & Manning, 2012). However, by treating words and phrases as unique and discrete symbols, BoW often fails to capture the similarity between words or phrases and also suffers from sparsity and high dimensionality.
Recent works on using neural networks to learn distributed vector representations of words have gained great popularity. The well celebrated Word2Vec (Mikolov et al., 2013a), by learning to predict the target word using its neighboring words, maps words of similar meanings to nearby points in the continuous vector space. The surprisingly simple model has succeeded in generating high-quality word embeddings for tasks such as language modeling, text understanding and machine translation. Word2Vec naturally scales to large datasets thanks to its simple model architecture. It can be trained on billions of words per hour on a single machine.
Paragraph Vectors (Le & Mikolov, 2014) generalize the idea to learn vector representation for documents. A target word is predicted by the word embeddings of its neighbors in together with a unique document vector learned for each document. It outperforms established document representations, such as BoW and Latent Dirichlet Allocation (Blei et al., 2003), on various text understanding tasks (Dai et al., 2015). However, two caveats come with this approach: 1) the number of parameters grows with the size of the training corpus, which can easily go to billions; and 2) it is expensive to generate vector representations for unseen documents at test time.
We propose an efficient model architecture, referred to as Document Vector through Corruption (Doc2VecC), to learn vector representations for documents. It is motivated by the observation that linear operations on the word embeddings learned by Word2Vec can sustain substantial amount of syntactic and semantic meanings of a phrase or a sentence (Mikolov et al., 2013b). For example, vec(“Russia”) + vec(“river”) is close to vec(“Volga River”) (Mikolov & Dean, 2013), and
vec(“king”) - vec(“man”) + vec(“women”) is close to vec(“queen”) (Mikolov et al., 2013b). In Doc2VecC, we represent each document as a simple average of the word embeddings of all the words in the document. In contrast to existing approaches which post-process learned word embeddings to form document representation (Socher et al., 2013; Mesnil et al., 2014), Doc2VecC enforces a meaningful document representation can be formed by averaging the word embeddings during learning. Furthermore, we include a corruption model that randomly remove words from a document during learning, a mechanism that is critical to the performance and learning speed of our algorithm.
Doc2VecC has several desirable properties: 1. The model complexity of Doc2VecC is decoupled from the size of the training corpus, depending only on the size of the vocabulary; 2. The model architecture of Doc2VecC resembles that of Word2Vec, and can be trained very efficiently; 3. The new framework implicitly introduces a data-dependent regularization, which favors rare or informative words and suppresses words that are common but not discriminative; 4. Vector representation of a document can be generated by simply averaging the learned word embeddings of all the words in the document, which significantly boost test efficiency; 5. The vector representation generated by Doc2VecC matches or beats the state-of-the-art for sentiment analysis, document classification as well as semantic relatedness tasks.
2 RELATED WORKS AND NOTATIONS
Text representation learning has been extensively studied. Popular representations range from the simplest BoW and its term-frequency based variants (Salton & Buckley, 1988), language model based methods (Croft & Lafferty, 2013; Mikolov et al., 2010; Kim et al., 2015), topic models (Deerwester et al., 1990; Blei et al., 2003), Denoising Autoencoders and its variants (Vincent et al., 2008; Chen et al., 2012), and distributed vector representations (Mesnil et al., 2014; Le & Mikolov, 2014; Kiros et al., 2015). Another prominent line of work includes learning task-specific document representation with deep neural networks, such as CNN (Zhang & LeCun, 2015) or LSTM based approaches (Tai et al., 2015; Dai & Le, 2015).
In this section, we briefly introduce Word2Vec and Paragraph Vectors, the two approaches that are most similar to ours. There are two well-know model architectures used for both methods, referred to as Continuous Bag-of-Words (CBoW) and Skipgram models (Mikolov et al., 2013a). In this work, we focus on CBoW. Extending to Skipgram is straightforward. Here are the notations we are going to use throughout the paper:
D = {D1, · · · , Dn}: a training corpus of size n, in which each document Di contains a variablelength sequence of words w1i , · · · , w Ti i ;
V : the vocabulary used in the training corpus, of sizes v;
x ∈ Rv×1: BoW of a document, where xj = 1 iff word j does appear in the document. ct ∈ Rv×1: BoW of the local context wt−k, · · · , wt−1, wt+1, · · · , wt+k at the target position t. ctj = 1 iff word j appears within the sliding window of the target;
U ∈ Rh×v: the projection matrix from the input space to a hidden space of size h. We use uw to denote the column in U for word w, i.e., the “input“ vector of word w;
V> ∈ Rv×h: the projection matrix from the hidden space to output. Similarly, we use vw to denote the column in V for word w, i.e., the “output“ vector of word w.
Word2Vec. Word2Vec proposed a neural network architecture of an input layer, a projection layer parameterized by the matrix U and an output layer by V>. It defines the probability of observing the target word wt in a document D given its local context ct as
P (wt|ct) = exp(v>wtUc t)∑ w′∈V exp(v > w′Uc t)
The word vectors are then learned to maximize the log likelihood of observing the target word at each position of the document. Various techniques (Mitchell & Lapata, 2010; Zanzotto et al., 2010; Yessenalina & Cardie, 2011; Grefenstette et al., 2013; Socher et al., 2013; Kusner et al., 2015)
have been studied to generate vector representations of documents from word embeddings, among which the simplest approach is to use weighted average of word embeddings. Similarly, our method forms document representation by averaging word embeddings of all the words in the document. Differently, as our model encodes the compositionality of words in the learned word embeddings, heuristic weighting at test time is not required.
Paragraph Vectors. Paragraph Vectors, on the other hands, explicitly learns a document vector with the word embeddings. It introduces another projection matrix D ∈ Rh×n. Each column of D acts as a memory of the global topic of the corresponding document. It then defines the probability of observing the target word wt in a document D given its local context ct as
P (wt|ct,d) = exp(v>wt(Uc t + d))∑ w′∈V exp(v > w′(Uc t + d))
where d ∈ D is the vector representation of the document. As we can see from this formula, the complexity of Paragraph Vectors grows with not only the size of the vocabulary, but also the size of the training corpus. While we can reasonably limit the size of a vocabulary to be within a million for most datasets, the size of a training corpus can easily go to billions. What is more concerning is that, in order to come up with the vector representations of unseen documents, we need to perform an expensive inference by appending more columns to D and gradient descent on D while fixing other parameters of the learned model.
3 METHOD
Several works (Mikolov & Dean, 2013; Mikolov et al., 2013b) showcased that syntactic and semantic regularities of phrases and sentences are reasonably well preserved by adding or subtracting word embeddings learned through Word2Vec. It prompts us to explore the option of simply representing a document as an average of word embeddings. Figure 1 illustrates the new model architecture.
Similar to Word2Vec or Paragraph Vectors, Doc2VecC consists of an input layer, a projection layer as well as an output layer to predict the target word, “ceremony” in this example. The embeddings of neighboring words (“opening”, “for”, “the”) provide local context while the vector representation of the entire document (shown in grey) serves as the global context. In contrast to Paragraph Vectors, which directly learns a unique vector for each document, Doc2VecC represents each document as an average of the embeddings of words randomly sampled from the document (“performance” at position p, “praised” at position q, and “brazil” at position r).
Huang et al. (2012) also proposed the idea of using average of word embeddings to represent the global context of a document. Different from their work, we choose to corrupt the original document by randomly removing significant portion of words, and represent the document using only the embeddings of the words remained. This corruption mechanism offers us great speedup during training as it significantly reduces the number of parameters to update in back propagation. At the same time, as we are going to detail in the next section, it introduces a special form of regularization, which brings great performance improvement.
Here we describe the stochastic process we used to generate a global context at each update. The global context, which we denote as x̃, is generated through a unbiased mask-out/drop-out corruption, in which we randomly overwrites each dimension of the original document x with probability q. To make the corruption unbiased, we set the uncorrupted dimensions to 1/(1 − q) times its original value. Formally,
x̃d = { 0, with probability q xd 1−q , otherwise
(1)
Doc2VecC then defines the probability of observing a target word wt given its local context ct as well as the global context x̃ as
P (wt|ct, x̃) = exp(v>wt(
local context︷︸︸︷ Uct + global context︷ ︸︸ ︷ 1
T Ux̃ ))∑
w′∈V exp(v > w′ ( Uct + 1T Ux̃) ) (2) Here T is the length of the document. Exactly computing the probability is impractical, instead we approximate it with negative sampling (Mikolov et al., 2013a).
f(w, c, x̃) ≡ logP (wt|ct, x̃) ≈ log σ ( v>w(Uc + 1
T Ux̃)
) + ∑
w′∼Pv
log σ ( −v>w′(Uc + 1
T Ux̃)
) (3)
here Pv stands for a uniform distribution over the terms in the vocabulary. The two projection matrices U and V are then learned to minimize the loss:
` = − n∑
i=1 Ti∑ t=1 f(wti , c t i, x̃ t i) (4)
Given the learned projection matrix U, we then represent each document simply as an average of the embeddings of the words in the document,
d = 1
T ∑ w∈D uw. (5)
We are going to elaborate next why we choose to corrupt the original document with the corruption model in eq.(1) during learning, and how it enables us to simply use the average word embeddings as the vector representation for documents at test time.
3.1 CORRUPTION AS DATA-DEPENDENT REGULARIZATION
We approximate the log likelihood for each instance f(w, c, x̃) in eq.(4) with its Taylor expansion with respect to x̃ up to the second-order (Van Der Maaten et al., 2013; Wager et al., 2013; Chen et al., 2014). Concretely, we choose to expand at the mean of the corruption µx = Ep(x̃|x)[x̃]:
f(w, c, x̃) ≈ f(w, c, µx) + (x̃− µx)>∇x̃f + 1
2 (x̃− µx)>∇2x̃f(x̃− µx)
where ∇x̃f and ∇2x̃f are the first-order (i.e., gradient) and second-order (i.e., Hessian) of the log likelihood with respect to x̃. Expansion at the mean µx is crucial as shown in the following steps. Let us assume that for each instance, we are going to sample the global context x̃ infinitely many times, and thus compute the expected log likelihood with respect to the corrupted x̃.
Ep(x̃|x)[f(w, c, x̃)] ≈ f(w, c, µx) + 1 2 tr ( E[(x̃− x)(x̃− x)>]∇2x̃f ) The linear term disappears as Ep(x̃|x)[x̃ − µx] = 0. We substitute in x for the mean µx of the corrupting distribution (unbiased corruption) and the matrix Σx = E[(x̃ − µx)(x̃ − µx)>] for the variance, and obtain
Ep(x̃|x)[f(w, c, x̃)] ≈ f(w, c,x) + 1 2 tr ( Σx∇2x̃f ) (6)
As each word in a document is corrupted independently of others, the variance matrix Σx is simplified to a diagonal matrix with jth element equals q1−qx 2 j . As a result, we only need to compute the diagonal terms of the Hessian matrix∇2x̃f .
The jth dimension of the Hessian’s diagonal evaluated at the mean x is given by
∂2f ∂x2j = −σw,c,x(1− σw,c,x)( 1 T v>wuj)
2 − ∑
w′∼Pv
σw′,c,x(1− σw′,c,x)( 1
T v>w′uj) 2
Plug the Hessian matrix and the variance matrix back into eq.(6), and then back to the loss defined in eq.(4), we can see that Doc2VecC intrinsically minimizes
` = − n∑
i=1 Ti∑ t=1 f(wti , c t i,xi) + q 1− q v∑ j=1 R(uj) (7)
Each f(wti , c t i,xi) in the first term measures the log likelihood of observing the target word w t i given its local context cti and the document vector di = 1 T Uxi. As such, Doc2VecC enforces that a document vector generated by averaging word embeddings can capture the global semantics of the document, and fill in information missed in the local context.
The second term here is a data-dependent regularization. The regularization on the embedding uj of each word j takes the following form, R(uj) ∝ n∑
i=1 Ti∑ t=1 x2ij [ σwti ,cti,xi(1− σwti ,cti,xi)( 1 T v>wti uj) 2 + ∑ w′∼Pv σw′,cti,xi(1− σw′,cti,xi)( 1 T v>w′uj) 2 ] where σw,c,x = σ(v>w(Uc+ 1 T Ux)) prescribes the confidence of predicting the target wordw given its neighboring context c as well as the document vector d = 1T Ux.
Closely examining R(uj) leads to several interesting findings: 1. the regularizer penalizes more on the embeddings of common words. A word j that frequently appears across the training corpus, i.e, xij = 1 often, will have a bigger regularization than a rare word; 2. on the other hand, the regularization is modulated by σw,c,x(1 − σw,c,x), which is small if σw,c,x → 1 or 0. In other words, if uj is critical to a confident prediction σw,c,x when it is active, then the regularization is diminished. Similar effect was observed for dropout training for logistic regression model (Wager et al., 2013) and denoising autoencoders (Chen et al., 2014).
4 EXPERIMENTS
We evaluate Doc2VecC on a sentiment analysis task, a document classification task and a semantic relatedness task, along with several document representation learning algorithms. All experiments can be reproduced using the code available at https://github.com/mchen24/iclr2017
4.1 BASELINES
We compare against the following document representation baselines: bag-of-words (BoW); Denoising Autoencoders (DEA) (Vincent et al., 2008), a representation learned from reconstructing original document x using corrupted one x̃. SDAs have been shown to be the state-of-the-art for sentiment analysis tasks (Glorot et al., 2011). We used Kullback-Liebler divergence as the reconstruction error and an affine encoder. To scale up the algorithm to large vocabulary, we only take into account the non-zero elements of x in the reconstruction error and employed negative sampling for the remainings; Word2Vec (Mikolov et al., 2013a)+IDF, a representation generated through weighted average of word vectors learned using Word2Vec; Doc2Vec (Le & Mikolov, 2014); Skip-thought Vectors(Kiros et al., 2015), a generic, distributed sentence encoder that extends the Word2Vec skipgram model to sentence level. It has been shown to produce highly generic sentence representations that apply to various natural language processing tasks. We also include RNNLM (Mikolov et al., 2010), a recurrent neural network based language model in the comparison. In the semantic relatedness task, we further compare to LSTM-based methods (Tai et al., 2015) that have been reported on this dataset.
4.2 SENTIMENT ANALYSIS
For sentiment analysis, we use the IMDB movie review dataset. It contains 100,000 movies reviews categorized as either positive or negative. It comes with predefined train/test split (Maas et al., 2011): 25,000 reviews are used for training, 25,000 for testing, and the rest as unlabeled data. The two classes are balanced in the training and testing sets. We remove words that appear less than 10 times in the training set, resulting in a vocabulary of 43,375 distinct words and symbols.
Setup. We test the various representation learning algorithms under two settings: one follows the same protocol proposed in (Mesnil et al., 2014), where representation is learned using all the available data, including the test set; another one where the representation is learned using training and unlabeled set only. For both settings, a linear support vector machine (SVM) (Fan et al., 2008) is trained afterwards on the learned representation for classification. For Skip-thought Vectors, we used the generic model1 trained on a much bigger book corpus to encode the documents. A vector of 4800 dimensions, first 2400 from the uni-skip model, and the last 2400 from the bi-skip model, are generated for each document. In comparison, all the other algorithms produce a vector representation of size 100. The supervised RNN-LM is learned on the training set only. The hyper-parameters are tuned on a validation set subsampled from the training set.
Accuracy. Comparing the two columns in Table 1, we can see that all the representation learning algorithms benefits from including the testing data during the representation learning phrase. Doc2VecC achieved similar or even better performance than Paragraph Vectors. Both methods outperforms the other baselines, beating the BOW representation by 15%. In comparison with Word2Vec+IDF, which applies post-processing on learned word embeddings to form document representation, Doc2VecC naturally enforces document semantics to be captured by averaged word embeddings during training. This leads to better performance. Doc2VecC reduces to Denoising Autoencoders (DEA) if the local context words are removed from the paradigm shown in Figure 1. By including the context words, Doc2VecC allows the document vector to focus more on capturing the global context. Skip-thought vectors perform surprisingly poor on this dataset comparing to other methods. We hypothesized that it is due to the length of paragraphs in this dataset. The average length of paragraphs in the IMDB movie review dataset is 296.5, much longer than the ones used for training and testing in the original paper, which is in the order of 10. As noted in (Tai et al., 2015), the performance of LSTM based method (similarly, the gated RNN used in Skip-thought vectors) drops significantly with increasing paragraph length, as it is hard to preserve state over long sequences of words.
Time. Table 2 summarizes the time required by these algorithms to learn and generate the document representation. Word2Vec is the fastest one to train. Denoising Autoencoders and Doc2VecC second that. The number of parameters that needs to be back-propagated in each update was increased by the number of surviving words in x̃. We found that both models are not sensitive to the corruption rate q in the noise model. Since the learning time decreases with higher corruption rate, we used q = 0.9 throughout the experiments. Paragraph Vectors takes longer time to train as there are more parameters (linear to the number of document in the learning set) to learn. At test time, Word2Vec+IDF, DEA and Doc2VecC all use (weighted) averaging of word embeddings as document
1available at https://github.com/ryankiros/skip-thoughts
representation. Paragraph Vectors, on the other hand, requires another round of inference to produce the vector representation of unseen test documents. It takes Paragraph Vectors 4 minutes and 17 seconds to infer the vector representations for the 25,000 test documents, in comparison to 7 seconds for the other methods. As we did not re-train the Skip-thought vector models on this dataset, the training time2 reported in the table is the time it takes to generate the embeddings for the 25,000 training documents. Due to repeated high-dimensional matrix operations required for encoding long paragraphs, it takes fairly long time to generate the representations for these documents. Similarly for testing. The experiments were conducted on a desktop with Intel i7 2.2Ghz cpu.
Data dependent regularization. As explained in Section 3.1, the corruption introduced in Doc2VecC acts as a data-dependent regularization that suppresses the embeddings of frequent but uninformative words. Here we conduct an experiment to exam the effect. We used a cutoff of 100 in this experiment. Table 3 lists the words having the smallest l2 norm of embeddings found by different algorithms. The number inside the parenthesis after each word is the number of times this word appears in the learning set. In word2Vec or Paragraph Vectors, the least frequent words have embeddings that are close to zero, despite some of them being indicative of sentiment such as debacle, bliss and shabby. In contrast, Doc2VecC manages to clamp down the representation of words frequently appear in the training set, but are uninformative, such as symbols and stop words.
Subsampling frequent words. Note that for all the numbers reported, we applied the trick of subsampling of frequent words introduced in (Mikolov & Dean, 2013) to counter the imbalance between frequent and rare words. It is critical to the performance of simple Word2Vec+AVG as the sole remedy to diminish the contribution of common words in the final document representation. If we were to remove this step, the error rate of Word2Vec+AVG will increases from 12.1% to 13.2%. Doc2VecC on the other hand naturally exerts a stronger regularization toward embeddings of words that are frequent but uninformative, therefore does not rely on this trick.
4.3 WORD ANALOGY
In table 3, we demonstrated that the corruption model introduced in Doc2VecC dampens the embeddings of words which are common and non-discriminative (stop words). In this experiment, we are going to quantatively compare the word embeddings generated by Doc2VecC to the ones generated by Word2Vec, or Paragraph Vectors on the word analogy task introduced by Mikolov et al. (2013a). The dataset contains five types of semantic questions, and nine types of syntactic questions, with a total of 8,869 semantic and 10,675 syntactic questions. The questions are answered through simple linear algebraic operations on the word embeddings generated by different methods. Please refer to the original paper for more details on the evaluation protocol.
2As reported in the original paper, training of the skip-thought vector model on the book corpus dataset takes around 2 weeks on GPU.
We trained the word embeddings of different methods using the English news dataset released under the ACL workshop on statistical machine translation. The training set includes close to 15M paragraphs with 355M tokens. We compare the performance of word embeddings trained by different methods with increasing embedding dimensionality as well as increasing training data.
We observe similar trends as in Mikolov et al. (2013a). Increasing embedding dimensionality as well as training data size improves performance of the word embeddings on this task. However, the improvement is diminishing. Doc2VecC produces word embeddings which performs significantly better than the ones generated by Word2Vec. We observe close to 20% uplift when we train on the full training corpus. Paragraph vectors on the other hand performs surprisingly bad on this dataset. Our hypothesis is that due to the large capacity of the model architecture, Paragraph Vectors relies mostly on the unique document vectors to capture the information in a text document instead of learning the word semantic or syntactic similarities. This also explains why the PV-DBOW Le & Mikolov (2014) model architecture proposed in the original work, which completely removes word embedding layers, performs comparable to the distributed memory version.
In table 5, we list a detailed comparison of the performance of word embeddings generated by Word2Vec and Doc2VecC on the 14 subtasks, when trained on the full dataset with embedding of size 100. We can see that Doc2VecC significantly outperforms the word embeddings produced by Word2Vec across almost all the subtasks.
4.4 DOCUMENT CLASSIFICATION
For the document classification task, we use a subset of the wikipedia dump, which contains over 300,000 wikipedia pages in 100 categories. The 100 categories includes categories under sports,
Table 5: Classification error (%) of a linear classifier trained on various document representations on the Wikipedia dataset.
Model BOW DEA Word2Vec + AVG Word2Vec + IDF ParagraphVectors Doc2VecC h = 100 36.03 32.30 33.2 33.16 35.78 31.92 h = 200 36.03 31.36 32.46 32.48 34.92 30.84 h = 500 36.03 31.10 32.02 32.13 33.93 30.43 h = 1000 36.03 31.13 31.78 32.06 33.02 30.24
(a) Doc2Vec (b) Doc2VecC
Figure 3: Visualization of document vectors on Wikipedia dataset using t-SNE.
entertainment, literature, and politics etc. Examples of categories include American drama films, Directorial debut films, Major League Baseball pitchers and Sydney Swans players. Body texts (the second paragraph) were extracted for each page as a document. For each category, we select 1,000 documents with unique category label, and 100 documents were used for training and 900 documents for testing. The remaining documents are used as unlabeled data. The 100 classes are balanced in the training and testing sets. For this data set, we learn the word embedding and document representation for all the algorithms using all the available data. We apply a cutoff of 10, resulting in a vocabulary of size 107, 691.
Table 5 summarizes the classification error of a linear SVM trained on representations of different sizes. We can see that most of the algorithms are not sensitive to the size of the vector representation. Doc2Vec benefits most from increasing representation size. Across all sizes of representations, Doc2VecC outperform the existing algorithms by a significant margin. In fact, Doc2VecC can achieve same or better performance with a much smaller representation vector.
Figure 3 visualizes the document representations learned by Doc2Vec (left) and Doc2VecC (right) using t-SNE (Maaten & Hinton, 2008). We can see that documents from the same category are nicely clustered using the representation generated by Doc2VecC. Doc2Vec, on the other hand, does not produce a clear separation between different categories, which explains its worse performance reported in Table 5.
Figure 4 visualizes the vector representation generated by Doc2VecC w.r.t. coarser categorization. we manually grouped the 100 categories into 7 coarse categories, television, albums, writers, musicians, athletes, species and actors. Categories that do no belong to any of these 7 groups are not included in the figure.
We can see that documents belonging to a coarser category are grouped together. This subset includes is a wide range of sports descriptions, ranging from football, crickets, baseball, and cycling etc., which explains why the athletes category are less concentrated. In the projection, we can see documents belonging to the musician category are closer to those belonging to albums category than those of athletes or species.
4.5 SEMANTIC RELATEDNESS
We test Doc2VecC on the SemEval 2014 Task 1: semantic relatedness SICK dataset (Marelli et al., 2014). Given two sentences, the task is to determine how closely they are semantically related. The set contains 9,927 pairs of sentences with human annotated relatedness score, ranging from 1 to 5. A score of 1 indicates that the two sentences are not related, while 5 indicates high relatedness. The set is splitted into a training set of 4,500 instances, a validation set of 500, and a test set of 4,927.
We compare Doc2VecC with several winning solutions of the competition as well as several more recent techniques reported on this dataset, including bi-directional LSTM and Tree-LSTM3 trained from scratch on this dataset, Skip-thought vectors learned a large book corpus 4 (Zhu et al., 2015) and produced sentence embeddings of 4,800 dimensions on this dataset. We follow the same protocol as in skip-thought vectors, and train Doc2VecC on the larger book corpus dataset. Contrary to the vocabulary expansion technique used in (Kiros et al., 2015) to handle out-of-vocabulary words, we extend the vocabulary of the learned model directly on the target dataset in the following way: we use the pre-trained word embedding as an initialization, and fine-tune the word and sentence representation on the SICK dataset. Notice that the fine-tuning is done for sentence representation learning only, and we did not use the relatedness score in the learning. This step brings small improvement to the performance of our algorithm. Given the sentence embeddings, we used the exact same training and testing protocol as in (Kiros et al., 2015) to score each pair of sentences: with two sentence embedding u1 and u2, we concatenate their component-wise product, u1 ·u2 and their absolute difference, |u1 − u2| as the feature representation. Table 6 summarizes the performance of various algorithms on this dataset. Despite its simplicity, Doc2VecC significantly out-performs the winning solutions of the competition, which are heavily feature engineered toward this dataset and several baseline methods, noticeably the dependency-tree RNNs introduced in (Socher et al., 2014), which relies on expensive dependency parsers to compose sentence vectors from word embeddings. The performance of Doc2VecC is slightly worse than the LSTM based methods or skip-thought vectors on this dataset, while it significantly outperforms skip-thought vectors on the IMDB movie review dataset (11.70% error rate vs 17.42%). As we hypothesized in previous section, while Doc2VecC is better at handling longer paragraphs, LSTMbased methods are superior for relatively short sentences (of length in the order of 10s). We would like to point out that Doc2VecC is much faster to train and test comparing to skip-thought vectors. It takes less than 2 hours to learn the embeddings on the large book corpus for Doc2VecC on a desktop with Intel i7 2.2Ghz cpu, in comparison to the 2 weeks on GPU required by skip-thought vectors.
5 CONCLUSION
We introduce a new model architecture Doc2VecC for document representation learning. It is very efficient to train and test thanks to its simple model architecture. Doc2VecC intrinsically makes sure document representation generated by averaging word embeddings capture semantics of document during learning. It also introduces a data-dependent regularization which favors informative or rare words while dampening the embeddings of common and non-discriminative words. As such, each document can be efficiently represented as a simple average of the learned word embeddings. In comparison to several existing document representation learning algorithms, Doc2VecC outperforms not only in testing efficiency, but also in the expressiveness of the generated representations.
3The word representation was initialized using publicly available 300-dimensional Glove vectors trained on 840 billion tokens of Common Crawl data
4The dataset contains 11,038 books with over one billion words

This paper proposes learning document embeddings as a sum of the constituent word embeddings, which are jointly learned and randomly dropped out ('corrupted') during training. While none of the pieces of this model are particularly novel, the result is an efficient learning algorithm for document representation with good empirical performance.
Joint training of word and document embeddings is not a new idea, nor is the idea of enforcing the document to be represented by the sum of its word embeddings (see, e.g. '“The Sum of Its Parts”: Joint Learning of Word and Phrase Representations with Autoencoders' by Lebret and Collobert). Furthermore, the corruption mechanism is nothing other than traditional dropout on the input layer. Coupled with the word2vec-style loss and training methods, this paper offers little on the novelty front.
On the other hand, it is very efficient at generation time, requiring only an average of the word embeddings rather than a complicated inference step as in Doc2Vec. Moreover, by construction, the embedding captures salient global information about the document -- it captures specifically that information that aids in local-context prediction. For such a simple model, the performance on sentiment analysis and document classification is quite encouraging.
Overall, despite the lack of novelty, the simplicity, efficiency, and performance of this model make it worthy of wider readership and study, and I recommend acceptance.