Learning to navigate in complex environments with dynamic elements is an important milestone in developing AI agents. In this work we formulate the navigation question as a reinforcement learning problem and show that data efficiency and task performance can be dramatically improved by relying on additional auxiliary tasks leveraging multimodal sensory inputs. In particular we consider jointly learning the goal-driven reinforcement learning problem with auxiliary depth prediction and loop closure classification tasks. This approach can learn to navigate from raw sensory input in complicated 3D mazes, approaching human-level performance even under conditions where the goal location changes frequently. We provide detailed analysis of the agent behaviour1, its ability to localise, and its network activity dynamics, showing that the agent implicitly learns key navigation abilities.
1 INTRODUCTION
The ability to navigate efficiently within an environment is fundamental to intelligent behavior. Whilst conventional robotics methods, such as Simultaneous Localisation and Mapping (SLAM), tackle navigation through an explicit focus on position inference and mapping (Dissanayake et al., 2001), here we follow recent work in deep reinforcement learning (Mnih et al., 2015; 2016) and propose that navigational abilities could emerge as the by-product of an agent learning a policy that maximizes reward. One advantage of an intrinsic, end-to-end approach is that actions are not divorced from representation, but rather learnt together, thus ensuring that task-relevant features are present in the representation. Learning to navigate from reinforcement learning in partially observable environments, however, poses several challenges.
First, rewards are often sparsely distributed in the environment, where there may be only one goal location. Second, environments often comprise dynamic elements, requiring the agent to use memory at different timescales: rapid one-shot memory for the goal location, together with short term memory subserving temporal integration of velocity signals and visual observations, and longer term memory for constant aspects of the environment (e.g. boundaries, cues).
To improve statistical efficiency we bootstrap the reinforcement learning procedure by augmenting our loss with auxiliary tasks that provide denser training signals that support navigation-relevant representation learning. We consider two additional losses: the first one involves reconstruction of a low-dimensional depth map at each time step by predicting one input modality (the depth channel) from others (the colour channels). This auxiliary task concerns the 3D geometry of the environment, and is aimed to encourage the learning of representations that aid obstacle avoidance and short-term trajectory planning. The second task directly invokes loop closure from SLAM: the agent is trained to predict if the current location has been previously visited within a local trajectory. ∗Denotes equal contribution 1A video illustrating the navigation agents is available at: https://youtu.be/lNoaTyMZsWI
To address the memory requirements of the task we rely on a stacked LSTM architecture (Graves et al., 2013; Pascanu et al., 2013). We evaluate our approach using five 3D maze environments and demonstrate the accelerated learning and increased performance of the proposed agent architecture. These environments feature complex geometry, random start position and orientation, dynamic goal locations, and long episodes that require thousands of agent steps (see Figure 1). We also provide detailed analysis of the trained agent to show that critical navigation skills are acquired. This is important as neither position inference nor mapping are directly part of the loss; therefore, raw performance on the goal finding task is not necessarily a good indication that these skills are acquired. In particular, we show that the proposed agent resolves ambiguous observations and quickly localizes itself in a complex maze, and that this localization capability is correlated with higher task reward.
2 APPROACH
We rely on a end-to-end learning framework that incorporates multiple objectives. Firstly it tries to maximize cumulative reward using an actor-critic approach. Secondly it minimizes an auxiliary loss of inferring the depth map from the RGB observation. Finally, the agent is trained to detect loop closures as an additional auxiliary task that encourages implicit velocity integration.
The reinforcement learning problem is addressed with the Asynchronous Advantage Actor-Critic (A3C) algorithm (Mnih et al., 2016) that relies on learning both a policy π(at|st; θ) and value function V (st; θV ) given a state observation st. Both the policy and value function share all intermediate representations, both being computed using a separate linear layer from the topmost layer of the model. The agent setup closely follows the work of (Mnih et al., 2016) and we refer to this work for the details (e.g. the use of a convolutional encoder followed by either an MLP or an LSTM, the use of action repetition, entropy regularization to prevent the policy saturation, etc.). These details can also be found in the Appendix B.
The baseline that we consider in this work is an A3C agent (Mnih et al., 2016) that receives only RGB input from the environment, using either a recurrent or a purely feed-forward model (see Figure 2a,b). The encoder for the RGB input (used in all other considered architectures) is a 3 layer convolutional network. To support the navigation capability of our approach, we also rely on the Nav A3C agent (Figure 2c) which employs a two-layer stacked LSTM after the convolutional encoder. We expand the observations of the agents to include agent-relative velocity, the action sampled from the stochastic policy and the immediate reward, from the previous time step. We opt to feed the velocity and previously selected action directly to the second recurrent layer, with the first layer only receiving the reward. We postulate that the first layer might be able to make associations between reward and visual observations that are provided as context to the second layer from which the policy is computed. Thus, the observation st may include an image xt ∈ R3×W×H (where W and H are the width and
height of the image), the agent-relative lateral and rotational velocity vt ∈ R6, the previous action at−1 ∈ RNA , and the previous reward rt−1 ∈ R. Figure 2d shows the augmentation of the Nav A3C with the different possible auxiliary losses. In particular we consider predicting depth from the convolutional layer (we will refer to this choice as D1), or from the top LSTM layer (D2) or predicting loop closure (L). The auxiliary losses are computed on the current frame via a single layer MLP. The agent is trained by applying a weighted sum of the gradients coming from A3C, the gradients from depth prediction (multiplied with βd1 , βd2 ) and the gradients from the loop closure (scaled by βl). More details of the online learning algorithm are given in Appendix B.
2.1 DEPTH PREDICTION
The primary input to the agent is in the form of RGB images. However, depth information, covering the central field of view of the agent, might supply valuable information about the 3D structure of the environment. While depth could be directly used as an input, we argue that if presented as an additional loss it is actually more valuable to the learning process. In particular if the prediction loss shares representation with the policy, it could help build useful features for RL much faster, bootstrapping learning. Since we know from (Eigen et al., 2014) that a single frame can be enough to predict depth, we know this auxiliary task can be learnt. A comparison between having depth as input versus as an additional loss is given in Appendix C, which shows significant gain for depth as a loss.
Since the role of the auxiliary loss is just to build up the representation of the model, we do not necessarily care about the specific performance obtained or nature of the prediction. We do care about the data efficiency aspect of the problem and also computational complexity. If the loss is to be useful for the main task, we should converge faster on it compared to solving the RL problem (using less data samples), and the additional computational cost should be minimal. To achieve this we use a low resolution variant of the depth map, reducing the screen resolution to 4x16 pixels2.
We explore two different variants for the loss. The first choice is to phrase it as a regression task, the most natural choice. While this formulation, combined with a higher depth resolution, extracts the most information, mean square error imposes a unimodal distribution (van den Oord et al., 2016). To address this possible issue, we also consider a classification loss, where depth at each position is discretised into 8 different bands. The bands are non-uniformally distributed such that we pay more attention to far-away objects (details in Appendix B). The motivation for the classification formulation is that while it greatly reduces the resolution of depth, it is more flexible from a learning perspective and can result in faster convergence (hence faster bootstrapping).
2The image is cropped before being subsampled to lessen the floor and ceiling which have little relevant depth information.
2.2 LOOP CLOSURE PREDICTION
Loop closure, like depth, is valuable for a navigating agent, since can be used for efficient exploration and spatial reasoning. To produce the training targets, we detect loop closures based on the similarity of local position information during an episode, which is obtained by integrating 2D velocity over time. Specifically, in a trajectory noted {p0, p1, . . . , pT }, where pt is the position of the agent at time t, we define a loop closure label lt that is equal to 1 if the position pt of the agent is close to the position pt′ at an earlier time t′. In order to avoid trivial loop closures on consecutive points of the trajectory, we add an extra condition on an intermediary position pt′′ being far from pt. Thresholds η1 and η2 provide these two limits. Learning to predict the binary loop label is done by minimizing the Bernoulli loss Ll between lt and the output of a single-layer output from the hidden representation ht of the last hidden layer of the model, followed by a sigmoid activation.
3 RELATED WORK
There is a rich literature on navigation, primarily in the robotics literature. However, here we focus on related work in deep RL. Deep Q-networks (DQN) have had breakthroughs in extremely challenging domains such as Atari (Mnih et al., 2015). Recent work has developed on-policy RL methods such as advantage actor-critic that use asynchronous training of multiple agents in parallel (Mnih et al., 2016). Recurrent networks have also been successfully incorporated to enable state disambiguation in partially observable environments (Koutnik et al., 2013; Hausknecht & Stone, 2015; Mnih et al., 2016; Narasimhan et al., 2015).
Deep RL has recently been used in the navigation domain. Kulkarni et al. (2016) used a feedforward architecture to learn deep successor representations that enabled behavioral flexibility to reward changes in the MazeBase gridworld, and provided a means to detect bottlenecks in 3D VizDoom. Zhu et al. (2016) used a feedforward siamese actor-critic architecture incorporating a pretrained ResNet to support navigation to a target in a discretised 3D environment. Oh et al. (2016) investigated the performance of a variety of networks with external memory (Weston et al., 2014) on simple navigation tasks in the Minecraft 3D block world environment. Tessler et al. (2016) also used the Minecraft domain to show the benefit of combining feedforward deep-Q networks with the learning of resuable skill modules (cf options: (Sutton et al., 1999)) to transfer between navigation tasks. Tai & Liu (2016) trained a convnet DQN-based agent using depth channel inputs for obstacle avoidance in 3D environments. Barron et al. (2016) investigated how well a convnet can predict the depth channel from RGB in the Minecraft environment, but did not use depth for training the agent.
Auxiliary tasks have often been used to facilitate representation learning (Suddarth & Kergosien, 1990). Recently, the incorporation of additional objectives, designed to augment representation learning through auxiliary reconstructive decoding pathways (Zhang et al., 2016; Rasmus et al., 2015; Zhao et al., 2015; Mirowski et al., 2010), has yielded benefits in large scale classification tasks. In deep RL settings, however, only two previous papers have examined the benefit of auxiliary tasks. Specifically, Li et al. (2016) consider a supervised loss for fitting a recurrent model on the hidden representations to predict the next observed state, in the context of imitation learning of sequences provided by experts, and Lample & Chaplot (2016) show that the performance of a DQN agent in a first-person shooter game in the VizDoom environment can be substantially enhanced by the addition of a supervised auxiliary task, whereby the convolutional network was trained on an enemy-detection task, with information about the presence of enemies, weapons, etc., provided by the game engine.
In contrast, our contribution addresses fundamental questions of how to learn an intrinsic representation of space, geometry, and movement while simultaneously maximising rewards through reinforcement learning. Our method is validated in challenging maze domains with random start and goal locations.
4 EXPERIMENTS
We consider a set of first-person 3D mazes from the DeepMind Lab environment (Beattie et al., 2016) (see Fig. 1) that are visually rich, with additional observations available to the agent such as inertial
information and local depth information.3 The action space is discrete, yet allows finegrained control, comprising 8 actions: the agent can rotate in small increments, accelerate forward or backward or sideways, or induce rotational acceleration while moving. Reward is achieved in these environments by reaching a goal from a random start location and orientation. If the goal is reached, the agent is respawned to a new start location and must return to the goal. The episode terminates when a fixed amount of time expires, affording the agent enough time to find the goal several times. There are sparse ‘fruit’ rewards which serve to encourage exploration. Apples are worth 1 point, strawberries 2 points and goals are 10 points. Videos of the agent solving the maze are linked in Appendix A.
In the static variant of the maze, the goal and fruit locations are fixed and only the agent’s start location changes. In the dynamic (Random Goal) variant, the goal and fruits are randomly placed on every episode. Within an episode, the goal and apple locations stay fixed until the episode ends. This encourages an explore-exploit strategy, where the agent should initially explore the maze, then retain the goal location and quickly refind it after each respawn. For both variants (static and random goal) we consider a small and large map. The small mazes are 5× 10 and episodes last for 3600 timesteps, and the large mazes are 9× 15 with 10800 steps (see Figure 1). The RGB observation is 84× 84. The I-Maze environment (see Figure 1, right) is inspired by the classic T-maze used to investigate navigation in rodents (Olton et al., 1979): the layout remains fixed throughout, the agent spawns in the central corridor where there are apple rewards and has to locate the goal which is placed in the alcove of one of the four arms. Because the goal is hidden in the alcove, the optimal agent behaviour must rely on memory of the goal location in order to return to the goal using the most direct route. Goal location is constant within an episode but varies randomly across episodes.
The different agent architectures described in Section 2 are evaluated by training on the five mazes. Figure 3 shows learning curves (averaged over the 5 top performing agents). The agents are a feedforward model (FF A3C), a recurrent model (LSTM A3C), the stacked LSTM version with velocity, previous action and reward as input (Nav A3C), and Nav A3C with depth prediction from the convolution layer (Nav A3C+D1), Nav A3C with depth prediction from the last LSTM layer (Nav A3C+D2), Nav A3C with loop closure prediction (Nav A3C+L) as well as the Nav A3C with
3The environments used in this paper are publicly available at https://github.com/deepmind/lab.
all auxiliary losses considered together (Nav A3C+D1D2L). In each case we ran 64 experiments with randomly sampled hyper-parameters (for ranges and details please see the appendix). The mean over the top 5 runs as well as the top 5 curves are plotted. Expert human scores, established by a professional game player, are compared to these results. The Nav A3C+D2 agents reach human-level performance on Static 1 and 2, and attain about 91% and 59% of human scores on Random Goal 1 and 2.
In Mnih et al. (2015) reward clipping is used to stabilize learning, technique which we employed in this work as well. Unfortunately, for these particular tasks, this yields slightly suboptimal policies because the agent does not distinguish apples (1 point) from goals (10 points). Removing the reward clipping results in unstable behaviour for the base A3C agent (see Appendix C). However it seems that the auxiliary signal from depth prediction mediates this problem to some extent, resulting in stable learning dynamics (e.g. Figure 3f, Nav A3C+D1 vs Nav A3C*+D1). We clearly indicate whether reward clipping is used by adding an asterisk to the agent name.
Figure 3f also explores the difference between the two formulations of depth prediction, as a regression task or a classification task. We can see that the regression agent (Nav A3C*+D1[MSE]) performs worse than one that does classification (Nav A3C*+D1). This result extends to other maps, and we therefore only use the classification formulation in all our other results4. Also we see that predicting depth from the last LSTM layer (hence providing structure to the recurrent layer, not just the convolutional ones) performs better.
We note some particular results from these learning curves. In Figure 3 (a and b), consider the feedforward A3C model (red curve) versus the LSTM version (pink curve). Even though navigation seems to intrinsically require memory, as single observations could often be ambiguous, the feedforward model achieves competitive performance on static mazes. This suggest that there might be good strategies that do not involve temporal memory and give good results, namely a reactive policy held by the weights of the encoder, or learning a wall-following strategy. This motivates the dynamic environments that encourage the use of memory and more general navigation strategies.
Figure 3 also shows the advantage of adding velocity, reward and action as an input, as well as the impact of using a two layer LSTM (orange curve vs red and pink). Though this agent (Nav A3C) is better than the simple architectures, it is still relatively slow to train on all of the mazes. We believe that this is mainly due to the slower, data inefficient learning that is generally seen in pure RL approaches. Supporting this we see that adding the auxiliary prediction targets of depth and loop closure (Nav A3C+D1D2L, black curve) speeds up learning dramatically on most of the mazes (see Table 1: AUC metric). It has the strongest effect on the static mazes because of the accelerated learning, but also gives a substantial and lasting performance increase on the random goal mazes.
Although we place more value on the task performance than on the auxiliary losses, we report the results from the loop closure prediction task. Over 100 test episodes of 2250 steps each, within a large maze (random goal 2), the Nav A3C*+D1L agent demonstrated very successful loop detection, reaching an F-1 score of 0.83. A sample trajectory can be seen in Figure 4 (right).
4An exception is the Nav A3C*+D1L agent on the I-maze (Figure 3c), which uses depth regression and reward clipping. While it does worse, we include it because some analysis is based on this agent.
5 ANALYSIS
5.1 POSITION DECODING
In order to evaluate the internal representation of location within the agent (either in the hidden units ht of the last LSTM, or, in the case of the FF A3C agent, in the features ft on the last layer of the conv-net), we train a position decoder that takes that representation as input, consisting of a linear classifier with multinomial probability distribution over the discretized maze locations. Small mazes (5× 10) have 50 locations, large mazes (9× 15) have 135 locations, and the I-maze has 77 locations. Note that we do not backpropagate the gradients from the position decoder through the rest of the network. The position decoder can only see the representation exposed by the model, not change it.
An example of position decoding by the Nav A3C+D2 agent is shown in Figure 6, where the initial uncertainty in position is improved to near perfect position prediction as more observations are acquired by the agent. We observe that position entropy spikes after a respawn, then decreases once the agent acquires certainty about its location. Additionally, videos of the agent’s position decoding are linked in Appendix A. In these complex mazes, where localization is important for the purpose of reaching the goal, it seems that position accuracy and final score are correlated, as shown in Table 1. A pure feed-forward architecture still achieves 64.3% accuracy in a static maze with static goal, suggesting that the encoder memorizes the position in the weights and that this small maze is solvable by all the agents, with sufficient training time. In Random Goal 1, it is Nav A3C+D2 that achieves the best position decoding performance (85.5% accuracy), whereas the FF A3C and the LSTM A3C architectures are at approximately 50%.
In the I-maze, the opposite branches of the maze are nearly identical, with the exception of very sparse visual cues. We observe that once the goal is first found, the Nav A3C*+D1L agent is capable of directly returning to the correct branch in order to achieve the maximal score. However, the linear position decoder for this agent is only 68.5% accurate, whereas it is 87.8% in the plain LSTM A3C agent. We hypothesize that the symmetry of the I-maze will induce a symmetric policy that need not be sensitive to the exact position of the agent (see analysis below).
A desired property of navigation agents in our Random Goal tasks is to be able to first find the goal, and reliably return to the goal via an efficient route after subsequent re-spawns. The latency column in Table 1 shows that the Nav A3C+D2 agents achieve the lowest latency to goal once the goal has been discovered (the first number shows the time in seconds to find the goal the first time, and the second number is the average time for subsequent finds). Figure 5 shows clearly how the agent finds the goal, and directly returns to that goal for the rest of the episode. For Random Goal 2, none of the agents achieve lower latency after initial goal acquisition; this is presumably due to the larger, more challenging environment.
5.2 STACKED LSTM GOAL ANALYSIS
Figure 7(a) shows shows the trajectories traversed by an agent for each of the four goal locations. After an initial exploratory phase to find the goal, the agent consistently returns to the goal location. We visualize the agent’s policy by applying tSNE dimension reduction (Maaten & Hinton, 2008) to the cell activations at each step of the agent for each of the four goal locations. Whilst clusters corresponding to each of the four goal locations are clearly distinct in the LSTM A3C agent, there are 2 main clusters in the Nav A3C agent – with trajectories to diagonally opposite arms of the maze represented similarly. Given that the action sequence to opposite arms is equivalent (e.g. straight, turn left twice for top left and bottom right goal locations), this suggests that the Nav A3C policy-dictating LSTM maintains an efficient representation of 2 sub-policies (i.e. rather than 4 independent policies) – with critical information about the currently relevant goal provided by the additional LSTM.
5.3 INVESTIGATING DIFFERENT COMBINATIONS OF AUXILIARY TASKS
Our results suggest that depth prediction from the policy LSTM yields optimal results. However, several other auxiliary tasks have been concurrently introduced in (Jaderberg et al., 2017), and thus we provide a comparison of reward prediction against depth prediction. Following that paper, we implemented two additional agent architectures, one performing reward prediction from the convnet using a replay buffer, called Nav A3C*+R, and one combining reward prediction from the convnet and depth prediction from the LSTM (Nav A3C+RD2). Table 2 suggests that reward prediction (Nav A3C*+R) improves upon the plain stacked LSTM architecture (Nav A3C*) but not as much as depth prediction from the policy LSTM (Nav A3C+D2). Combining reward prediction and depth prediction (Nav A3C+RD2) yields comparable results to depth prediction alone (Nav A3C+D2); normalised average AUC values are respectively 0.995 vs. 0.981. Future work will explore other auxiliary tasks.
6 CONCLUSION
We proposed a deep RL method, augmented with memory and auxiliary learning targets, for training agents to navigate within large and visually rich environments that include frequently changing start and goal locations. Our results and analysis highlight the utility of un/self-supervised auxiliary objectives, namely depth prediction and loop closure, in providing richer training signals that bootstrap learning and enhance data efficiency. Further, we examine the behavior of trained agents, their ability to localise, and their network activity dynamics, in order to analyse their navigational abilities.
Our approach of augmenting deep RL with auxiliary objectives allows end-end learning and may encourage the development of more general navigation strategies. Notably, our work with auxiliary losses is related to (Jaderberg et al., 2017) which independently looks at data efficiency when exploiting auxiliary losses. One difference between the two works is that our auxiliary losses are online (for the current frame) and do not rely on any form of replay. Also the explored losses are very different in nature. Finally our focus is on the navigation domain and understanding if navigation emerges as a bi-product of solving an RL problem, while Jaderberg et al. (2017) is concerned with data efficiency for any RL-task.
Whilst our best performing agents are relatively successful at navigation, their abilities would be stretched if larger demands were placed on rapid memory (e.g. in procedurally generated mazes), due to the limited capacity of the stacked LSTM in this regard. It will be important in the future to combine visually complex environments with architectures that make use of external memory (Graves et al., 2016; Weston et al., 2014; Olton et al., 1979) to enhance the navigational abilities of agents. Further, whilst this work has focused on investigating the benefits of auxiliary tasks for developing the ability to navigate through end-to-end deep reinforcement learning, it would be interesting for future work to compare these techniques with SLAM-based approaches.
ACKNOWLEDGEMENTS
We would like to thank Alexander Pritzel, Thomas Degris and Joseph Modayil for useful discussions, Charles Beattie, Julian Schrittwieser, Marcus Wainwright, and Stig Petersen for environment design and development, and Amir Sadik and Sarah York for expert human game testing.
Supplementary Material
A VIDEOS OF TRAINED NAVIGATION AGENTS
We show the behaviour of Nav A3C*+D1L agent in 5 videos, corresponding to the 5 navigation environments: I-maze5, (small) static maze6, (large) static maze7, (small) random goal maze8 and (large) random goal maze9. Each video shows a high-resolution video (the actual inputs to the agent are down-sampled to 84×84 RGB images), the value function over time (with fruit reward and goal acquisitions), the layout of the mazes with consecutive trajectories of the agent marked in different colours and the output of the trained position decoder, overlayed on top of the maze layout.
B NETWORK ARCHITECTURE AND TRAINING
B.1 THE ONLINE MULTI-LEARNER ALGORITHM FOR MULTI-TASK LEARNING
We introduce a class of neural network-based agents that have modular structures and that are trained on multiple tasks, with inputs coming from different modalities (vision, depth, past rewards and past actions). Implementing our agent architecture is simplified by its modular nature. Essentially, we construct multiple networks, one per task, using shared building blocks, and optimise these networks jointly. Some modules, such as the conv-net used for perceiving visual inputs, or the LSTMs used for learning the navigation policy, are shared among multiple tasks, while other modules, such as depth predictor gd or loop closure predictor gl, are task-specific. The navigation network that outputs the policy and the value function is trained using reinforcement learning, while the depth prediction and loop closure prediction networks are trained using self-supervised learning.
Within each thread of the asynchronous training environment, the agent plays on its own episode of the game environment, and therefore sees observation and reward pairs {(st, rt)} and takes actions that are different from those experienced by agents from the other, parallel threads. Within a thread, the multiple tasks (navigation, depth and loop closure prediction) can be trained at their own schedule, and they add gradients to the shared parameter vector as they arrive. Within each thread, we use a flag-based system to subordinate gradient updates to the A3C reinforcement learning procedure.
B.2 NETWORK AND TRAINING DETAILS
For all the experiments we use an encoder model with 2 convolutional layers followed by a fully connected layer, or recurrent layer(s), from which we predict the policy and value function. The architecture is similar to the one in (Mnih et al., 2016). The convolutional layers are as follows. The first convolutional layer has a kernel of size 8x8 and a stride of 4x4, and 16 feature maps. The second layer has a kernel of size 4x4 and a stride of 2x2, and 32 feature maps. The fully connected layer, in the FF A3C architecture in Figure 2a has 256 hidden units (and outputs visual features ft). The LSTM in the LSTM A3C architecture has 256 hidden units (and outputs LSTM hidden activations ht). The LSTMs in Figure 2c and 2d are fed extra inputs (past reward rt−1, previous action at expressed as a one-hot vector of dimension 8 and agent-relative lateral and rotational velocity vt encoded by a 6-dimensional vector), which are all concatenated to vector ft. The Nav A3C architectures (Figure 2c,d) have a first LSTM with 64 or 128 hiddens and a second LSTM with 256 hiddens. The depth predictor modules gd, g′d and the loop closure detection module gl are all single-layer MLPs with 128 hidden units. The depth MLPs are followed by 64 independent 8-dimensional softmax outputs (one per depth pixel). The loop closure MLP is followed by a 2-dimensional softmax output. We illustrate on Figure 8 the architecture of the Nav A3C+D+L+Dr agent.
Depth is taken as the Z-buffer from the Labyrinth environment (with values between 0 and 255), divided by 255 and taken to power 10 to spread the values in interval [0, 1]. We empirically decided to use the following quantization: {0, 0.05, 0.175, 0.3, 0.425, 0.55, 0.675, 0.8, 1} to ensure a uniform
5Video of the Nav A3C*+D1L agent on the I-maze: https://youtu.be/PS4iJ7Hk_BU 6Video of the Nav A3C*+D1L agent on static maze 1: https://youtu.be/-HsjQoIou_c 7Video of the Nav A3C*+D1L agent on static maze 2: https://youtu.be/kH1AvRAYkbI 8Video of the Nav A3C*+D1L agent on random goal maze 1: https://youtu.be/5IBT2UADJY0 9Video of the Nav A3C*+D1L agent on random goal maze 2: https://youtu.be/e10mXgBG9yo
binning across 8 classes. The previous version of the agent had a single depth prediction MLP gd for regressing 8× 16 = 128 depth pixels from the convnet outputs ft. The parameters of each of the modules point to a subset of a common vector of parameters. We optimise these parameters using an asynchronous version of RMSProp (Tieleman & Hinton, 2012). (Nair et al., 2015) was a recent example of asynchronous and parallel gradient updates in deep reinforcement learning; in our case, we focus on the specific Asynchronous Advantage Actor Critic (A3C) reinforcement learning procedure in (Mnih et al., 2016).
Learning follows closely the paradigm described in (Mnih et al., 2016). We use 16 workers and the same RMSProp algorithm without momentum or centering of the variance. Gradients are computed over non-overlaping chunks of the episode. The score for each point of a training curve is the average over all the episodes the model gets to finish in 5e4 environment steps.
The whole experiments are run for a maximum of 1e8 environment step. The agent has an action repeat of 4 as in (Mnih et al., 2016), which means that for 4 consecutive steps the agent will use the same action picked at the beginning of the series. For this reason through out the paper we actually report results in terms of agent perceived steps rather than environment steps. That is, the maximal number of agent perceived step that we do for any particular run is 2.5e7.
In our grid we sample hyper-parameters from categorical distributions:
• Learning rate was sampled from [10−4, 5 · 10−4]. • Strength of the entropy regularization from [10−4, 10−3]. • Rewards were not scaled and not clipped in the new set of experiments. In our previous set
of experiments, rewards were scaled by a factor from {0.3, 0.5} and clipped to 1 prior to back-propagation in the Advantage Actor-Critic algorithm. • Gradients are computed over non-overlaping chunks of 50 or 75 steps of the episode. In our
previous set of experiments, we used chunks of 100 steps.
The auxiliary tasks, when used, have hyperparameters sampled from:
• Coefficient βd of the depth prediction loss from convnet features Ld sampled from {3.33, 10, 33}. • Coefficient β′d of the depth prediction loss from LSTM hiddens Ld′ sampled from {1, 3.33, 10}. • Coefficient βl of the loop closure prediction loss Ll sampled from {1, 3.33, 10}.
Loop closure uses the following thresholds: maximum distance for position similarity η1 = 1 square and minimum distance for removing trivial loop-closures η2 = 2 squares.
C ADDITIONAL RESULTS
C.1 REWARD CLIPPING
Figure 9 shows additional learning curves. In particular in the left plot we show that the baselines (A3C FF and A3C LSTM) as well as Nav A3C agent without auxiliary losses, perform worse without reward clipping than with reward clipping. It seems that removing reward clipping makes learning unstable in absence of auxiliary tasks. For this particular reason we chose to show the baselines with reward clipping in our main results.
C.2 DEPTH PREDICTION AS REGRESSION OR CLASSIFICATION TASKS
The right subplot of Figure 9 compares having depth as an input versus as a target. Note that using RGBD inputs to the Nav A3C agent performs even worse than predicting depth as a regression task, and in general is worse than predicting depth as a classification task.
C.3 NON-NAVIGATION TASKS IN 3D MAZE ENVIRONMENTS
We have evaluated the behaviour of the agents introduced in this paper, as well as agents with reward prediction, introduced in (Jaderberg et al., 2017) (Nav A3C*+R) and with a combination of reward prediction from the convnet and depth prediction from the policy LSTM (Nav A3C+RD2), on different 3D maze environments with non-navigation specific tasks. In the first environment, Seek-Avoid Arena, there are apples (yielding 1 point) and lemons (yielding -1 point) disposed in an arena, and the agents needs to pick all the apples before respawning; episodes last 20 seconds. The second environment, Stairway to Melon, is a thin square corridor; in one direction, there is a lemon followed by a stairway to a melon (10 points, resets the level) and in the other direction are 7 apples and a dead end, with the melon visible but not reachable. The agent spawns between the lemon and the apples with a random orientation. Both environments have been released in DeepMind Lab (Beattie et al., 2016). These environments do not require navigation skills such as shortest path planning, but a simple reward identification (lemon vs. apple or melon) and persistent exploration. As Figure 10 shows, there is no major difference between auxiliary tasks related to depth prediction or reward prediction. Depth prediction boosts the performance of the agent beyond that of the stacked LSTM architecture, hinting at a more general applicability of depth prediction beyond navigation tasks.
C.4 SENSITIVITY TOWARDS HYPER-PARAMETER SAMPLING
For each of the experiments in this paper, 64 replicas were run with hyperparameters (learning rate, entropy cost) sampled from the same interval. Figure 11 shows that the Nav architectures with
auxiliary tasks achieve higher results for a comparatively larger number of replicas, hinting at the fact that auxiliary tasks make learning more robust to the choice of hyperparameters.
C.5 ASYMPTOTIC PERFORMANCE OF THE AGENTS
Finally, we compared the asymptotic performance of the agents, both in terms of navigation (final rewards obtained at the end of the episode) and in terms of their representation in the policy LSTM. Rather than visualising the convolutional filters, we quantify the change in representation, with and
without auxiliary task, in terms of position decoding, following the approach explained in Section 5.1. Specifically, we compare the baseline agent (LSTM A3C*) to a navigation agent with one auxiliary task (depth prediction), that gets about twice as many gradient updates for the same number of frames seen in the environment: once for the RL task and once for the auxiliary depth prediction task. As Table 3 shows, the performance of the baseline agent as well as the position decoding accuracy do significantly increase after twice the number of training steps (going from 57 points to 90 points, and from 33.4% to 66.5%, but do not reach the performance and position decoding accuracy of the Nav A3C+D2 agent after half the number of training frames. For this reason, we believe that the auxiliary task do more than simply accelerate training.

This relatively novel work proposes to augment current RL models by adding self-supervised tasks encouraging better internal representations.
The proposed tasks are depth prediction and loop closure detection. While these tasks assume a 3D environment as well some position information, such priors are well suited to a large variety of tasks pertaining to navigation and robotics.
Extensive experiments suggest to incorporating such auxiliary tasks increase performance and to a large extent learning speed.
Additional analysis of value functions and internal representations suggest that some structure is being discovered by the model, which would not be without the auxiliary tasks.
While specific to 3D-environment tasks, this work provides additional proof that using input data in addition to sparse external reward signals helps to boost learning speed as well as learning better internal representation. It is original, clearly presented, and strongly supported by empirical evidence.
One small downside of the experimental method (or maybe just the results shown) is that by picking top-5 runs, it is hard to judge whether such a model is better suited to the particular hyperparameter range that was chosen, or is simply more robust to these hyperparameter settings. Maybe an analysis of performance as a function of hyperparameters would help confirm the superiority of the approach to the baselines. My own suspicion is that adding auxiliary tasks would make the model robust to bad hyperparameters.
Another downside is that the authors dismiss navigation literature as "not RL". I sympathize with the limit on the number of things that can fit in a paper, but some experimental comparison with such literature may have proven insightful, if just in measuring the quality of the learned representations.


This paper presents a method of learning distributed representation for multi-field categorical data, which is a common data format with various applications such as recommender systems, social link prediction, and computational advertising. The success of non-linear models, e.g., factorisation machines, boosted trees, has proved the potential of exploring the interactions among inter-field categories. Inspired by Word2Vec, the distributed representation for natural language, we propose Cat2Vec (categories to vectors) model. In Cat2Vec, a low-dimensional continuous vector is automatically learned for each category in each field. The interactions among inter-field categories are further explored by different neural gates and the most informative ones are selected by pooling layers. In our experiments, with the exploration of the interactions between pairwise categories over layers, the model attains great improvement over state-of-the-art models in a supervised learning task, e.g., click prediction, while capturing the most significant interactions from the data.
This paper presents a method of learning distributed representation for multi-field categorical data, which is a common data format with various applications such as recommender systems, social link prediction, and computational advertising. The success of non-linear models, e.g., factorisation machines, boosted trees, has proved the potential of exploring the interactions among inter-field categories. Inspired by Word2Vec, the distributed representation for natural language, we propose Cat2Vec (categories to vectors) model. In Cat2Vec, a low-dimensional continuous vector is automatically learned for each category in each field. The interactions among inter-field categories are further explored by different neural gates and the most informative ones are selected by pooling layers. In our experiments, with the exploration of the interactions between pairwise categories over layers, the model attains great improvement over state-of-the-art models in a supervised learning task, e.g., click prediction, while capturing the most significant interactions from the data.
1 INTRODUCTION
There are different abstraction levels within data. For the low-abstraction continuous sensory data (such as images, videos, and audio) directly acquired from the physical world, quite often, the strong correlations (local patterns) are known a priori within the data. As such, one can directly embed the prior knowledge into a learning model such as neural networks to automatically distil such patterns and perform predictions (Krizhevsky et al., 2012; Graves et al., 2013). However, on the other hand, for high-abstraction data from our social and business activities, such as natural language and transnational log data, the data is commonly discrete and contains atomic symbols, whose meaning and correlation are unknown a priori. A typical solution is to employ embedding techniques (Bengio et al., 2003; Mikolov et al., 2013) to map the discrete tokens into a (low-dimensional) continuous space and further build neural networks to learn the latent patterns.
Multi-field categorical data is a type of high-abstraction data where the categories in each field are heterogeneous with those in other fields. Such a type of data is very widely used in data mining tasks based on transaction logs from many social or commercial applications, such as recommender systems, social link prediction, and computational advertising. Table 1 gives an example of multifield categorical data in user behaviour targeting where we observe user browsing patterns, and given those multi-field categorical features, a common task is to predict their actions such as clicks and conversions (Zhang et al., 2014; Liao et al., 2014; Yuan et al., 2013).
As there is no explicit dependency among these inter-field categories, two solutions are mainly used for building machine learning models that extract the local patterns of the data and make good predictions. The first solution is to create combining features across fields, such as CITY:SHANGHAI&WEEKDAY:FRIDAY (Chapelle et al., 2015). Such feature engineering is expensive on human efforts and feature/parameter space. The second solution is to build functions (Rendle, 2012) or neural networks based on the feature embeddings (Zhang et al., 2016). These solutions are of low efficiency because of the brute-force feature engineering or aimless embedding interactions.
In this paper, we propose an unsupervised pairwise interaction model to learning the distributed representation of multi-field categorical data. The interactions among inter-field categories are explored by different neural gates and the informative ones are selected by K-max pooling layers. Note that the K-max pooling process acts like the classic Apriori algorithm in frequent itemset mining and association rule learning (Agrawal et al., 1994). Repeating this pairwise interaction with K-max pooling, our Cat2Vec model automatically extracts salient feature interactions and further explores higher-order interactions.
To train the pairwise interaction Cat2Vec model effectively, we present a discriminant training method to estimate the category vectors. Furthermore, with the exploration of the pairwise and high-order category interactions, our Cat2Vec model attains great performance improvement over state-of-the-art models in supervised learning tasks, such as user response rate prediction, while successfully captures the most significant interactions in unsupervised learning tasks.
2 RELATED WORK AND PRELIMINARIES
In this section, we outline the major data representation methods that are used for representing the discrete categorical data. These methods serves as the preliminaries of our Cat2Vec model.
2.1 ONE-HOT REPRESENTATION
It is common to use one-hot representation for discrete data in natural language processing or computational advertising tasks. For the first data sample as an example, the data is vectorised by one-hot encoding as
[0, 1]︸︷︷︸ GENDER:MALE , [0, 1, 0, 0, 0, 0, 0]︸ ︷︷ ︸ WEEKDAY:TUESDAY , [0, . . . , 0, 1, 0, . . . , 0]351︸ ︷︷ ︸ CITY:BEIJING , [1, 0, 0, 0, 0, 0]︸ ︷︷ ︸ BROWSER:CHROME . (1)
With each category as a dimension, one-hot representation preserves full information of the original data. Two main problems of one-hot representation are that (i) it may suffer from the curse of dimensionality, especially in deep learning-related applications; (ii) it cannot capture the similarity of each word/category pair, and we cannot even find any relationships among the synonyms or categories in the same field.
2.2 DISTRIBUTED REPRESENTATION
Distributed representation is first proposed by Hinton (1986). The basic idea of distributed representation is training the model to map each word into a d-dimension vector (generally, d is the hyperparameter of the model, and d is far smaller than whole vocabulary size N of words/categories), and the semantic similarity between the words/categories can be measured through the distance (such as cosine similarity, Euclidean distance) of their corresponding low dimension vectors. The Word2Vec (Mikolov et al., 2013) is one of the most common methods to train the distributed word vector representation. Compared with text, with the local patterns among the neighbour words, multi-field categorical data has no explicit order relationships among inter-field categories. Also, the text vocabulary size (105) is often much smaller than the category size (106 ∼ 108), making our problem more difficult. Another difference between our Cat2Vec and Word2Vec is that Cat2Vec does not take the order into account or use any sliding window for context; in other words, we take all categories in the same training sample as the neighbour of a category.
3 PAIRWISE INTERACTION CAT2VEC MODEL
In this section, we introduce a pairwise interaction Cat2Vec model and its training method in detail. We design neural gates in the model to capture the interactions between each pair of categories, followed by the K-max pooling layers to select the most important interactions. We then repeat this processes to explore higher level interactions. Figure 1 illustrates the overview of the proposed architecture.
3.1 INTERACTION AND POOLING LAYERS
Interaction Layer. To evaluate the interaction between each pair of categories, we use a gate to obtain the interaction result. Mathematically, a gate is a function f : Rd × Rd → Rd that takes any pair of category vectors ci and cj in the same sample c as input, and outputs interaction result vector c′i,j = f(ci, cj). The interaction output vector c ′ i,j acts as a certain combining feature of ci and cj . Note that c′i,j keeps the same dimension as the category embedding vectors like ci and cj so that it can be further used to interact with other categories.
We provide several options of gate f as:
f sum(ci, cj) = ci + cj , (2)
fmul(ci, cj) = ci cj , (3)
where is the element-wise multiplication operator. We can also can employ more complex gates, such as the highway gate (Srivastava et al., 2015), which is formulated as
f highway(ci, cj) =τ g(WH(ci + cj) + bH) + (1− τ ) (ci + cj), (4)
where g is a nonlinear function and τ = σ(Wτ (ci + cj) + bτ ) represents a “transform gate”.
Then we apply the gate f on each pair of category vectors ci,cj :
c′ = [c′1,2, c ′ 1,3, · · · , c′1,n, · · · , c′n−2,n−1, c′n−1,n]. (5)
After the interaction, an activation function will be applied to implement the non-liner transformation.
K-Max Pooling Layer. We next describe a pooling operation that is a generalisation of the max pooling based on the norm length of interaction outputs of each pair of category vectors. We keep
the K maximum interaction output vectors c′i,j according to their norm length, where K is the number of the original categories of the training sample. It would keep the max-pooling result c′kmax = [c ′ 1, c ′ 2, · · · , c′K ] having the same size with the original embedding matrix c and c′K is the embedding vector in c′ in Eq. (5) that has top-K normal length.
Before producing an output for the interaction results, the interaction andK-max pooling operations will be repeated for several times in order to capture high-level interactions among the different field category vectors. After that, we output a prediction from the final interaction vector representation by a fully connected layer. Note that the above network structure can be used to build an autoencoder to conduct unsupervised learning (Vincent et al., 2008). We leave this for future work, while staying with the label output network for both supervised (containing both negative and positive examples) and unsupervised (only containing positive examples where negative examples are generated randomly) learning tasks.
An interesting discussion is to compare our Cat2Vec model with association rules mining, which aims to identify the most frequently appeared joint category instances (items), with or without a condition. Apriori (Agrawal et al., 1994) is a popular algorithm for association rules mining by exploiting dependencies between candidate frequent itemsets of length K and frequent itemsets of length K − 1. In our pairwise interaction Cat2Vec model, with neural networks, we provide an alternative way of generating such high-order interactions (thus itemsets) among category instances. Via the pooling operation, our model can also find the most frequent category set automatically, which will be demonstrated and tested from our experiments in the following Sections 4 and 5.
3.2 DISCRIMINANT CAT2VEC - TRAINING METHOD OF PAIRWISE INTERACTION CAT2VEC
To train the pairwise interaction Cat2Vec model, we design a training scheme called discriminant Cat2Vec, which would train the model in a supervised way for unsupervised learning of the data.
In the discriminant Cat2Vec, we feed the Sample Encoding Module showed in Figure 1 with a true or fake sample, the encoded sample vector will be followed by an MLP to predict the probability p of a true sample. As such, the generation of a fake sample would influence the learned category vector. In this paper, we generate a fake sample following this way: first, randomly choose a sample from the training set; second, randomly choose several categories in this sample and replace them with randomly chosen categories that belong to the same field. For example, we get a user behaviour instance x = [WEEKDAY:WEDNESDAY, IP:1.1.*.*, GENDER:MALE, CITY:BEIJING], and we randomly choose the category CITY:BEIJING and replace it with CITY:SHANGHAI, then we build a fake sample x′ = [WEEKDAY:WEDNESDAY, IP:1.1.*.*, GENDER:MALE, CITY:SHANGHAI]. The discriminant network is then trained to predict whether the new sample should be a true sample. The loss function of discriminant network is average cross entropy, which would maximise the likelihood of correct prediction:
L = 1
M M∑ i=1 −yi log(pi)− (1− yi) log(1− pi), (6)
where M is the number of training samples. The i-th sample is labelled with yi ∈ {1, 0}, which means true or fake sample, and pi is the predicted probability that the given training sample is true.
4 SYNTHETIC DATA EXPERIMENTS
To explore and add our understanding of the pairwise interaction Cat2Vec model, we conduct a simulation test with synthetic data. In particular, we are interested in understanding how the learned vectors would be able to capture and leverage the most significant patterns embedded in the data.
4.1 SYNTHETIC DATASET AND EVALUATION METRICS
To simulate the real-world multi-field categorical data, we use multivariate normal sampling to generate the true data distribution for the following experiments 1 . Suppose the data has 4 fields {A,B,C,D}, each field contains 10 categories, and a sample can be represented as x = (ai, bi, ci, di). We then randomly generate the means and covariance matrix for 4-dimensional truncated multivariate normal sampling with two-sided truncation. This sampling method can generate 4 float numbers between 0 and 10. We can convert the float numbers to integer which can represent the categories in 4 fields. In such a way, we can generate the data with specific joint distribution, which means certain categorical pair or 3-tuple like p(a4, b4) or p(a3, c5, d6) may have a higher joint distribution probability. Recall that in our pairwise interaction Cat2Vec model, we have a Kmax pooling layer, which will select the most popular category pairs in the dataset. Repeating the pairwise interaction layers and K-max pooling layers, we can also explore a high order categorical 3-tuple or 4-tuple etc. Therefore, our task here is to evaluate if our model would be able to capture these frequently occurred patterns from a given dataset; in other words, to test if our model would be able to keep the category pairs with the highest joint distribution probabilities in the K-max pooling results. This processes is in line with association rule mining (Agrawal et al., 1994), exploring the frequent categorical n-tuple from frequent categorical (n− 1)-tuple. We generate the positive data according to the above truncated multivariate normal sampling and choose uniform sampling to generate the fake (negative) data. We then apply discriminant Cat2Vec to train the model. Because we know the true distribution of the generated real data, the most frequent category pairs/triples are known. We use precision and Spearman’s rank correlation coefficient to evaluate the results of 1st/2nd K-max pooling layer (category pairs/triples pooling results), to see if the model can learn the true joint distribution in the real data. The details of the evaluation metrics are described in the following section.
To evaluate how our network structure and K-max pooling help identify the significant n-tuples, we feed 1000 samples to the trained model and record the 1st and 2nd K-max pooling layers’ results. Then we count the frequency of the category pairs/3-tuples in the real samples, and select top 20 ranked category pairs/3-tuples as target. Then we count the frequency of max-pooled category pairs/triples in the results and compare the top 20 frequent category pairs/3-tuples in the results to calculate precision and Spearman’s rank correlation coefficient. Precision measures the fraction of category pairs/triples in the results that are also in the target. The Spearman’s rank correlation coefficient measures the correlation between two ranked lists.
1the experiment code is available: https://github.com/wenying45/cat2vec
4.2 RESULT AND DISCUSSION
Figure 3 summarises the results of the precision and the rank correlation on synthetic data. We can see that our model can easily find over 80% of the category pairs with high joint distribution probabilities under the best parameter settings. From the rank correlation, our model can make the ranking correlation over 0.6 of category pairs which means the category pairs with higher joint distribution probability would be more possible to appear in the K-max pooling result. As for the category triples case, the precision and rank correlation become lower than the category pairs’, because finding 3-order combination is harder and relies on the accuracy from the 2-order. We also vary the dropout rate against those measures. It shows that dropout tends to help improving the accuracy of captured patterns. This can be explained by considering the fact that dropout brings randomness into the selection and allows exploration. But the best dropout rate seems rather arbitrary and highly dependent on the other parameter settings.
5 REAL-WORLD DATA EXPERIMENTS
In this section, we continue our experiment using a real-world advertising dataset for click-through rate estimation2. The iPinYou dataset (Liao et al., 2014) is a public real-world display ad dataset with each ad display information and corresponding user click feedback (Zhang et al., 2014). This dataset contains around 19.5M ad display instances with 14.8k positive user feedback (click). Each instance has 23 fields, and we choose 18 fields of them which have categories with occurrence larger than 10.3
5.1 UNSUPERVISED LEARNING EXPERIMENT
We continue our study on the model’s ability of capturing the most significant patterns as we described in Section 3.2. Because the iPinYou dataset contains the unencrypted fields and categories, e.g. city, region and tag, so we choose the iPinYou dataset which has been introduced above as real positive) data. As for the fake (negative) data, we randomly choose a sample in the iPinYou dataset and randomly replace some categories with other categories in the same field to generate the fake data, similar to what we have introduced in Section 3.2. We also set up two baseline models to compare the model accuracy performance: (i) DNN Concat model, which concatenates category embedding vectors to make prediction, and (ii) DNN Sum model, which sums up the category embedding vectors to make the prediction.
We have tried different parameter settings and the performance is measured by the accuracy of our model to predict real samples. We also calculate the rank correlation coefficient and the precision to evaluate our model the same as we described in Section 4.1.
5.1.1 RESULT AND DISCUSSION
From Table 2, we see that on the iPinYou dataset, our pairwise interaction models can achieve the accuracy of 85% which is about 1.7% improvement comparing with the simple DNN models. Even the worst case in our model is better than the DNN models’ best case. It means our model can find the extra information during the interactions and the K-max pooling processes. In addition, the model with interaction times as 3 usually yields better performance than that with interaction times as 2, which may be due to the fact that the more interaction times capture higher-order interactions and help make more accurate predictions. But the model with different gate types does not lead to significant difference.
We next use the same evaluation metrics that described in Section 4.1 to test the ability of capturing data patterns. We find that in the real-world dataset, our model is still able to keep high precision and rank correlation and can achieve even better performance. The precision and rank correlation on category pairs are over 0.8 which is a 30% improvement comparing to the performance on synthetic
2http://data.computational-advertising.org/ 3The selected fields are WEEKDAY, HOUR, USER AGENT, IP, REGION, CITY, AD EXCHANGE, DOMAIN, URL, AD SLOT ID, AD SLOT WIDTH, AD SLOT HEIGHT, AD SLOT VISIBILITY, AD SLOT FORMAT, AD SLOT FLOOR PRICE, CREATIVE ID, KEY PAGE URL, AND USER TAGS.
dataset. For the category triples case, we also have similar performance compared with the synthetic dataset.
5.2 CLICK-THROUGH RATE PREDICTION EXPERIMENT
We now move to the evaluation on a supervised learning task. We consider click-through rate (CTR) prediction, which is important for many personalised Web services such as E-commerce, social recommendation and computational advertising (Yuan et al., 2013). The most widely used CTR estimation model is the logistic regression based on one-hot data representation. Many deep learning models have been further investigated in CTR prediction. Zhang et al. (2016) proposed FactorisationMachine Supported Neural Networks (FNN) models for user response prediction. Convolutional Click Prediction Model (CCPM) (Liu et al., 2015) has been used in CTR prediction and gain some improvement on this task. To our knowledge, all of above previous work focuses on directly improving the prediction performance in supervised learning tasks and none of them investigates the learned representation of multi-field categorical data or how to learn the better representation.
In order to investigate our pairwise interaction model in the CTR task, we use the pairwise interaction sample encoding module to encode a training sample concatenated with the embedding vectors, which is followed by an MLP (multi-layer perceptron) to predict click-through probability. We choose following models as strong baselines:
• Logistic Regression (LR): LR is a widely used linear model (Richardson et al., 2007).
• Factorisation Machine (FM): Simply apply the factorisation machine on one-hot encoded sparse features of the training sample (Rendle, 2010).
• CCPM: CCPM (Liu et al., 2015) is a convolutional model for click prediction.
• FNN: A DNN model based on concatenated category vectors following with MLPs, being able to capture high-order latent patterns of multi-field categorical data (Zhang et al., 2016).
• Cat2Vec-FNN-1: This is our proposed architecture that only concatenates pairwise interaction output vectors among K-max pooling results to form the final vector representation and make prediction.
We use Area Under ROC Curve (AUC) as the evaluation metrics to measure the performance of a prediction. Also we conduct the grid search for each model to make sure that each model has achieved its best performance. Specifically, empirically optimal hyperparameters are set as: the category embedding size is 16, the SGD batch size is 64, the Nadam (Sutskever et al., 2013) is set as SGD optimiser with default settings, the gate type is MUL and the norm type for K-Max Pooling is L2 norm, and the activation function as tanh. Then the model followed by three fully connected layer with width [128, 32, 1]. We also try different interaction times and finally set it as two (3-tuple), suggesting that a high order of interactions helps improve the performance, but more than two would overfit the data and thus managed the performance,
5.2.1 RESULT AND DISCUSSION
Table 3 gives the results of our CTR experiment, compared with various baselines. We see that there is about 3% improvement over LR. The AUC performance of the proposed Discrimination Cat2Vec models also outperforms the FM/CCPM/FNN model, as our model would be able to take higher order information into consideration, which helps make better decision.
In our pairwise interaction model, we also test different hyperparameters and settings, and the result is given in Figure 5. First, we evaluate the performance over different dropout rates, and find that setting dropout as 0.1 would be the best, as shown in Figure 5. We also explore the impact of interaction. From the result, the model with 2 interaction times would have better generalisation on the test set. Finally, we compare three different activation functions (sigmoid, tanh, relu) and set identity mapping as the baseline. The result shows that “tanh” yields the best performance, which has the advantages of non-linear transformation between (−1, 1), and it may help gain more benefits on multi-field categorical data.
6 CONCLUSION
In this paper we have proposed a novel Cat2Vec model working on the multi-field categorical data. Different from the other models, Cat2Vec repetitively computes and selects inter-field category pairwise interactions to explore high-level interactions, which is analogous to the Apriori algorithm in association rule mining. Moreover, we present an efficient discriminant training method to estimate the category vectors. We also apply our pairwise interaction model on CTR prediction, of which we have observed a significant performance gain over several strong baselines. For future work, we plan to design more sophisticated gates to explore different interaction patterns among inter-field categories; also leveraging Cat2Vec in various data mining problems is of great interest to us.

The paper proposes a way to learn continuous features for input data which consists of multiple categorical data. The idea is to embed each category in a learnable low dimensional continuous space, explicitly compute the pair-wise interaction among different categories in a given input sample (which is achieved by either taking a component-wise dot product or component-wise addition), perform k-max pooling to select a subset of the most informative interactions, and repeat the process some number of times, until you get the final feature vector of the given input. This feature vector is then used as input to a classifier/regressor to accomplish the final task. The embeddings of the categories are learnt in the usual way. In the experiment section, the authors show on a synthetic dataset that their procedure is indeed able to select the relevant interactions in the data. On one real world dataset (iPinYou) the model seems to outperform a couple of simple baselines.
My major concern with this paper is that their's nothing new in it. The idea of embedding the categorical data having mixed categories has already been handled in the past literature, where essentially one learns a separate lookup table for each class of categories: an input is represented by concatenation of the embeddings from these lookup table, and a non-linear function (a deep network) is plugged on top to get the features of the input. The only rather marginal contribution is the explicit modeling of the interactions among categories in equations 2/3/4/5. Other than that there's nothing else in the paper.
Not only that, I feel that these interactions can (and should) automatically be learned by plugging in a deep convolutional network on top of the embeddings of the input. So I'm not sure how useful the contribution is.
The experimental section is rather weak. They authors test their method on a single real world data set against a couple of rather weak baselines. I would have much preferred for them to evaluate against numerous models proposed in the literature which handle similar problems, including wsabie.
While the authors argued in their response that wsabie was not suited for their problem, i strongly disagree with that claim. While the original wsabie paper showed experiments using images as inputs, their training methodology can easily be extended to other types of data sets, including categorical data. For instance, I conjecture that the model i proposed above (embed all the categorical inputs, concatenate the embeddings, plug a deep conv-net on top and train using some margin loss) will perform as well if not better than the hand coded interaction model proposed in this paper. Of course I could be wrong, but it would be far more convincing if their model was tested against such baselines.