AN ANALYSIS OF DEEP NEURAL NETWORK MODELS FOR PRACTICAL APPLICATIONS
Since the emergence of Deep Neural Networks (DNNs) as a prominent technique in the field of computer vision, the ImageNet classification challenge has played a major role in advancing the state-of-the-art. While accuracy figures have steadily increased, the resource utilisation of winning models has not been properly taken into account. In this work, we present a comprehensive analysis of important metrics in practical applications: accuracy, memory footprint, parameters, operations count, inference time and power consumption. Key findings are: (1) power consumption is independent of batch size and architecture; (2) accuracy and inference time are in a hyperbolic relationship; (3) energy constraint are an upper bound on the maximum achievable accuracy and model complexity; (4) the number of operations is a reliable estimate of the inference time. We believe our analysis provides a compelling set of information that helps design and engineer efficient DNNs.
1 INTRODUCTION
Since the breakthrough in 2012 ImageNet competition (Russakovsky et al., 2015) achieved by AlexNet (Krizhevsky et al., 2012) — the first entry that used a Deep Neural Network (DNN) — several other DNNs with increasing complexity have been submitted to the challenge in order to achieve better performance.
In the ImageNet classification challenge, the ultimate goal is to obtain the highest accuracy in a multi-class classification problem framework, regardless of the actual inference time. We believe that this has given rise to several problems. Firstly, it is now normal practice to run several trained instances of a given model over multiple similar instances of each validation image. This practice, also know as model averaging or ensemble of DNNs, dramatically increases the amount of computation required at inference time to achieve the published accuracy. Secondly, model selection is hindered by the fact that different submissions are evaluating their (ensemble of) models a different number of times on the validation images, and therefore the reported accuracy is biased on the specific sampling technique (and ensemble size). Thirdly, there is currently no incentive in speeding up inference time, which is a key element in practical applications of these models, and affects resource utilisation, power-consumption, and latency.
This article aims to compare state-of-the-art DNN architectures, submitted for the ImageNet challenge over the last 4 years, in terms of computational requirements and accuracy. We compare these architectures on multiple metrics related to resource utilisation in actual deployments: accuracy, memory footprint, parameters, operations count, inference time and power consumption. The purpose of this paper is to stress the importance of these figures, which are essential hard constraints for the optimisation of these networks in practical deployments and applications.
2 METHODS
In order to compare the quality of different models, we collected and analysed the accuracy values reported in the literature. We immediately found that different sampling techniques do not allow for a direct comparison of resource utilisation. For example, central-crop (top-5 validation) errors of a
single run of VGG-161 (Simonyan & Zisserman, 2014) and GoogLeNet (Szegedy et al., 2014) are 8.70% and 10.07% respectively, revealing that VGG-16 performs better than GoogLeNet. When models are run with 10-crop sampling,2 then the errors become 9.33% and 9.15% respectively, and therefore VGG-16 will perform worse than GoogLeNet, using a single central-crop. For this reason, we decided to base our analysis on re-evaluations of top-1 accuracies3 for all networks with a single central-crop sampling technique (Zagoruyko, 2016).
For inference time and memory usage measurements we have used Torch7 (Collobert et al., 2011) with cuDNN-v5 (Chetlur et al., 2014) and CUDA-v8 back-end. All experiments were conducted on a JetPack-2.3 NVIDIA Jetson TX1 board (nVIDIA): an embedded visual computing system with a 64-bit ARM R© A57 CPU, a 1 T-Flop/s 256-core NVIDIA Maxwell GPU and 4 GB LPDDR4 of shared RAM. We use this resource-limited device to better underline the differences between network architecture, but similar results can be obtained on most recent GPUs, such as the NVIDIA K40 or Titan X, to name a few. Operation counts were obtained using an open-source tool that we developed (Paszke, 2016). For measuring the power consumption, a Keysight 1146B Hall effect current probe has been used with a Keysight MSO-X 2024A 200MHz digital oscilloscope with a sampling period of 2 s and 50 kSa/s sample rate. The system was powered by a Keysight E3645A GPIB controlled DC power supply.
3 RESULTS
In this section we report our results and comparisons. We analysed the following DDNs: AlexNet (Krizhevsky et al., 2012), batch normalised AlexNet (Zagoruyko, 2016), batch normalised Network In Network (NIN) (Lin et al., 2013), ENet (Paszke et al., 2016) for ImageNet (Culurciello, 2016), GoogLeNet (Szegedy et al., 2014), VGG-16 and -19 (Simonyan & Zisserman, 2014), ResNet-18, -34, -50, -101 and -152 (He et al., 2015), Inception-v3 (Szegedy et al., 2015) and Inception-v4 (Szegedy et al., 2016) since they obtained the highest performance, in these four years, on the ImageNet (Russakovsky et al., 2015) challenge.
1 In the original paper this network is called VGG-D, which is the best performing network. Here we prefer to highlight the number of layer utilised, so we will call it VGG-16 in this publication.
2 From a given image multiple patches are extracted: four corners plus central crop and their horizontal mirrored twins.
3 Accuracy and error rate always sum to 100, therefore in this paper they are used interchangeably.
3.1 ACCURACY
Figure 1 shows one-crop accuracies of the most relevant entries submitted to the ImageNet challenge, from the AlexNet (Krizhevsky et al., 2012), on the far left, to the best performing Inception-v4 (Szegedy et al., 2016). The newest ResNet and Inception architectures surpass all other architectures by a significant margin of at least 7%.
Figure 2 provides a different, but more informative view of the accuracy values, because it also visualises computational cost and number of network’s parameters. The first thing that is very apparent is that VGG, even though it is widely used in many applications, is by far the most expensive architecture — both in terms of computational requirements and number of parameters. Its 16- and 19-layer implementations are in fact isolated from all other networks. The other architectures form a steep straight line, that seems to start to flatten with the latest incarnations of Inception and ResNet. This might suggest that models are reaching an inflection point on this data set. At this inflection point, the costs — in terms of complexity — start to outweigh gains in accuracy. We will later show that this trend is hyperbolic.
3.2 INFERENCE TIME
Figure 3 reports inference time per image on each architecture, as a function of image batch size (from 1 to 64). We notice that VGG processes one image in a fifth of a second, making it a less likely contender in real-time applications on an NVIDIA TX1. AlexNet shows a speed up of roughly 3× going from batch of 1 to 64 images, due to weak optimisation of its fully connected layers. It is a very surprising finding, that will be further discussed in the next subsection.
3.3 POWER
Power measurements are complicated by the high frequency swings in current consumption, which required high sampling current read-out to avoid aliasing. In this work, we used a 200MHz digital oscilloscope with a current probe, as reported in section 2. Other measuring instruments, such as an AC power strip with 2Hz sampling rate, or a GPIB controlled DC power supply with 12Hz sampling rate, did not provide enough bandwidth to properly conduct power measurements.
In figure 4 we see that the power consumption is mostly independent with the batch size. Low power values for AlexNet (batch of 1) and VGG (batch of 2) are associated to slower forward times per image, as shown in figure 3.
3.4 MEMORY
We analysed system memory consumption of the TX1 device, which uses shared memory for both CPU and GPU. Figure 5 shows that the maximum system memory usage is initially constant and then raises with the batch size. This is due the initial memory allocation of the network model — which is the large static component — and the contribution of the memory required while processing the batch, proportionally increasing with the number of images. In figure 6 we can also notice that the initial allocation never drops below 200MB, for network sized below 100MB, and it is linear afterwards, with respect to the parameters and a slope of 1.30.
3.5 OPERATIONS
Operations count is essential for establishing a rough estimate of inference time and hardware circuit size, in case of custom implementation of neural network accelerators. In figure 7, for a batch of 16 images, there is a linear relationship between operations count and inference time per image. Therefore, at design time, we can pose a constraint on the number of operation to keep processing speed in a usable range for real-time applications or resource-limited deployments.
3.6 OPERATIONS AND POWER
In this section we analyse the relationship between power consumption and number of operations required by a given model. Figure 8 reports that there is no specific power footprint for different architectures. When full resources utilisation is reached, generally with larger batch sizes, all networks consume roughly an additional 11.8W, with a standard deviation of 0.7W. Idle power is 1.30W. This corresponds to the maximum system power at full utilisation. Therefore, if energy consumption is one of our concerns, for example for battery-powered devices, one can simply choose the slowest architecture which satisfies the application minimum requirements.
3.7 ACCURACY AND THROUGHPUT
We note that there is a non-trivial linear upper bound between accuracy and number of inferences per unit time. Figure 9 illustrates that for a given frame rate, the maximum accuracy that can be achieved is linearly proportional to the frame rate itself. All networks analysed here come from several publications, and have been independently trained by other research groups. A linear fit of the accuracy shows all architecture trade accuracy vs. speed. Moreover, chosen a specific inference time, one can now come up with the theoretical accuracy upper bound when resources are fully
utilised, as seen in section 3.6. Since the power consumption is constant, we can even go one step further, and obtain an upper bound in accuracy even for an energetic constraint, which could possibly be an essential designing factor for a network that needs to run on an embedded system.
As the spoiler in section 3.1 gave already away, the linear nature of the accuracy vs. throughput relationship translates into a hyperbolical one when the forward inference time is considered instead. Then, given that the operations count is linear with the inference time, we get that the accuracy has an hyperbolical dependency on the amount of computations that a network requires.
3.8 PARAMETERS UTILISATION
DNNs are known to be highly inefficient in utilising their full learning power (number of parameters / degrees of freedom). Prominent work (Han et al., 2015) exploits this flaw to reduce network file size up to 50×, using weights pruning, quantisation and variable-length symbol encoding. It is worth noticing that, using more efficient architectures to begin with may produce even more compact representations. In figure 10 we clearly see that, although VGG has a better accuracy than AlexNet (as shown by figure 1), its information density is worse. This means that the amount of degrees of freedom introduced in the VGG architecture bring a lesser improvement in terms of accuracy. Moreover, ENet (Paszke et al., 2016) — which we have specifically designed to be highly efficient and it has been adapted and retrained on ImageNet (Culurciello, 2016) for this work — achieves the highest score, showing that 24× less parameters are sufficient to provide state-of-the-art results.
4 CONCLUSIONS
In this paper we analysed multiple state-of-the-art deep neural networks submitted to the ImageNet challenge, in terms of accuracy, memory footprint, parameters, operations count, inference time and power consumption. Our goal is to provide insights into the design choices that can lead to efficient neural networks for practical application, and optimisation of the often-limited resources in actual deployments, which lead us to the creation of ENet — or Efficient-Network — for ImageNet. We show that accuracy and inference time are in a hyperbolic relationship: a little increment in accuracy costs a lot of computational time. We show that number of operations in a network model can effectively estimate inference time. We show that an energy constraint will set a specific upper bound on the maximum achievable accuracy and model complexity, in terms of operations counts. Finally, we show that ENet is the best architecture in terms of parameters space utilisation, squeezing up to 13× more information per parameter used respect to the reference model AlexNet, and 24× respect VGG-19.
ACKNOWLEDGMENTS
This paper would have not look so pretty without the Python Software Foundation, the matplotlib library and the communities of stackoverflow and TEX of StackExchange which I ought to thank. This work is partly supported by the Office of Naval Research (ONR) grants N00014-12-10167, N00014-15-1-2791 and MURI N00014-10-1-0278. We gratefully acknowledge the support of NVIDIA Corporation with the donation of the TX1, Titan X, K40 GPUs used for this research.

The authors did solid work in collecting all the reported data. However, most findings don't seem to be too surprising to me:
- Finding #1 mainly shows that all architectures and batch sizes manage to utilize the GPU fully (or to the same percentage).
- Regarding Finding #2, I agree that from a linear relationship in Figure 9 you could conclude said hyperbolic relationship.
However, for this finding to be relevant, it has to hold especially for the latest generations of models. These cluster in the upper left corner of Figure 9 and on their own do not seem to show too much of a linear behaviour. Therefore I think there is not enough evidence to conclude asymptotic hyperbolic behaviour: For this the linear behaviour would have to be the stronger, the more models approach the upper left corner.
- Finding #3 seems to be a simple conclusion from finding #1: As long as slower models are better and faster models do draw the same power, finding #3 holds.
- Finding #4 is again similar to finding #1: If all architectures manage to fully utilize the GPU, inference time should be proportional to the number of operations.
Maybe the most interesting finding would be that all tested models seem to use the same percentage of computational resources available on the GPU, while one might expect that more complex models don't manage to utilize as much computational resources due to inter-dependencies. However actual GPU utilization was not evaluated and as the authors choose to use an older GPU, one would expect that all models manage to make use of all available computational power.
Additionally, I think these findings would have to be put in relation with compressing techniques or tested on actual production networks to be of more interest.


Mastering a video game requires skill, tactics and strategy. While these attributes may be acquired naturally by human players, teaching them to a computer program is a far more challenging task. In recent years, extensive research was carried out in the field of reinforcement learning and numerous algorithms were introduced, aiming to learn how to perform human tasks such as playing video games. As a result, the Arcade Learning Environment (ALE) (Bellemare et al., 2013) has become a commonly used benchmark environment allowing algorithms to train on various Atari 2600 games. In many games the state-of-the-art algorithms outperform humans. In this paper we introduce a new learning environment, the Retro Learning Environment — RLE, that can run games on the Super Nintendo Entertainment System (SNES), Sega Genesis and several other gaming consoles. The environment is expandable, allowing for more video games and consoles to be easily added to the environment, while maintaining the same interface as ALE. Moreover, RLE is compatible with Python and Torch. SNES games pose a significant challenge to current algorithms due to their higher level of complexity and versatility.
1 INTRODUCTION
Controlling artificial agents using only raw high-dimensional input data such as image or sound is a difficult and important task in the field of Reinforcement Learning (RL). Recent breakthroughs in the field allow its utilization in real-world applications such as autonomous driving (Shalev-Shwartz et al., 2016), navigation (Bischoff et al., 2013) and more. Agent interaction with the real world is usually either expensive or not feasible, as the real world is far too complex for the agent to perceive. Therefore in practice the interaction is simulated by a virtual environment which receives feedback on a decision made by the algorithm. Traditionally, games were used as a RL environment, dating back to Chess (Campbell et al., 2002), Checkers (Schaeffer et al., 1992), backgammon (Tesauro, 1995) and the more recent Go (Silver et al., 2016). Modern games often present problems and tasks which are highly correlated with real-world problems. For example, an agent that masters a racing game, by observing a simulated driver’s view screen as input, may be usefull for the development of an autonomous driver. For high-dimensional input, the leading benchmark is the Arcade Learning Environment (ALE) (Bellemare et al., 2013) which provides a common interface to dozens of Atari 2600 games, each presents a different challenge. ALE provides an extensive benchmarking platform, allowing a controlled experiment setup for algorithm evaluation and comparison. The main challenge posed by ALE is to successfully play as many Atari 2600 games as possible (i.e., achieving a score higher than an expert human player) without providing the algorithm any game-specific information (i.e., using the same input available to a human - the game screen and score). A key work to tackle this problem is the Deep Q-Networks algorithm (Mnih et al., 2015), which made a breakthrough in the field of Deep Reinforcement Learning by achieving human level performance on 29 out of 49 games. In this work we present a new environment — the Retro Learning Environment (RLE). RLE sets new challenges by providing a unified interface for Atari 2600 games as well as more advanced gaming consoles. As a start we focused on the Super Nintendo Entertainment
System (SNES). Out of the five SNES games we tested using state-of-the-art algorithms, only one was able to outperform an expert human player. As an additional feature, RLE supports research of multi-agent reinforcement learning (MARL) tasks (Buşoniu et al., 2010). We utilize this feature by training and evaluating the agents against each other, rather than against a pre-configured in-game AI. We conducted several experiments with this new feature and discovered that agents tend to learn how to overcome their current opponent rather than generalize the game being played. However, if an agent is trained against an ensemble of different opponents, its robustness increases. The main contributions of the paper are as follows:
• Introducing a novel RL environment with significant challenges and an easy agent evaluation technique (enabling agents to compete against each other) which could lead to new and more advanced RL algorithms.
• A new method to train an agent by enabling it to train against several opponents, making the final policy more robust.
• Encapsulating several different challenges to a single RL environment.
2 RELATED WORK
2.1 ARCADE LEARNING ENVIRONMENT
The Arcade Learning Environment is a software framework designed for the development of RL algorithms, by playing Atari 2600 games. The interface provided by ALE allows the algorithms to select an action and receive the Atari screen and a reward in every step. The action is the equivalent to a human’s joystick button combination and the reward is the difference between the scores at time stamp t and t− 1. The diversity of games for Atari provides a solid benchmark since different games have significantly different goals. Atari 2600 has over 500 games, currently over 70 of them are implemented in ALE and are commonly used for algorithm comparison.
2.2 INFINITE MARIO
Infinite Mario (Togelius et al., 2009) is a remake of the classic Super Mario game in which levels are randomly generated. On these levels the Mario AI Competition was held. During the competition, several algorithms were trained on Infinite Mario and their performances were measured in terms of the number of stages completed. As opposed to ALE, training is not based on the raw screen data but rather on an indication of Mario’s (the player’s) location and objects in its surrounding. This environment no longer poses a challenge for state of the art algorithms. Its main shortcoming lie in the fact that it provides only a single game to be learnt. Additionally, the environment provides hand-crafted features, extracted directly from the simulator, to the algorithm. This allowed the use of planning algorithms that highly outperform any learning based algorithm.
2.3 OPENAI GYM
The OpenAI gym (Brockman et al., 2016) is an open source platform with the purpose of creating an interface between RL environments and algorithms for evaluation and comparison purposes. OpenAI Gym is currently very popular due to the large number of environments supported by it. For example ALE, Go, MouintainCar and VizDoom (Zhu et al., 2016), an environment for the learning of the 3D first-person-shooter game ”Doom”. OpenAI Gym’s recent appearance and wide usage indicates the growing interest and research done in the field of RL.
2.4 OPENAI UNIVERSE
Universe (Universe, 2016) is a platform within the OpenAI framework in which RL algorithms can train on over a thousand games. Universe includes very advanced games such as GTA V, Portal as well as other tasks (e.g. browser tasks). Unlike RLE, Universe doesn’t run the games locally and requires a VNC interface to a server that runs the games. This leads to a lower frame rate and thus longer training times.
2.5 MALMO
Malmo (Johnson et al., 2016) is an artificial intelligence experimentation platform of the famous game ”Minecraft”. Although Malmo consists of only a single game, it presents numerous challenges since the ”Minecraft” game can be configured differently each time. The input to the RL algorithms include specific features indicating the ”state” of the game and the current reward.
2.6 DEEPMIND LAB
DeepMind Lab (Dee) is a first-person 3D platform environment which allows training RL algorithms on several different challenges: static/random map navigation, collect fruit (a form of reward) and a laser-tag challenge where the objective is to tag the opponents controlled by the in-game AI. In LAB the agent observations are the game screen (with an additional depth channel) and the velocity of the character. LAB supports four games (one game - four different modes).
2.7 DEEP Q-LEARNING
In our work, we used several variant of the Deep Q-Network algorithm (DQN) (Mnih et al., 2015), an RL algorithm whose goal is to find an optimal policy (i.e., given a current state, choose action that maximize the final score). The state of the game is simply the game screen, and the action is a combination of joystick buttons that the game responds to (i.e., moving ,jumping). DQN learns through trial and error while trying to estimate the ”Q-function”, which predicts the cumulative discounted reward at the end of the episode given the current state and action while following a policy π. The Q-function is represented using a convolution neural network that receives the screen as input and predicts the best possible action at it’s output. The Q-function weights θ are updated according to:
θt+1(st, at) = θt + α(Rt+1 + γmax a
(Qt(st+1, a; θ ′ t))−Qt(st, at; θt))∇θQt(st, at; θt), (1)
where st, st+1 are the current and next states, at is the action chosen, α is the step size, γ is the discounting factor Rt+1 is the reward received by applying at at st. θ′ represents the previous weights of the network that are updated periodically. Other than DQN, we examined two leading algorithms on the RLE: Double Deep Q-Learning (D-DQN) (Van Hasselt et al., 2015), a DQN based algorithm with a modified network update rule. Dueling Double DQN (Wang et al., 2015), a modification of D-DQN’s architecture in which the Q-function is modeled using a state (screen) dependent estimator and an action dependent estimator.
3 THE RETRO LEARNING ENVIRONMENT
3.1 SUPER NINTENDO ENTERTAINMENT SYSTEM
The Super Nintendo Entertainment System (SNES) is a home video game console developed by Nintendo and released in 1990. A total of 783 games were released, among them, the iconic Super Mario World, Donkey Kong Country and The Legend of Zelda. Table (1) presents a comparison between Atari 2600, Sega Genesis and SNES game consoles, from which it is clear that SNES and Genesis games are far more complex.
3.2 IMPLEMENTATION
To allow easier integration with current platforms and algorithms, we based our environment on the ALE, with the aim of maintaining as much of its interface as possible. While the ALE is highly coupled with the Atari emulator, Stella1, RLE takes a different approach and separates the learning environment from the emulator. This was achieved by incorporating an interface named LibRetro (libRetro site), that allows communication between front-end programs to game-console emulators. Currently, LibRetro supports over 15 game consoles, each containing hundreds of games, at an estimated total of over 7,000 games that can potentially be supported using this interface. Examples of supported game consoles include Nintendo Entertainment System, Game Boy, N64, Sega Genesis,
1http://stella.sourceforge.net/
Saturn, Dreamcast and Sony PlayStation. We chose to focus on the SNES game console implemented using the snes9x2 as it’s games present interesting, yet plausible to overcome challenges. Additionally, we utilized the Genesis-Plus-GX3 emulator, which supports several Sega consoles: Genesis/Mega Drive, Master System, Game Gear and SG-1000.
3.3 SOURCE CODE
RLE is fully available as open source software for use under GNU’s General Public License4. The environment is implemented in C++ with an interface to algorithms in C++, Python and Lua. Adding a new game to the environment is a relatively simple process.
3.4 RLE INTERFACE
RLE provides a unified interface to all games in its supported consoles, acting as an RL-wrapper to the LibRetro interface. Initialization of the environment is done by providing a game (ROM file) and a gaming-console (denoted by ’core’). Upon initialization, the first state is the initial frame of the game, skipping all menu selection screens. The cores are provided with the RLE and installed together with the environment. Actions have a bit-wise representation where each controller button is represented by a one-hot vector. Therefore a combination of several buttons is possible using the bit-wise OR operator. The number of valid buttons combinations is larger than 700, therefore only the meaningful combinations are provided. The environments observation is the game screen, provided as a 3D array of 32 bit per pixel with dimensions which vary depending on the game. The reward can be defined differently per game, usually we set it to be the score difference between two consecutive frames. By setting different configuration to the environment, it is possible to alter in-game properties such as difficulty (i.e easy, medium, hard), its characters, levels, etc.
3.5 ENVIRONMENT CHALLENGES
Integrating SNES and Genesis with RLE presents new challenges to the field of RL where visual information in the form of an image is the only state available to the agent. Obviously, SNES games are significantly more complex and unpredictable than Atari games. For example in sports games, such as NBA, while the player (agent) controls a single player, all the other nine players’ behavior is determined by pre-programmed agents, each exhibiting random behavior. In addition, many SNES games exhibit delayed rewards in the course of their play (i.e., reward for an actions is given many time steps after it was performed). Similarly, in some of the SNES games, an agent can obtain a reward that is indirectly related to the imposed task. For example, in platform games, such as Super Mario, reward is received for collecting coins and defeating enemies, while the goal of the challenge is to reach the end of the level which requires to move to keep moving to the right. Moreover, upon completing a level, a score bonus is given according to the time required for its completion. Therefore collecting coins or defeating enemies is not necessarily preferable if it consumes too much time. Analysis of such games is presented in section 4.2. Moreover, unlike Atari that consists of
2http://www.snes9x.com/ 3https://github.com/ekeeke/Genesis-Plus-GX 4https://github.com/nadavbh12/Retro-Learning-Environment
eight directions and one action button, SNES has eight-directions pad and six actions buttons. Since combinations of buttons are allowed, and required at times, the actual actions space may be larger than 700, compared to the maximum of 18 actions in Atari. Furthermore, the background in SNES is very rich, filled with details which may move locally or across the screen, effectively acting as non-stationary noise since it provided little to no information regarding the state itself. Finally, we note that SNES utilized the first 3D games. In the game Wolfenstein, the player must navigate a maze from a first-person perspective, while dodging and attacking enemies. The SNES offers plenty of other 3D games such as flight and racing games which exhibit similar challenges. These games are much more realistic, thus inferring from SNES games to ”real world” tasks, as in the case of self driving cars, might be more beneficial. A visual comparison of two games, Atari and SNES, is presented in Figure (1).
4 EXPERIMENTS
4.1 EVALUATION METHODOLOGY
The evaluation methodology that we used for benchmarking the different algorithms is the popular method proposed by (Mnih et al., 2015). Each examined algorithm is trained until either it reached convergence or 100 epochs (each epoch corresponds to 50,000 actions), thereafter it is evaluated by performing 30 episodes of every game. Each episode ends either by reaching a terminal state or after 5 minutes. The results are averaged per game and compared to the average result of a human player. For each game the human player was given two hours for training, and his performances were evaluated over 20 episodes. As the various algorithms don’t use the game audio in the learning process, the audio was muted for both the agent and the human. From both, humans and agents
score, a random agent score (an agent performing actions randomly) was subtracted to assure that learning indeed occurred. It is important to note that DQN’s -greedy approach (select a random action with a small probability ) is present during testing thus assuring that the same sequence of actions isn’t repeated. While the screen dimensions in SNES are larger than those of Atari, in our experiments we maintained the same pre-processing of DQN (i.e., downscaling the image to 84x84 pixels and converting to gray-scale). We argue that downscaling the image size doesn’t affect a human’s ability to play the game, therefore suitable for RL algorithms as well. To handle the large action space, we limited the algorithm’s actions to the minimal button combinations which provide unique behavior. For example, on many games the R and L action buttons don’t have any use therefore their use and combinations were omitted.
4.1.1 RESULTS
A thorough comparison of the four different agents’ performances on SNES games can be seen in Figure (). The full results can be found in Table (3). Only in the game Mortal Kombat a trained agent was able to surpass a expert human player performance as opposed to Atari games where the same algorithms have surpassed a human player on the vast majority of the games.
One example is Wolfenstein game, a 3D first-person shooter game, requires solving 3D vision tasks, navigating in a maze and detecting object. As evident from figure (2), all agents produce poor results indicating a lack of the required properties. By using -greedy approach the agents weren’t able to explore enough states (or even other rooms in our case). The algorithm’s final policy appeared as a random walk in a 3D space. Exploration based on visited states such as presented in Bellemare et al. (2016) might help addressing this issue. An interesting case is Gradius III, a side-scrolling, flight-shooter game. While the trained agent was able to master the technical aspects of the game, which includes shooting incoming enemies and dodging their projectiles, it’s final score is still far from a human’s. This is due to a hidden game mechanism in the form of ”power-ups”, which can be accumulated, and significantly increase the players abilities. The more power-ups collected without being use — the larger their final impact will be. While this game-mechanism is evident to a human, the agent acts myopically and uses the power-up straight away5.
4.2 REWARD SHAPING
As part of the environment and algorithm evaluation process, we investigated two case studies. First is a game on which DQN had failed to achieve a better-than-random score, and second is a game on which the training duration was significantly longer than that of other games.
In the first case study, we used a 2D back-view racing game ”F-Zero”. In this game, one is required to complete four laps of the track while avoiding other race cars. The reward, as defined by the score of the game, is only received upon completing a lap. This is an extreme case of a reward delay. A lap may last as long as 30 seconds, which span over 450 states (actions) before reward is received. Since DQN’s exploration is a simple -greedy approach, it was not able to produce a useful strategy. We approached this issue using reward shaping, essentially a modification of the reward to be a function of the reward and the observation, rather than the reward alone. Here, we define the reward to be the sum of the score and the agent’s speed (a metric displayed on the screen of the game). Indeed when the reward was defined as such, the agents learned to finish the race in first place within a short training period.
The second case study is the famous game of Super Mario. In this game the agent, Mario, is required to reach the right-hand side of the screen, while avoiding enemies and collecting coins. We found this case interesting as it involves several challenges at once: dynamic background that can change drastically within a level, sparse and delayed rewards and multiple tasks (such as avoiding enemies and pits, advancing rightwards and collecting coins). To our surprise, DQN was able to reach the end of the level without any reward shaping, this was possible since the agent receives rewards for events (collecting coins, stomping on enemies etc.) that tend to appear to the right of the player, causing the agent to prefer moving right. However, the training time required for convergence was significantly longer than other games. We defined the reward as the sum of the in-game reward and a bonus granted according the the player’s position, making moving right preferable. This reward
5A video demonstration can be found at https://youtu.be/nUl9XLMveEU
proved useful, as training time required for convergence decreased significantly. The two games above can be seen in Figure (3).
Figure (4) illustrates the agent’s average value function . Though both were able complete the stage trained upon, the convergence rate with reward shaping is significantly quicker due to the immediate realization of the agent to move rightwards.
4.3 MULTI-AGENT REINFORCEMENT LEARNING
In this section we describe our experiments with RLE’s multi-agent capabilities. We consider the case where the number of agents, n = 2 and the goals of the agents are opposite, as in r1 = −r2. This scheme is known as fully competitive (Buşoniu et al., 2010). We used the simple singleagent RL approach (as described by Buşoniu et al. (2010) section 5.4.1) which is to apply to single agent approach to the multi-agent case. This approach was proved useful in Crites and Barto (1996) and Matarić (1997). More elaborate schemes are possible such as the minimax-Q algorithm (Littman, 1994), (Littman, 2001). These may be explored in future works. We conducted three experiments on this setup: the first use was to train two different agents against the in-game AI, as done in previous sections, and evaluate their performance by letting them compete against each other. Here, rather than achieving the highest score, the goal was to win a tournament which consist of 50 rounds, as common in human-player competitions. The second experiment was to initially train two agents against the in-game AI, and resume the training while competing against each other. In this case, we evaluated the agent by playing again against the in-game AI, separately. Finally, in our last experiment we try to boost the agent capabilities by alternated it’s opponents, switching between the in-game AI and other trained agents.
4.3.1 MULTI-AGENT REINFORCEMENT LEARNING RESULTS
We chose the game Mortal Kombat, a two character side viewed fighting game (a screenshot of the game can be seen in Figure (1), as a testbed for the above, as it exhibits favorable properties: both players share the same screen, the agent’s optimal policy is heavily dependent on the rival’s behavior, unlike racing games for example. In order to evaluate two agents fairly, both were trained using the same characters maintaining the identity of rival and agent. Furthermore, to remove the impact of the starting positions of both agents on their performances, the starting positions were initialized randomly.
In the first experiment we evaluated all combinations of DQN against D-DQN and Dueling D-DQN. Each agent was trained against the in-game AI until convergence. Then 50 matches were performed between the two agents. DQN lost 28 out of 50 games against Dueling D-DQN and 33 against D-DQN. D-DQN lost 26 time to Dueling D-DQN. This win balance isn’t far from the random case, since the algorithms converged into a policy in which movement towards the opponent is not
required rather than generalize the game. Therefore, in many episodes, little interaction between the two agents occur, leading to a semi-random outcome.
In our second experiment, we continued the training process of a the D-DQN network by letting it compete against the Dueling D-DQN network. We evaluated the re-trained network by playing 30 episodes against the in-game AI. After training, D-DQN was able to win 28 out of 30 games, yet when faced again against the in-game AI its performance deteriorated drastically (from an average of 17000 to an average of -22000). This demonstrated a form of catastrophic forgetting (Goodfellow et al., 2013) even though the agents played the same game.
In our third experiment, we trained a Dueling D-DQN agent against three different rivals: the ingame AI, a trained DQN agent and a trained Dueling-DQN agent, in an alternating manner, such that in each episode a different rival was playing as the opponent with the intention of preventing the agent from learning a policy suitable for just one opponent. The new agent was able to achieve a score of 162,966 (compared to the ”normal” dueling D-DQN which achieved 169,633). As a new and objective measure of generalization, we’ve configured the in-game AI difficulty to be ”very hard” (as opposed to the default ”medium” difficulty). In this metric the alternating version achieved 83,400 compared to -33,266 of the dueling D-DQN which was trained in default setting. Thus, proving that the agent learned to generalize to other policies which weren’t observed while training.
4.4 FUTURE CHALLENGES
As demonstrated, RLE presents numerous challenges that have yet to be answered. In addition to being able to learn all available games, the task of learning games in which reward delay is extreme, such as F-Zero without reward shaping, remains an unsolved challenge. Additionally, some games, such as Super Mario, feature several stages that differ in background and the levels structure. The task of generalizing platform games, as in learning on one stage and being tested on the other, is another unexplored challenge. Likewise surpassing human performance remains a challenge since current state-of-the-art algorithms still struggling with the many SNES games.
5 CONCLUSION
We introduced a rich environment for evaluating and developing reinforcement learning algorithms which presents significant challenges to current state-of-the-art algorithms. In comparison to other environments RLE provides a large amount of games with access to both the screen and the ingame state. The modular implementation we chose allows extensions of the environment with new consoles and games, thus ensuring the relevance of the environment to RL algorithms for years to come (see Table (2)). We’ve encountered several games in which the learning process is highly dependent on the reward definition. This issue can be addressed and explored in RLE as reward definition can be done easily. The challenges presented in the RLE consist of: 3D interpretation, delayed reward, noisy background, stochastic AI behavior and more. Although some algorithms were able to play successfully on part of the games, to fully overcome these challenges, an agent must incorporate both technique and strategy. Therefore, we believe, that the RLE is a great platform for future RL research.
6 ACKNOWLEDGMENTS
The authors are grateful to the Signal and Image Processing Lab (SIPL) staff for their support, Alfred Agrell and the LibRetro community for their support and Marc G. Bellemare for his valuable inputs.

This paper introduces a new reinforcement learning environment called « The Retro Learning Environment”, that interfaces with the open-source LibRetro API to offer access to various emulators and associated games (i.e. similar to the Atari 2600 Arcade Learning Environment, but more generic). The first supported platform is the SNES, with 5 games (more consoles and games may be added later). Authors argue that SNES games pose more challenges than Atari’s (due to more complex graphics, AI and game mechanics). Several DQN variants are evaluated in experiments, and it is also proposed to compare learning algorihms by letting them compete against each other in multiplayer games.
I like the idea of going toward more complex games than those found on Atari 2600, and having an environment where new consoles and games can easily be added sounds promising. With OpenAI Universe and DeepMind Lab that just came out, though, I am not sure we really need another one right now. Especially since using ROMs of emulated games we do not own is technically illegal: it looks like this did not cause too much trouble for Atari but it might start raising eyebrows if the community moves to more advanced and recent games, especially some Nintendo still makes money from.
Besides the introduction of the environment, it is good to have DQN benchmarks on five games, but this does not add a lot of value. The authors also mention as contribution "A new benchmarking technique, allowing algorithms to compete against each other, rather than playing against the in-game AI", but this seems a bit exaggerated to me: the idea of pitting AIs against each other has been at the core of many AI competitions for decades, so it is hardly something new. The finding that reinforcement learning algorithms tend to specialize to their opponent is also not particular surprising.
Overall I believe this is an ok paper but I do not feel it brings enough to the table for a major conference. This does not mean, however, that this new environment won't find a spot in the (now somewhat crowded) space of game-playing frameworks.
Other small comments:
- There are lots of typos (way too many to mention them all)
- It is said that Infinite Mario "still serves as a benchmark platform", however as far as I know it had to be shutdown due to Nintendo not being too happy about it
- "RLE requires an emulator and a computer version of the console game (ROM file) upon initialization rather than a ROM file only. The emulators are provided with RLE" => how is that different from ALE that requires the emulator Stella which is also provided with ALE?
- Why is there no DQN / DDDQN result on Super Mario?
- It is not clear if Figure 2 displays the F-Zero results using reward shaping or not
- The Du et al reference seems incomplete


GRAPH CONVOLUTIONAL RECURRENT NETWORKS
This paper introduces Graph Convolutional Recurrent Network (GCRN), a deep learning model able to predict structured sequences of data. Precisely, GCRN is a generalization of classical recurrent neural networks (RNN) to data structured by an arbitrary graph. Such structured sequences can represent series of frames in videos, spatio-temporal measurements on a network of sensors, or random walks on a vocabulary graph for natural language modeling. The proposed model combines convolutional neural networks (CNN) on graphs to identify spatial structures and RNN to find dynamic patterns. We study two possible architectures of GCRN, and apply the models to two practical problems: predicting moving MNIST data, and modeling natural language with the Penn Treebank dataset. Experiments show that exploiting simultaneously graph spatial and dynamic information about data can improve both precision and learning speed.
1 INTRODUCTION
Many real-world data can be cast as structured sequences, with spatio-temporal sequences being a special case. A well-studied example of spatio-temporal data are videos, where succeeding frames share temporal and spatial structures. Many works, such as Donahue et al. (2015); Karpathy & FeiFei (2015); Vinyals et al. (2015), leveraged a combination of CNN and RNN to exploit such spatial and temporal regularities. Their models are able to process possibly time-varying visual inputs for variable-length prediction. These neural network architectures consist of combining a CNN for visual feature extraction followed by a RNN for sequence learning. Such architectures have been successfully used for video activity recognition, image captioning and video description.
More recently, interest has grown in properly fusing the CNN and RNN models for spatio-temporal sequence modeling. Inspired by language modeling, Ranzato et al. (2014) proposed a model to represent complex deformations and motion patterns by discovering both spatial and temporal correlations. They showed that prediction of the next video frame and interpolation of intermediate frames can be achieved by building a RNN-based language model on the visual words obtained by quantizing the image patches. Their highest-performing model, recursive CNN (rCNN), uses convolutions for both inputs and states. Shi et al. (2015) then proposed the convolutional LSTM network (convLSTM), a recurrent model for spatio-temporal sequence modeling which uses 2D-grid convolution to leverage the spatial correlations in input data. They successfully applied their model to the prediction of the evolution of radar echo maps for precipitation nowcasting.
The spatial structure of many important problems may however not be as simple as regular grids. For instance, the data measured from meteorological stations lie on a irregular grid, i.e. a network of heterogeneous spatial distribution of stations. More challenging, the spatial structure of data may not even be spatial, as it is the case for social or biological networks. Eventually, the interpretation that sentences can be regarded as random walks on vocabulary graphs, a view popularized by Mikolov et al. (2013), allows us to cast language analysis problems as graph-structured sequence models.
This work leverages on the recent models of Defferrard et al. (2016); Ranzato et al. (2014); Shi et al. (2015) to design the GCRN model for modeling and predicting time-varying graph-based data. The core idea is to merge CNN for graph-structured data and RNN to identify simultaneously meaningful spatial structures and dynamic patterns. A generic illustration of the proposed GCRN architecture is given by Figure 1.
2 PRELIMINARIES
2.1 STRUCTURED SEQUENCE MODELING
Sequence modeling is the problem of predicting the most likely future length-K sequence given the previous J observations:
x̂t+1, . . . , x̂t+K = arg max xt+1,...,xt+K
P (xt+1, . . . , xt+K |xt−J+1, . . . , xt), (1)
where xt ∈ D is an observation at time t and D denotes the domain of the observed features. The archetypal application being the n-gram language model (with n = J + 1), where P (xt+1|xt−J+1, . . . , xt) models the probability of word xt+1 to appear conditioned on the past J words in the sentence (Graves, 2013).
In this paper, we are interested in special structured sequences, i.e. sequences where features of the observations xt are not independent but linked by pairwise relationships. Such relationships are universally modeled by weighted graphs.
Data xt can be viewed as a graph signal, i.e. a signal defined on an undirected and weighted graph G = (V, E , A), where V is a finite set of |V| = n vertices, E is a set of edges and A ∈ Rn×n is a weighted adjacency matrix encoding the connection weight between two vertices. A signal xt : V → Rdx defined on the nodes of the graph may be regarded as a matrix xt ∈ Rn×dx whose column i is the dx-dimensional value of xt at the ith node. While the number of free variables in a structured sequence of length K is in principle O(nKdxK), we seek to exploit the structure of the space of possible predictions to reduce the dimensionality and hence make those problems more tractable.
2.2 LONG SHORT-TERM MEMORY
A special class of recurrent neural networks (RNN) that prevents the gradient from vanishing too quickly is the popular long short-term memory (LSTM) introduced by Hochreiter & Schmidhuber (1997). This architecture has proven stable and powerful for modeling long-range dependencies in various general-purpose sequence modeling tasks (Graves, 2013; Srivastava et al., 2015; Sutskever et al., 2014). A fully-connected LSTM (FC-LSTM) may be seen as a multivariate version of LSTM where the input xt ∈ Rdx , cell output ht ∈ [−1, 1]dh and states ct ∈ Rdh are all vectors. In this paper, we follow the FC-LSTM formulation of Graves (2013), that is:
i = σ(Wxixt +Whiht−1 + wci ct−1 + bi), f = σ(Wxfxt +Whfht−1 + wcf ct−1 + bf ), ct = ft ct−1 + it tanh(Wxcxt +Whcht−1 + bc), o = σ(Wxoxt +Whoht−1 + wco ct + bo), ht = o tanh(ct),
(2)
where denotes the Hadamard product, σ(·) the sigmoid function σ(x) = 1/(1 + e−x) and i, f, o ∈ [0, 1]dh are the input, forget and output gates. The weights Wx· ∈ Rdh×dx , Wh· ∈ Rdh×dh , wc· ∈ Rdh and biases bi, bf , bc, bo ∈ Rdh are the model parameters.1 Such a model is called fullyconnected because the dense matricesWx· andWh· linearly combine all the components of x and h. The optional peephole connections wc· ct, introduced by Gers & Schmidhuber (2000), have been found to improve performance on certain tasks.
2.3 CONVOLUTIONAL NEURAL NETWORKS ON GRAPHS
Generalizing convolutional neural networks (CNNs) to arbitrary graphs is a recent area of interest. Two approaches have been explored in the literature: (i) a generalization of the spatial definition of a convolution (Masci et al., 2015; Niepert et al., 2016) and (ii), a multiplication in the graph Fourier domain by the way of the convolution theorem (Bruna et al., 2014; Defferrard et al., 2016). Masci et al. (2015) introduced a spatial generalization of CNNs to 3D meshes. The authors used geodesic polar coordinates to define convolution operations on mesh patches, and formulated a deep learning architecture which allows comparison across different meshes. Hence, this method is tailored to manifolds and is not directly generalizable to arbitrary graphs. Niepert et al. (2016) proposed a spatial approach which may be decomposed in three steps: (i) select a node, (ii) construct its neighborhood and (iii) normalize the selected sub-graph, i.e. order the neighboring nodes. The extracted patches are then fed into a conventional 1D Euclidean CNN. As graphs generally do not possess a natural ordering (temporal, spatial or otherwise), a labeling procedure should be used to impose it. Bruna et al. (2014) were the first to introduce the spectral framework described below in the context of graph CNNs. The major drawback of this method is itsO(n2) complexity, which was overcome with the technique of Defferrard et al. (2016), which offers a linear complexity O(|E|) and provides strictly localized filters. Kipf & Welling (2016) took a first-order approximation of the spectral filters proposed by Defferrard et al. (2016) and successfully used it for semi-supervised classification of nodes. While we focus on the framework introduced by Defferrard et al. (2016), the proposed model is agnostic to the choice of the graph convolution operator ∗G . As it is difficult to express a meaningful translation operator in the vertex domain (Bruna et al., 2014; Niepert et al., 2016), Defferrard et al. (2016) chose a spectral formulation for the convolution operator on graph ∗G . By this definition, a graph signal x ∈ Rn×dx is filtered by a non-parametric kernel gθ(Λ) = diag(θ), where θ ∈ Rn is a vector of Fourier coefficients, as
y = gθ ∗G x = gθ(L)x = gθ(UΛUT )x = Ugθ(Λ)UTx ∈ Rn×dx , (3)
where U ∈ Rn×n is the matrix of eigenvectors and Λ ∈ Rn×n the diagonal matrix of eigenvalues of the normalized graph Laplacian L = In − D−1/2AD−1/2 = UΛUT ∈ Rn×n, where In is the identity matrix and D ∈ Rn×n is the diagonal degree matrix with Dii = ∑ j Aij (Chung, 1997). Note that the signal x is filtered by gθ with an element-wise multiplication of its graph Fourier transform UTx with gθ (Shuman et al., 2013). Evaluating (3) is however expensive, as the multiplication with U is O(n2). Furthermore, computing the eigendecomposition of L might
1A practical trick is to initialize the biases bi, bf and bo to one such that the gates are initially open.
be prohibitively expensive for large graphs. To circumvent this problem, Defferrard et al. (2016) parametrizes gθ as a truncated expansion, up to order K − 1, of Chebyshev polynomials Tk such that
gθ(Λ) = K−1∑ k=0 θkTk(Λ̃), (4)
where the parameter θ ∈ RK is a vector of Chebyshev coefficients and Tk(Λ̃) ∈ Rn×n is the Chebyshev polynomial of order k evaluated at Λ̃ = 2Λ/λmax − In. The graph filtering operation can then be written as
y = gθ ∗G x = gθ(L)x = K−1∑ k=0 θkTk(L̃)x, (5)
where Tk(L̃) ∈ Rn×n is the Chebyshev polynomial of order k evaluated at the scaled Laplacian L̃ = 2L/λmax − In. Using the stable recurrence relation Tk(x) = 2xTk−1(x) − Tk−2(x) with T0 = 1 and T1 = x, one can evaluate (5) in O(K|E|) operations, i.e. linearly with the number of edges. Note that as the filtering operation (5) is an order K polynomial of the Laplacian, it is K-localized and depends only on nodes that are at maximum K hops away from the central node, the K-neighborhood. The reader is referred to Defferrard et al. (2016) for details and an in-depth discussion.
3 RELATED WORKS
Shi et al. (2015) introduced a model for regular grid-structured sequences, which can be seen as a special case of the proposed model where the graph is an image grid where the nodes are well ordered. Their model is essentially the classical FC-LSTM (2) where the multiplications by dense matrices W have been replaced by convolutions with kernels W :
i = σ(Wxi ∗ xt +Whi ∗ ht−1 + wci ct−1 + bi), f = σ(Wxf ∗ xt +Whf ∗ ht−1 + wcf ct−1 + bf ), ct = ft ct−1 + it tanh(Wxc ∗ xt +Whc ∗ ht−1 + bc), o = σ(Wxo ∗ xt +Who ∗ ht−1 + wco ct + bo), ht = o tanh(ct),
(6)
where ∗ denotes the 2D convolution by a set of kernels. In their setting, the input tensor xt ∈ Rnr×nc×dx is the observation of dx measurements at time t of a dynamical system over a spatial region represented by a grid of nr rows and nc columns. The model holds spatially distributed hidden and cell states of size dh given by the tensors ct, ht ∈ Rnr×nc×dh . The size m of the convolutional kernels Wh· ∈ Rm×m×dh×dh and Wx· ∈ Rm×m×dh×dx determines the number of parameters, which is independent of the grid size nr × nc. Earlier, Ranzato et al. (2014) proposed a similar RNN variation which uses convolutional layers instead of fully connected layers. The hidden state at time t is given by
ht = tanh(σ(Wx2 ∗ σ(Wx1 ∗ xt)) + σ(Wh ∗ ht−1)), (7) where the convolutional kernels Wh ∈ Rdh×dh are restricted to filters of size 1x1 (effectively a fully connected layer shared across all spatial locations).
Observing that natural language exhibits syntactic properties that naturally combine words into phrases, Tai et al. (2015) proposed a model for tree-structured topologies, where each LSTM has access to the states of its children. They obtained state-of-the-art results on semantic relatedness and sentiment classification. Liang et al. (2016) followed up and proposed a variant on graphs. Their sophisticated network architecture obtained state-of-the-art results for semantic object parsing on four datasets. In those models, the states are gathered from the neighborhood by way of a weighted sum with trainable weight matrices. Those weights are however not shared across the graph, which would otherwise have required some ordering of the nodes, alike any other spatial definition of graph convolution. Moreover, their formulations are limited to the one-neighborhood of the current node, with equal weight given to each neighbor.
Motivated by spatio-temporal problems like modeling human motion and object interactions, Jain et al. (2016) developed a method to cast a spatio-temporal graph as a rich RNN mixture which
essentially associates a RNN to each node and edge. Again, the communication is limited to directly connected nodes and edges.
The closest model to our work is probably the one proposed by Li et al. (2015), which showed stat-of-the-art performance on a problem from program verification. Whereas they use the iterative procedure of the Graph Neural Networks (GNNs) model introduced by Scarselli et al. (2009) to propagate node representations until convergence, we instead use the graph CNN introduced by Defferrard et al. (2016) to diffuse information across the nodes. While their motivations are quite different, those models are related by the fact that a spectral filter defined as a polynomial of order K can be implemented as a K-layer GNN.2
4 PROPOSED GCRN MODELS
We propose two GCRN architectures that are quite natural, and investigate their performances in real-world applications in Section 5.
Model 1. The most straightforward definition is to stack a graph CNN, defined as (5), for feature extraction and an LSTM, defined as (2), for sequence learning:
xCNNt = CNNG(xt)
i = σ(Wxix CNN t +Whiht−1 + wci ct−1 + bi),
f = σ(Wxfx CNN t +Whfht−1 + wcf ct−1 + bf ), ct = ft ct−1 + it tanh(WxcxCNNt +Whcht−1 + bc), o = σ(Wxox CNN t +Whoht−1 + wco ct + bo),
ht = o tanh(ct).
(8)
In that setting, the input matrix xt ∈ Rn×dx may represent the observation of dx measurements at time t of a dynamical system over a network whose organization is given by a graph G. xCNNt is the output of the graph CNN gate. For a proof of concept, we simply choose here xCNNt = W
CNN ∗G xt, where WCNN ∈ RK×dx×dx are the Chebyshev coefficients for the graph convolutional kernels of support K. The model also holds spatially distributed hidden and cell states of size dh given by the matrices ct, ht ∈ Rn×dh . Peepholes are controlled by wc· ∈ Rn×dh . The weights Wh· ∈ Rdh×dh and Wx· ∈ Rdh×dx are the parameters of the fully connected layers. An architecture such as (8) may be enough to capture the data distribution by exploiting local stationarity and compositionality properties as well as the dynamic properties.
Model 2. To generalize the convLSTM model (6) to graphs we replace the Euclidean 2D convolution ∗ by the graph convolution ∗G :
i = σ(Wxi ∗G xt +Whi ∗G ht−1 + wci ct−1 + bi), f = σ(Wxf ∗G xt +Whf ∗G ht−1 + wcf ct−1 + bf ), ct = ft ct−1 + it tanh(Wxc ∗G xt +Whc ∗G ht−1 + bc), o = σ(Wxo ∗G xt +Who ∗G ht−1 + wco ct + bo), ht = o tanh(ct).
(9)
In that setting, the support K of the graph convolutional kernels defined by the Chebyshev coefficients Wh· ∈ RK×dh×dh and Wx· ∈ RK×dh×dx determines the number of parameters, which is independent of the number of nodes n. To keep the notation simple, we write Wxi ∗G xt to mean a graph convolution of xt with dhdx filters which are functions of the graph Laplacian L parametrized by K Chebyshev coefficients, as noted in (4) and (5). In a distributed computing setting, K controls the communication overhead, i.e. the number of nodes any given node i should exchange with in order to compute its local states.
The proposed blend of RNNs and graph CNNs is not limited to LSTMs and is straightforward to apply to any kind of recursive networks. For example, a vanilla RNN ht = tanh(Wxxt +Whht−1)
2The basic idea is to set the transition function as a diffusion and the output function such as to realize the polynomial recurrence, then stack K of those. See Defferrard et al. (2016) for details.
would be modified as ht = tanh(Wx ∗G xt +Wh ∗G ht−1), (10)
and a Gated Recurrent Unit (GRU) (Cho et al., 2014) as
z = σ(Wxz ∗G xt +Whz ∗G ht−1), r = σ(Wxr ∗G xt +Whr ∗G ht−1), h̃ = tanh(Wxh ∗G xt +Whh ∗G (r ht−1)), ht = z ht−1 + (1− z) h̃.
(11)
As demonstrated by Shi et al. (2015), structure-aware LSTM cells can be stacked and used as sequence-to-sequence models using an architecture composed of an encoder, which processes the input sequence, and a decoder, which generates an output sequence. A standard practice for machine translation using RNNs (Cho et al., 2014; Sutskever et al., 2014).
5 EXPERIMENTS
5.1 SPATIO-TEMPORAL SEQUENCE MODELING ON MOVING-MNIST
For this synthetic experiment, we use the moving-MNIST dataset generated by Shi et al. (2015). All sequences are 20 frames long (10 frames as input and 10 frames for prediction) and contain two handwritten digits bouncing inside a 64 × 64 patch. Following their experimental setup, all models are trained by minimizing the binary cross-entropy loss using back-propagation through time (BPTT) and RMSProp with a learning rate of 10−3 and a decay rate of 0.9. We choose the best model with early-stopping on validation set. All implementations are based on their Theano code and dataset.3 The adjacency matrix A is constructed as a k-nearest-neighbor (knn) graph with Euclidean distance and Gaussian kernel between pixel locations. For a fair comparison with Shi et al. (2015) defined in (6), all GCRN experiments are conducted with Model 2 defined in (9), which is the same architecture with the 2D convolution ∗ replaced by a graph convolution ∗G . To further explore the impact of the isotropic property of our filters, we generated a variant of the moving MNIST dataset where digits are also rotating (see Figure 4).
Table 1 shows the performance of various models: (i) the baseline FC-LSTM from Shi et al. (2015), (ii) the 1-layer LSTM+CNN from Shi et al. (2015) with different filter sizes, and (iii) the proposed LSTM+graph CNN(GCNN) defined in (9) with different supports K. These results show the ability of the proposed method to capture spatio-temporal structures. Perhaps surprisingly, GCNNs can offer better performance than regular CNNs, even when the domain is a 2D grid and the data is images, the problem CNNs were initially developed for. The explanation is to be found in the differences between 2D filters and spectral graph filters. While a spectral filter of support K = 3 corresponds to the reach of a patch of size 5×5 (see Figure 2), the difference resides in the isotropic nature of the former and the number of parameters: K = 3 for the former and 52 = 25 for the later.
3http://www.wanghao.in/code/SPARNN-release.zip
Table 1 indeed shows that LSTM+CNN(5 × 5) rivals LSTM+GCNN with K = 3. However, when increasing the filter size to 9× 9 or K = 5, the GCNN variant clearly outperforms the CNN variant. This experiment demonstrates that graph spectral filters can obtain superior performance on regular domains with much less parameters thanks to their isotropic nature, a controversial property. Indeed, as the nodes are not ordered, there is no notion of an edge going up, down, on the right or on the left. All edges are treated equally, inducing some sort of rotation invariance. Additionally, Table 1 shows that the computational complexity of each model is linear with the filter size, and Figure 3 shows the learning dynamic of some of the models.
5.2 NATURAL LANGUAGE MODELING ON PENN TREEBANK
The Penn Treebank dataset has 1,036,580 words. It was pre-processed in Zaremba et al. (2014) and split4 into a training set of 929k words, a validation set of 73k words, and a test set of 82k words. The size of the vocabulary of this corpus is 10,000. We use the gensim library5 to compute a word2vec model (Mikolov et al., 2013) for embedding the words of the dictionary in a 200-dimensional space. Then we build the adjacency matrix of the word embedding using a 4-nearest neighbor graph with cosine distance. Figure 6 presents the computed adjacency matrix, and its 3D visualization. We used the hyperparameters of the small configuration given by the code6 based on Zaremba et al. (2014): the size of the data mini-batch is 20, the number of temporal steps to unroll is 20, the dimension of the hidden state is 200. The global learning rate is 1.0 and the norm of the gradient is bounded by 5. The learning decay function is selected to be 0.5max(0,#epoch−4). All experiments have 13 epochs, and dropout value is 0.75. For Zaremba et al. (2014), the input representation xt can be either the 200-dim embedding vector of the word, or the 10,000-dim one-hot representation of the word. For
4https://github.com/wojzaremba/lstm 5https://radimrehurek.com/gensim/models/word2vec.html
our models, the input representation is a one-hot representation of the word. This choice allows us to use the graph structure of the words.
Table 2 reports the final train and test perplexity values for each investigated model and Figure 5 plots the perplexity value vs. the number of epochs for the train and test sets with and without dropout regularization. Numerical experiments show:
1. Given the same experimental conditions in terms of architecture and no dropout regularization, the standalone model of LSTM is more accurate than LSTM using the spatial graph information (120.16 vs. 177.14), extracted by graph CNN with the GCRN architecture of Model 1, Eq. (8).
2. However, using dropout regularization, the graph LSTM model overcomes the standalone LSTM with perplexity values 98.67 vs. 112.98.
3. The use of spatial graph information found by graph CNN speeds up the learning process, and overfits the training dataset in the absence of dropout regularization. The graph structure likely acts a constraint on the learning system that is forced to move in the space of language topics.
4. We performed the same experiments with LSTM and Model 2 defined in (9). Model 1 significantly outperformed Model 2, and Model 2 did worse than standalone LSTM. This bad performance may be the result of the large increase of dimensionality in Model 2, as the dimension of the hidden and cell states changes from 200 to 10,000, the size of the vocabulary. A solution would be to downsize the data dimensionality, as done in Shi et al. (2015) in the case of image data.
6https://github.com/tensorflow/tensorflow/blob/master/tensorflow/ models/rnn/ptb/ptb_word_lm.py
6 CONCLUSION AND FUTURE WORK
This work aims at learning spatio-temporal structures from graph-structured and time-varying data. In this context, the main challenge is to identify the best possible architecture that combines simultaneously recurrent neural networks like vanilla RNN, LSTM or GRU with convolutional neural networks for graph-structured data. We have investigated here two architectures, one using a stack of CNN and RNN (Model 1), and one using convLSTM that considers convolutions instead of fully connected operations in the RNN definition (Model 2). We have then considered two applications: video prediction and natural language modeling. Model 2 has shown good performances in the case of video prediction, by improving the results of Shi et al. (2015). Model 1 has also provided promising performances in the case of language modeling, particularly in terms of learning speed. It has been shown that (i) isotropic filters, maybe surprisingly, can outperform classical 2D filters on images while requiring much less parameters, and (ii) that graphs coupled with graph CNN and RNN are a versatile way of introducing and exploiting side-information, e.g. the semantic of words, by structuring a data matrix.
Future work will investigate applications to data naturally structured as dynamic graph signals, for instance fMRI and sensor networks. The graph CNN model we have used is rotationally-invariant and such spatial property seems quite attractive in real situations where motion is beyond translation. We will also investigate how to benefit of the fast learning property of our system to speed up language modeling models. Eventually, it will be interesting to analyze the underlying dynamical property of generic RNN architectures in the case of graphs. Graph structures may introduce stability to RNN systems, and prevent them to express unstable dynamic behaviors.
ACKNOWLEDGMENT
This research was supported in part by the European Union’s H2020 Framework Programme (H2020-MSCA-ITN-2014) under grant No. 642685 MacSeNet, and Nvidia equipment grant.

The authors address the problem of modeling temporally-changing signal on a graph, where the signal at one node changes as a function of the inputs and the hidden states of its neighborhood, the size of which is a hyperparameter. The approach follows closely that of Shi et al. 2015, but it is generalized to arbitrary graph structures rather than a fixed grid by using graph convolutions of Defferrard et al. 2016. This is not a strict generalization because the graph formulation treats all edges equally, while the conv kernels in Shi et al. have a built in directionality. The authors show results on a moving MNIST and on the Penn Tree Bank Language Modeling task.
The paper, model and experiments are decent but I have some concerns:
1. The proposed model is not exceptionally novel from a technical perspective. I usually don't mind if this is the case provided that the authors make up for the deficiency with thorough experimental evaluation, clear write up, and interesting insights into the pros/cons of the approach with respect to previous models. In this case I lean towards this not being the case.
2. The experiment results section is rather terse and light on interpretation. I'm not fully up to date on the latest of Penn Tree Bank language modeling results but I do know that it is a hotly contested and well-known dataset. I am surprised to see a comparison only to Zaremba et al 2014 where I would expect to see multiple other results.
3. The writing is not very clear and the authors don't make sufficiently strong attempt to compare the models or provide insight or comparisons into why the proposed model works better. In particular, unless I'm mistaken the word probabilities are a function of the neighborhood in the graph. What is the width of this graph? For example, suppose I sample a word in one part of the graph, doesn't this information have to propagate to the other parts of the graph along the edges? Also, it's not clear to me how the model can achieve reasonable results on moving MNIST when it cannot distinguish the direction of the moving edges. The authors state this but do not provide satisfying insight into how this can work. How does a pixel know that it should turn on in the next frame? I wish the authors thought about this more and presented it more clearly.
In summary, the paper has somewhat weak technical contribution, the experiments section is not very thorough, and the insights are sparse.