Playing SNES in the Retro Learning Environment
Mastering a video game requires skill, tactics and strategy. While these attributes may be acquired naturally by human players, teaching them to a computer program is a far more challenging task. In recent years, extensive research was carriedout in the field of reinforcement learning and numerous algorithms were introduced, aiming to learn how to perform human tasks such as playing video games. As a result, the Arcade Learning Environment (ALE) (Bellemare et al., 2013) has become a commonly used benchmark environment allowing algorithms to train on various Atari 2600 games. In many games the state-of-the-art algorithms outperform humans. In this paper we introduce a new learning environment, the Retro Learning Environment — RLE, that can run games on the Super Nintendo Entertainment System (SNES), Sega Genesis and several other gaming consoles. The environment is expandable, allowing for more video games and consoles to be easily added to the environment, while maintaining the same interface as ALE. Moreover, RLE is compatible with Python and Torch. SNES games pose a significant challenge to current algorithms due to their higher level of complexity and versatility.
This paper presents a valuable new collection of video game benchmarks, in an extendable framework, and establishes initial baselines on a few of them. Reward structures: for how many of the possible games have you implemented the means to extract scores and incremental reward structures? From the github repo it looks like about 10 -- do you plan to add more, and when? “rivalry” training: this is one of the weaker components of the paper, and it should probably be emphasised less. On this topic, there is a vast body of (uncited) multi-agent literature, it is a well-studied problem setup (more so than RL itself). To avoid controversy, I would recommend not claiming any novel contribution on the topic (I don’t think that you really invented “a new method to train an agent by enabling it to train against several opponents” nor “a new benchmarking technique for agents evaluation, by enabling them to compete against each other, rather than playing against the in-game AI”). Instead, just explain that you have established single-agent and multi-agent baselines for your new benchmark suite. Your definition of Q-function (“predicts the score at the end of the game given the current state and selected action”) is incorrect. It should read something like: it estimates the cumulative discounted reward that can be obtained from state s, starting with action a (and then following a certain policy). Minor: * Eq (1): the Q-net inside the max() is the target network, with different parameters theta’ * the Du et al. reference is missing the year * some of the other references should point at the corresponding published papers instead of the arxiv versions
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Playing SNES in the Retro Learning Environment
Mastering a video game requires skill, tactics and strategy. While these attributes may be acquired naturally by human players, teaching them to a computer program is a far more challenging task. In recent years, extensive research was carriedout in the field of reinforcement learning and numerous algorithms were introduced, aiming to learn how to perform human tasks such as playing video games. As a result, the Arcade Learning Environment (ALE) (Bellemare et al., 2013) has become a commonly used benchmark environment allowing algorithms to train on various Atari 2600 games. In many games the state-of-the-art algorithms outperform humans. In this paper we introduce a new learning environment, the Retro Learning Environment — RLE, that can run games on the Super Nintendo Entertainment System (SNES), Sega Genesis and several other gaming consoles. The environment is expandable, allowing for more video games and consoles to be easily added to the environment, while maintaining the same interface as ALE. Moreover, RLE is compatible with Python and Torch. SNES games pose a significant challenge to current algorithms due to their higher level of complexity and versatility.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This paper presents a valuable new collection of video game benchmarks, in an extendable framework, and establishes initial baselines on a few of them. Reward structures: for how many of the possible games have you implemented the means to extract scores and incremental reward structures? From the github repo it looks like about 10 -- do you plan to add more, and when? “rivalry” training: this is one of the weaker components of the paper, and it should probably be emphasised less. On this topic, there is a vast body of (uncited) multi-agent literature, it is a well-studied problem setup (more so than RL itself). To avoid controversy, I would recommend not claiming any novel contribution on the topic (I don’t think that you really invented “a new method to train an agent by enabling it to train against several opponents” nor “a new benchmarking technique for agents evaluation, by enabling them to compete against each other, rather than playing against the in-game AI”). Instead, just explain that you have established single-agent and multi-agent baselines for your new benchmark suite. Your definition of Q-function (“predicts the score at the end of the game given the current state and selected action”) is incorrect. It should read something like: it estimates the cumulative discounted reward that can be obtained from state s, starting with action a (and then following a certain policy). Minor: * Eq (1): the Q-net inside the max() is the target network, with different parameters theta’ * the Du et al. reference is missing the year * some of the other references should point at the corresponding published papers instead of the arxiv versions
assistant
This paper presents a valuable new collection of video game benchmarks, in an extendable framework, and establishes initial baselines on a few of them. Reward structures: for how many of the possible games have you implemented the means to extract scores and incremental reward structures? From the github repo it looks like about 10 -- do you plan to add more, and when? “rivalry” training: this is one of the weaker components of the paper, and it should probably be emphasised less. On this topic, there is a vast body of (uncited) multi-agent literature, it is a well-studied problem setup (more so than RL itself). To avoid controversy, I would recommend not claiming any novel contribution on the topic (I don’t think that you really invented “a new method to train an agent by enabling it to train against several opponents” nor “a new benchmarking technique for agents evaluation, by enabling them to compete against each other, rather than playing against the in-game AI”). Instead, just explain that you have established single-agent and multi-agent baselines for your new benchmark suite. Your definition of Q-function (“predicts the score at the end of the game given the current state and selected action”) is incorrect. It should read something like: it estimates the cumulative discounted reward that can be obtained from state s, starting with action a (and then following a certain policy). Minor: * Eq (1): the Q-net inside the max() is the target network, with different parameters theta’ * the Du et al. reference is missing the year * some of the other references should point at the corresponding published papers instead of the arxiv versions