Answers.Body: <p>If I understand your problem correctly, you can test on just about any environment, and just omit parts of the observations to ensure your RNN is learning. For example, you can test on cartpole, ignoring the velocity and angular velocity states. This way the MDP isn't actually Markovian and you'll need the RNN to learn.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-21T16:44:43.463Z
Answers.Text: Hi harwiltz, appreciate your quick reply. I am in fact not familiar with the Gym environment and didn't know there are observations like velocity. My understanding on Gym comes from reading a few well cited paper such as the one for Rainbow DQN where they seem to use the game frame images as input and hence using CNN as the internal network. I don't know how I can feed such environment data to a LSTM/RNN instead. Or do I not have to use frame image data as input?
Answers.UserDisplayName: ZXY
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-21T18:35:14.813Z
Answers.Text: Not all gym environments have pixel state spaces. If you want to train from pixels, you should probably make a CNN feature extractor and pass the extracted featured to your RNN. If you just want to test your network though, I'd recommend training on simpler environments (look up cartpole, lunarlander, they're both in openai gym for example), whose state spaces are basically just position, velocity, etc, and discard some dimensions to force your RNN to learn.
Answers.UserDisplayName: harwiltz
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-21T23:08:25.143Z
Answers.Text: I think I get the picture. Will go and do some study. Thanks. Last question, if I only use data like position, velocity etc as state space for training, will I still be able to get good testing results? After all, getting good Atari game results is how I would tell if my own model is good or not.
Answers.UserDisplayName: ZXY
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-22T04:26:51.357Z
Answers.Text: With Atari, you can only get pixel or RAM states, both of which you'll probably need to implement a feature extractor to preprocess your RNN inputs. Otherwise, when you have access to states like position and velocity and such, that generally makes the problem much easier (the data will have much lower dimension).
Answers.UserDisplayName: harwiltz
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-21T15:10:04.220Z
Answers.LastActivityDate: 2020-06-21T15:10:04.220Z
Answers.UserDisplayName: harwiltz
Body: <p>I am writing a couple of different reinforcement learning models based on Rainbow DQN or some PG models. All of them internally use an LSTM network because my project is using time series data.</p>
  <p>I wanted to test my models using OpenAI Gym before I add too many domain specific code to the models.</p>
  <p>The problem is that, all of the Atari games seem to fall into the CNN area which I don't use.</p>
  <p>Is it possible to use OpenAI Gym to test any time series data driven RL models/networks?</p>
  <p>If not, is there any good environment that I can use to examine the validity of my models?</p>
  
Tags: <machine-learning><reinforcement-learning><long-short-term-memory><open-ai><time-series>
Title: How do I test an LSTM-based reinforcement learning model using any Atari games in OpenAI gym?
UserDisplayName: ZXY
ThreadText: # How do I test an LSTM-based reinforcement learning model using any Atari games in OpenAI gym?
  
  (Asked by: username_0 on 2020-06-21)
  
  I am writing a couple of different reinforcement learning models based on Rainbow DQN or some PG models. All of them internally use an LSTM network because my project is using time series data.
  
  I wanted to test my models using OpenAI Gym before I add too many domain specific code to the models.
  
  The problem is that, all of the Atari games seem to fall into the CNN area which I don't use.
  
  Is it possible to use OpenAI Gym to test any time series data driven RL models/networks?
  
  If not, is there any good environment that I can use to examine the validity of my models?
  
  **Tags:** machine-learning, reinforcement-learning, long-short-term-memory, open-ai, time-series
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-06-21. Score: 1
  
  If I understand your problem correctly, you can test on just about any environment, and just omit parts of the observations to ensure your RNN is learning. For example, you can test on cartpole, ignoring the velocity and angular velocity states. This way the MDP isn't actually Markovian and you'll need the RNN to learn.
  
  > **Comments:**
  > **username_0**: Hi username_1, appreciate your quick reply. I am in fact not familiar with the Gym environment and didn't know there are observations like velocity. My understanding on Gym comes from reading a few well cited paper such as the one for Rainbow DQN where they seem to use the game frame images as input and hence using CNN as the internal network. I don't know how I can feed such environment data to a LSTM/RNN instead. Or do I not have to use frame image data as input?  
  > **username_1**: Not all gym environments have pixel state spaces. If you want to train from pixels, you should probably make a CNN feature extractor and pass the extracted featured to your RNN. If you just want to test your network though, I'd recommend training on simpler environments (look up cartpole, lunarlander, they're both in openai gym for example), whose state spaces are basically just position, velocity, etc, and discard some dimensions to force your RNN to learn.  
  > **username_0**: I think I get the picture. Will go and do some study. Thanks. Last question, if I only use data like position, velocity etc as state space for training, will I still be able to get good testing results? After all, getting good Atari game results is how I would tell if my own model is good or not.  
  > **username_1**: With Atari, you can only get pixel or RAM states, both of which you'll probably need to implement a feature extractor to preprocess your RNN inputs. Otherwise, when you have access to states like position and velocity and such, that generally makes the problem much easier (the data will have much lower dimension).  
  
  ---
  
  