@denizs
You can use this approach from the OpenAI baseline implementation that greatly optimizes memory requirements
https://github.com/openai/baselines/blob/master/baselines/common/atari_wrappers_deprecated.py#L152 63