Hi @ethancaballero,
I know you posted this a long time ago, but did you ever manage to get anything (non Atari) working using OpenAi Universe?
Seems not many folks have had success with OpenAIs Universe - not many papers use/cite it- but would love to hear otherwise?