My guess at the moment is the replayMemory memory, which at every frame pushes 2 states of my game and since each of my states are 300x300, the memory blows up
memory.push(state, action, next_state, reward)
edit: indeed, I commented this out, along with the optimizer function. Just commenting out the optimizer, i still ran out of memory, but once I commented out the memory push, my GPU memory was free for 1000 games. I guess I shall try to train using the difference of current and last screen like in the cartpool training