It is not super clear. But if the checkpoint proposed above didn’t work to reduce the memory enough. The you most likely won’t have enough memory.
In particular, can you estimate the memory used by one of such forward + state for it. And see if that will fit?