Ah ok, I see.
The input is usually always defined with require_grad=False, so you won’t save any memory.
You are most likely right, that it will be really close to the limit. Maybe using checkpoint it’ll be possible.
Let me know, if you’ve tried it out.
I’ve edited the previous post, since I think you were right about the forward/backward memory usage.
Sorry for the confusion!