Ah ok, I thought you’ve added another Conv layer in front of it. 
The weights are shared, but each sample produces an activation, which has to be stored for backward.
Or what do you mean by weight sharing?
I guess you can’t pass the input as a few smaller batches, since you need to stack it and then perform some other operations on it, right?
Alternatively, you could try out model checkpoint 23.
It’s quite a new method, which allows you to trade memory for compute performance.
The article describes it pretty clearly.
You would have to wait a few days for the 0.4 release or just build PyTorch from source. It should be quite easy to build it, but let me know if you encounter any errors.