The thing is that for example the conv needs it’s output’s value to compute the backward. So even if you consider these as a single op, if you want maximum speed, you will need to save this Tensor.
If you are ready to trade off some speed for memory, you can use the checkpointing 7 tool that will reduce the memory usage (but will slow down the backward).