It is true that some layers need to store intermediate tensors to be able to calculate the gradients during the backward pass.
If you are running out of memory you could either decrease the batch size (or number of points, if possible). I don’t know, if e.g. torch.utils.checkpoint can be used in libtorch, but that might be another approach to trade compute for memory.