@ptrblck
Thank you for your reply. The input tensor are 3D point clouds which is Nx3. N depends on the point clouds that in this batch, typical range of N is from 80k to 200k. Also, I have tried to export max allocated and memory of JointNet with Detection and JointNet without Detection.  The results are: 5GB allocated 4GB cached for JointNet with Detection and 17GB allocated 16GB cached for JointNet with Detection.
As for the intermediate activations, if the output of the detection block, the score map of each row of input tensor,  is used in loss function as a multiplier, will the output require gradient? I think it does right?