Thanks! I think the first point of you is correct.  But the second might be slightly different from what I manage to do.
I use binary gates self.alpha_gate, which is a tensor composed of 0 and 1, to specify the usage of candidate operations. During the forward process, the final result can be treated as the multiplication of self.alpha_gate and the output of each candidate operation. Then finally, I want to compute the gradient w.r.t self.alpha_gate.
However, during the computation, I want to save the time of forward computation of all the candidate operations, as there is only one of them contributing to the final output. Therefore, I derive active_idx from the location of 1 from self.alpha_gate, then compute only the output of this active operation. Then during the backward process, I still want to compute the gradient towards  self.alpha_gate.
I think that is roughly what I try to do.