  Efforts to detect a primordial $B$-mode of CMB polarization generated by
inflationary gravitational waves ought to mitigate the large variance
associated with the $B$-modes produced by gravitational lensing, a process
known as delensing. A popular approach to delensing entails building a lensing
$B$-mode template by mimicking the lensing operation, either at gradient order
or non-perturbatively, using high-resolution $E$-mode observations and some
proxy of the lensing potential. By explicitly calculating all contributions to
two-loop order in lensing to the power spectrum of $B$-modes delensed with such
a template in the noise-free limit, we are able to show that: (i) corrections
to the leading-order calculation of the lensing $B$-mode power spectrum only
enter at the $O(1)\,\%$ level because of extensive cancellations between large
terms at next-to-leading order; (ii) these cancellations would disappear if a
gradient-order template were to be built from unlensed or delensed $E$-modes,
giving rise to a residual delensing floor of $O(10)\,\%$ of the original power;
(iii) new cancellations arise when the lensed $E$-modes are used in the
gradient-order template, allowing for the delensing floor to be as low as
$O(1)\,\%$ of the original power in practical applications of this method; and
(iv) these new cancellations would disappear for a non-perturbative template
constructed from the lensed $E$-modes, reintroducing a residual delensing floor
of $O(10)\,\%$. We further show that the gradient-order template outperforms
the non-perturbative one in realistic scenarios with noisy estimates of the
$E$-mode polarization and lensing potential. We therefore recommend that in
practical applications of $B$-mode template delensing, where the template is
constructed directly from the (filtered) observed $E$-modes, the gradient-order
approach should be used rather than a non-perturbative remapping.

Cosmology and Nongalactic Astrophysics