I found out the reason. In my model, I have this tensor:
self.log_alpha = torch.zeros(1, requires_grad=True)
Using it in my loss is fine:
            alpha_loss = -(self.log_alpha * (log_pi + self.target_entropy).detach()).mean()
            self.alpha_optimizer.zero_grad()
            alpha_loss.backward()

But calling a view operation in my training loop, even with detach(), causes the ViewBackward recursion. Why?
    def _do_training(self):
        # self.log_alpha = self.log_alpha.view(1).detach()