Thank you K. Frank, that was exactly the problem.
Instead of adding an “interpolated write” (which is indeed a bit odd), I re-worked things to offset the read pointer by _delay, and reworked the read accordingly:
i0 = torch.floor(self._read_ptr).data[0].long()
i1 = (i0 + 1) % self.N
frac = self._read_ptr - torch.floor(self._read_ptr)
y[i] = self.circ_buffer[i0] + frac*(self.circ_buffer[i1]-self.circ_buffer[i0])

Now, it converges nicely (although slightly slowly).