loss <- c <- b ------\
          <- z <- b  - >x
               <- z_prev <- ....

So your loss depends on all z's which depends on all bs. Then each new iteration adds another (z, b) pair to the graph. Hence the slow down.
To fix your script depends on where you want the gradient to stop flowing back. E.g., if you want it to stop going to z_prev (and therefore all previous z's and b's), do instead z = updateZ(z.detach(), b.view(N, -1 , xdim[1]))