Each cat operation is append-only modification.

But the autograd cannot know that. You rewrite the whole buffer everytime. It just happen that the first part that you write is the same as what was already there?
Unfortunately we cannot detect that. If you write values in, we consider that the content has changed. We cannot check every value.