Based on the description it seems the the issue is not a detached computation graph, which wouldn’t create gradients at all and thus wouldn’t cause any updates, but apparently vanishing gradients with might go to zero, which would explain the initial updates.
Could you verify it by checking the gradients in each iterations and print the max. absolute magnitude of them?