OK if anyone is still interested, I had the tread author’s mystery solved, at least for the most part.
The problem lies in the implementation of the pure virtual function “reset”, which is called by the clone function. The key issue is that, it is not enough to re-register the submodules there, but it should also re-construct the submodules (or network layers; maybe I am not using the correct terminology), i.e. the “gru”, “fc” in the author’s example (I’m ignoring the testMember here as it is probably not a network parameter and the copy of this integer member, if needed, should be relatively easy), and the re-construction needs to be done before re-registering. To be more concrete, the reset function in the author’s example should be implemented something like below. (Disclaimer: I did not test this exact example but tested on my own example, but the issue is similar enough for having confidence.)
void reset() override {
    gru = torch::nn::GRUOptions(128, 1024).num_layers(2).batch_first(true);
    fc = torch::nn::Linear(1024, 16);
    register_module("gru", gru);
    register_module("fc", fc);
}

The reason behind this, if I am not understanding it wrong, is that if we don’t re-construct the submodules and only re-register them, the parameters are going to be registered to the same memory space as those in the old submodules, thus causing the clone being not clean.
This is really an annoying issue and took me quite some time to figure it out. Thank the author for providing me some inspirations on it. I hope the documentation can be clearer on this. Hope this can help whoever running into this issue later.