It seems that it is really related to how the torch_geometric.data.Dataset class handles the stuff. Hope there will be some more replies to this thread and bringing in some more new ideas.
Thank you so much for you help!