Well, I couldn’t help asking this, my textual data is structured, and no matter the loading is in C++ or Pytorch's Dataset, the loaded lines have to be parsed.
As tested, C++ loading (single thread) is much faster (like 10x) than the Python data-loader (single worker), maybe increase num_worker could help some, but do you think if it’s advisable to just plugin the C++ loader (py wrapped) into my pipeline?