Thanks for the reply @xzhu1900. I was able to find out that this functionality exists in the C++ API, but I’m not sure how to use that functionality in my Python code as there don’t seem to be any existing bindings. I was told that you can use pybind for this but I’m not exactly sure how to structure that so I can write the ChunkDataReader in Python. Are there any good examples of similar work in the pytorch codebase that might serve as a good example?
Edit:
Soon after writing this I saw that there are PRs under way to create Python bindings for this. For anyone who is interested, see https://github.com/pytorch/pytorch/pull/21232 30 for details.