Hi, I’m new to huggingface. For my current exeperience, the code I’ve read about shows that the data will be downloaded to local before next steps.
Is that possible to read stream data from remote source like hdfs? Since my training data mab be huge, it will exhaust my disk storage for downloading the data.
Will be thankful for any hint.