Hello,
Our team is in the process of creating (manually for now) a multilingual machine translation dataset for low resource languages. Currently, we have text files for each language sourced from different documents. The number of lines in the text files are the same. For example, for each document we have lang1.txt and lang2.txt each with n lines. Each line in lang1.txt maps to each line in lang2.txt. We currently have these text files in a Github repository.
Once we have completed this dataset curation (this will be actively on-going) we would like to upload to the Huggingface Hub. Essentially, we would like for this to be similar to glue with different configurations corresponding to different multi-lingual datasets.
When I was looking over the documentation page, I found several resources. One is the uploading to Huggingface Hub 1, another is creating a dataset loading script 2.
I’m confused about the function provided by these two methods. Also, I don’t if just uploading the raw text files that we have is enough to be able to work with.
Question: What is the easiest and most efficient way to upload the kind of dataset that I have?
Thank you.