I am trying to load a custom dataset locally.  This is a test dataset, will be revised soon, and will probably never be public so we would not want to put it on the HF Hub,
The dataset is in the same format as Conll2003.  The idea is to train Bert on conll2003+the custom dataset.
The setup I am testing (I am open to changes) is to use a folder under the project folder called “ADPConll” with all the data files (just like the Conll2003 folder in git datasets) in it like so:
MainProjectFolder
ADPConll
ADPConll.py                    ← copy of conll2003.oy with minor changes (see below)
ADPConll.py.lock             ← HF internal file
ADPConll.json                 ← train dataset in the format you see in the HF data browser.
ADPConll_test.json         ← test dataset
ADPConll_valid.json       ← validation dataset
Now the question comes as to how to set up the details.  I think the key is in the ADPConll.py script where I have the following:
_URL = “./ADPConll/”
_TRAINING_FILE = “ADPConll.json”
_DEV_FILE = “ADPConll_valid.txt”
_TEST_FILE = “ADPConll_test.txt”
I have tried several different settings but I am trying to cut short the trail/error approach and see if we can get this documented.
Note that the format of the files matches what you see in the HF Streamlit data explorer:
Streamlit 4
Ideally we would train on a combination of the 2 datasets but we would add new labels to our custom dataset so it can recognize additional entities.