Hi.
I have an h5 file which consists of two datasets. One is for metadata (labels and etc) and one is for the actual data which is a 2d array for each element. From my experience working on datasets, HF’s dataset has a very good caching mechanism and is usually way faster than plain vanilla dataset I can code via pure python. I wanted to know how can I create a HF dataset from this hdf5 file I have, and are there any examples for custom datasets?
The dataset loading script template has parts about downloading the data and split generators, which I don’t think are related to my current task.
Additionally. I also have to split the data based on conditions of the label of the data, and not pure randomly. How can I achieve that?