It seems that a single dataset can be split up into different partitions but in such a way that the connection between them is still clear (by using a DatasetDict), which is neat. I am having difficulties trying to figure out how I can create them, and use them, though. I’ve been going through the documentation [1] 20,[2] 14 and the source code [1] 8,[2] 5 but it hasn’t become any clearer.
In some parts you speak of only a train, test split 53 other times you include validation 12. It is not clear how to split an existing Dataset into a DatasetDict containing train, dev (validation), and test keys, even though I think this is a common scenario.
The focus of the library seems to be on the use of existing datasets (which is a great feature!), but time permitting I would like to see more information and use cases for creating and managing your own datasets. I’m usually pretty good at figuring out a code base by its documentation and source code (e.g. transformers), but dataset seems to give me a harder time. That is not to say that I think it is bad, of course. But it does show, I think, that even though it seems simple from the examples, it is actually quite complex behind the scenes. For people who want do custom stuff it is then super important to have clear, verbose documentation about as many aspects of the library as possible.