Hey guys,
After dataset map transformation, I have a new dataset with the following features:
{‘data’: {‘is_random_next’: Value(dtype=‘bool’, id=None),
‘tokens_a’: Sequence(feature=Value(dtype=‘int64’, id=None), length=-1, id=None),
‘tokens_b’: Sequence(feature=Value(dtype=‘int64’, id=None), length=-1, id=None)}}
The data column is useless so after invoking flatten, I get:
{‘data.is_random_next’: Value(dtype=‘bool’, id=None),
‘data.tokens_a’: Sequence(feature=Value(dtype=‘int64’, id=None), length=-1, id=None),
‘data.tokens_b’: Sequence(feature=Value(dtype=‘int64’, id=None), length=-1, id=None)}
which is exactly what I want. After renaming columns, I am done.
Since map forces us to return dict, I need to wrap a list of dict values (is_random_next, tokens_a, tokens_b) with dict to comply, so I did. How could I avoid that?