Example Format of Data for token classification
Hi!! I'd like to train the token classification model but I don't know what is the right format of data for token classification training. Thank you.
Hi @Michael95-m ,

the dataset format for e.g. the normal NER task is relatively simple: one token - label pair per line and an empty line specified a new sentence.

So here's a good example from Spanish CoNLL dataset for NER:

```bash
Melbourne B-LOC
( O
Australia B-LOC
) O
, O
25 O
may O
( O
EFE B-ORG
) O
. O

- O

El O
Abogado B-PER
General I-PER
del I-PER
Estado I-PER
, O
Daryl B-PER
Williams I-PER
, O
subrayó O
hoy O
la O
necesidad O
de O
tomar O
medidas O
para O
proteger O
al O
sistema O
judicial O
australiano O
frente O
a O
una O
página O
de O
internet O
que O
imposibilita O
el O
cumplimiento O
de O
los O
principios O
básicos O
de O
la O
Ley B-MISC
. O

```

Each line consists of token/word and its corresponding label, delimited by a space. An empty line denotes a new sentence.

Technically, the parsing of your input file is done here:

https://github.com/huggingface/transformers/blob/f5518e56318a79056ba3c80cbece29d9fe98558c/examples/token-classification/tasks.py#L18-L44

I hope this helps :)

<|||||>Thanks for your kind answer !!!