Hello everyone, I’ve been referencing this 4 paper on training transformer based models using metadata enhanced MIDI and was thinking about implementing this using the huggingface transformers and tokenizer libraries as an introduction to these libraries beyond the basic language modeling examples. As I’ve been researching and referencing this 4 tutorial I’ve ran into issues with tokenization and was wondering when training a tokenizer, how can I set up “word level” semantics? Technically each “word” in this case will be the data within a string like so ‘Event(name=Position, time=360, value=4/16, text=360)’ rather than just words and characters delimited on spaces like its doing now as listed below
 #version: 0.2 - Trained by huggingface/tokenizers
m e
    a l
    u e
    al ue
    i me
    n ame
    v alue
    Ġ value
    Ġt ex
    Ġt ime
    Ev en
    Ġtex t
    Even t
    ) ,
    Ġ Event
    N o
Apologies on if these questions are noobish I’m grokking a lot of this as I go along. Any help is greatly appreciated.