maroxtn:

tokenizers


it seems that the Tokenizer library does not directly support loading from directory,
an alternative way is to write a wrapper method that looks for  tokenize.json in the directory first, then load it via  the from_file method of Tokenizer  :
from os.path import isfile, join
from tokenizers import Tokenizer

def load_tokenizer_from_dir(Tokenizer, your_dir):
    # check the existance of tokenizer file
    if isfile(join(your_dir, "tokenizer.json")):
        return Tokenizer.from_file("tok/tokenizer.json")
    else:
        raise ValueError("tokenier.json not exist in dir {}".format(your_dir))

the transformers library implement  the from_pretrained  method in similar way