Hugging Face's logo

Spaces:
yhavinga
/
dutch-tokenizer-arena
Running

App Files Community
1
dutch-tokenizer-arena / vocab /gpt_nexo_20b /__init__.py
xu-song's picture
xu-song
add compression leaderboard
1b7fc74
raw
history blame contribute delete
114 Bytes
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("EleutherAI/gpt-neox-20b")