p50k_edit
An edit of the p50k_base tokenizer (adapted from openai/tiktoken).
Example usage:
Transformers/Tokenizers
from transformers import GPT2TokenizerFast
tokenizer = GPT2TokenizerFast.from_pretrained('nurdy/p50k_edit')
assert tokenizer.encode('hello world') == [31373, 995]
Inference Providers
NEW
This model isn't deployed by any Inference Provider.
🙋
Ask for provider support
HF Inference deployability: The model has no pipeline_tag.