*   [](/v0.2/)
*   [How-to guides](/v0.2/docs/how_to/)
*   How to split text by tokens

On this page

How to split text by tokens
===========================

Prerequisites

This guide assumes familiarity with the following concepts:

*   [Text splitters](/v0.2/docs/concepts#text-splitters)

Language models have a token limit. You should not exceed the token limit. When you split your text into chunks it is therefore a good idea to count the number of tokens. There are many tokenizers. When you count tokens in your text you should use the same tokenizer as used in the language model.

`js-tiktoken`[​](#js-tiktoken "Direct link to js-tiktoken")
-----------------------------------------------------------

note

[js-tiktoken](https://github.com/openai/js-tiktoken) is a JavaScript version of the `BPE` tokenizer created by OpenAI.

We can use `js-tiktoken` to estimate tokens used. It is tuned to OpenAI models.

1.  How the text is split: by character passed in.
2.  How the chunk size is measured: by the `js-tiktoken` tokenizer.

You can use the [`TokenTextSplitter`](https://v02.api.js.langchain.com/classes/langchain_textsplitters.TokenTextSplitter.html) like this:

    import { TokenTextSplitter } from "@langchain/textsplitters";import * as fs from "node:fs";// Load an example documentconst rawData = await fs.readFileSync(  "../../../../examples/state_of_the_union.txt");const stateOfTheUnion = rawData.toString();const textSplitter = new TokenTextSplitter({  chunkSize: 10,  chunkOverlap: 0,});const texts = await textSplitter.splitText(stateOfTheUnion);console.log(texts[0]);

    Madam Speaker, Madam Vice President, our

**Note:** Some written languages (e.g. Chinese and Japanese) have characters which encode to 2 or more tokens. Using the `TokenTextSplitter` directly can split the tokens for a character between two chunks causing malformed Unicode characters.

Next steps[​](#next-steps "Direct link to Next steps")
------------------------------------------------------

You’ve now learned a method for splitting text based on token count.

Next, check out the [full tutorial on retrieval-augmented generation](/v0.2/docs/tutorials/rag).

* * *

#### Was this page helpful?

  

#### You can also leave detailed feedback [on GitHub](https://github.com/langchain-ai/langchainjs/issues/new?assignees=&labels=03+-+Documentation&projects=&template=documentation.yml&title=DOC%3A+%3CPlease+write+a+comprehensive+title+after+the+%27DOC%3A+%27+prefix%3E).

[

Previous

How to chain runnables

](/v0.2/docs/how_to/sequence)[

Next

How to deal with large databases

](/v0.2/docs/how_to/sql_large_db)

*   [`js-tiktoken`](#js-tiktoken)
*   [Next steps](#next-steps)