*   [](/v0.2/)
*   [How-to guides](/v0.2/docs/how_to/)
*   How to track token usage for LLMs

On this page

How to track token usage for LLMs
=================================

Tracking token usage to calculate cost is an important part of putting your app in production. This guide goes over how to obtain this information from your LangChain model calls.

Prerequisites

This guide assumes familiarity with the following concepts:

*   [LLMs](/v0.2/docs/concepts/#llms)

Using LangSmith[​](#using-langsmith "Direct link to Using LangSmith")
---------------------------------------------------------------------

You can use [LangSmith](https://www.langchain.com/langsmith) to help track token usage in your LLM application. See the [LangSmith quick start guide](https://docs.smith.langchain.com/).

Using callbacks[​](#using-callbacks "Direct link to Using callbacks")
---------------------------------------------------------------------

There are some API-specific callback context managers that allow you to track token usage across multiple calls. You'll need to check whether such an integration is available for your particular model.

If such an integration is not available for your model, you can create a custom callback manager by adapting the implementation of the [OpenAI callback manager](https://api.python.langchain.com/en/latest/_modules/langchain_community/callbacks/openai_info.html#OpenAICallbackHandler).

### OpenAI[​](#openai "Direct link to OpenAI")

Let's first look at an extremely simple example of tracking token usage for a single Chat model call.

danger

The callback handler does not currently support streaming token counts for legacy language models (e.g., `langchain_openai.OpenAI`). For support in a streaming context, refer to the corresponding guide for chat models [here](/v0.2/docs/how_to/chat_token_usage_tracking/).

### Single call[​](#single-call "Direct link to Single call")

    from langchain_community.callbacks import get_openai_callbackfrom langchain_openai import OpenAIllm = OpenAI(model_name="gpt-3.5-turbo-instruct")with get_openai_callback() as cb:    result = llm.invoke("Tell me a joke")    print(result)    print("---")print()print(f"Total Tokens: {cb.total_tokens}")print(f"Prompt Tokens: {cb.prompt_tokens}")print(f"Completion Tokens: {cb.completion_tokens}")print(f"Total Cost (USD): ${cb.total_cost}")

**API Reference:**[get\_openai\_callback](https://api.python.langchain.com/en/latest/callbacks/langchain_community.callbacks.manager.get_openai_callback.html) | [OpenAI](https://api.python.langchain.com/en/latest/llms/langchain_openai.llms.base.OpenAI.html)

    Why don't scientists trust atoms?Because they make up everything.---Total Tokens: 18Prompt Tokens: 4Completion Tokens: 14Total Cost (USD): $3.4e-05

### Multiple calls[​](#multiple-calls "Direct link to Multiple calls")

Anything inside the context manager will get tracked. Here's an example of using it to track multiple calls in sequence to a chain. This will also work for an agent which may use multiple steps.

    from langchain_community.callbacks import get_openai_callbackfrom langchain_core.prompts import PromptTemplatefrom langchain_openai import OpenAIllm = OpenAI(model_name="gpt-3.5-turbo-instruct")template = PromptTemplate.from_template("Tell me a joke about {topic}")chain = template | llmwith get_openai_callback() as cb:    response = chain.invoke({"topic": "birds"})    print(response)    response = chain.invoke({"topic": "fish"})    print("--")    print(response)print()print("---")print(f"Total Tokens: {cb.total_tokens}")print(f"Prompt Tokens: {cb.prompt_tokens}")print(f"Completion Tokens: {cb.completion_tokens}")print(f"Total Cost (USD): ${cb.total_cost}")

**API Reference:**[get\_openai\_callback](https://api.python.langchain.com/en/latest/callbacks/langchain_community.callbacks.manager.get_openai_callback.html) | [PromptTemplate](https://api.python.langchain.com/en/latest/prompts/langchain_core.prompts.prompt.PromptTemplate.html) | [OpenAI](https://api.python.langchain.com/en/latest/llms/langchain_openai.llms.base.OpenAI.html)

    Why did the chicken go to the seance?To talk to the other side of the road!--Why did the fish need a lawyer?Because it got caught in a net!---Total Tokens: 50Prompt Tokens: 12Completion Tokens: 38Total Cost (USD): $9.400000000000001e-05

Streaming[​](#streaming "Direct link to Streaming")
---------------------------------------------------

danger

`get_openai_callback` does not currently support streaming token counts for legacy language models (e.g., `langchain_openai.OpenAI`). If you want to count tokens correctly in a streaming context, there are a number of options:

*   Use chat models as described in [this guide](/v0.2/docs/how_to/chat_token_usage_tracking/);
*   Implement a [custom callback handler](/v0.2/docs/how_to/custom_callbacks/) that uses appropriate tokenizers to count the tokens;
*   Use a monitoring platform such as [LangSmith](https://www.langchain.com/langsmith).

Note that when using legacy language models in a streaming context, token counts are not updated:

    from langchain_community.callbacks import get_openai_callbackfrom langchain_openai import OpenAIllm = OpenAI(model_name="gpt-3.5-turbo-instruct")with get_openai_callback() as cb:    for chunk in llm.stream("Tell me a joke"):        print(chunk, end="", flush=True)    print(result)    print("---")print()print(f"Total Tokens: {cb.total_tokens}")print(f"Prompt Tokens: {cb.prompt_tokens}")print(f"Completion Tokens: {cb.completion_tokens}")print(f"Total Cost (USD): ${cb.total_cost}")

**API Reference:**[get\_openai\_callback](https://api.python.langchain.com/en/latest/callbacks/langchain_community.callbacks.manager.get_openai_callback.html) | [OpenAI](https://api.python.langchain.com/en/latest/llms/langchain_openai.llms.base.OpenAI.html)

    Why don't scientists trust atoms?Because they make up everything!Why don't scientists trust atoms?Because they make up everything.---Total Tokens: 0Prompt Tokens: 0Completion Tokens: 0Total Cost (USD): $0.0

[Edit this page](https://github.com/langchain-ai/langchain/edit/master/docs/docs/how_to/llm_token_usage_tracking.ipynb)

* * *

#### Was this page helpful?

  

#### You can also leave detailed feedback [on GitHub](https://github.com/langchain-ai/langchain/issues/new?assignees=&labels=03+-+Documentation&projects=&template=documentation.yml&title=DOC%3A+%3CPlease+write+a+comprehensive+title+after+the+%27DOC%3A+%27+prefix%3E).

[

Previous

How to cache LLM responses

](/v0.2/docs/how_to/llm_caching/)[

Next

Run LLMs locally

](/v0.2/docs/how_to/local_llms/)

*   [Using LangSmith](#using-langsmith)
*   [Using callbacks](#using-callbacks)
    *   [OpenAI](#openai)
    *   [Single call](#single-call)
    *   [Multiple calls](#multiple-calls)
*   [Streaming](#streaming)