{
 "cells": [
  {
   "cell_type": "raw",
   "id": "e9437c8a-d8b7-4bf6-8ff4-54068a5a266c",
   "metadata": {},
   "source": [
    "---\n",
    "sidebar_position: 1.5\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d0df7646-b1e1-4014-a841-6dae9b3c50d9",
   "metadata": {},
   "source": [
    "# How to stream chat model responses\n",
    "\n",
    "\n",
    "All [chat models](https://api.python.langchain.com/en/latest/language_models/langchain_core.language_models.chat_models.BaseChatModel.html) implement the [Runnable interface](https://api.python.langchain.com/en/latest/runnables/langchain_core.runnables.base.Runnable.html#langchain_core.runnables.base.Runnable), which comes with a **default** implementations of standard runnable methods (i.e. `ainvoke`, `batch`, `abatch`, `stream`, `astream`, `astream_events`).\n",
    "\n",
    "The **default** streaming implementation provides an`Iterator` (or `AsyncIterator` for asynchronous streaming) that yields a single value: the final output from the underlying chat model provider.\n",
    "\n",
    ":::{.callout-tip}\n",
    "\n",
    "The **default** implementation does **not** provide support for token-by-token streaming, but it ensures that the the model can be swapped in for any other model as it supports the same standard interface.\n",
    "\n",
    ":::\n",
    "\n",
    "The ability to stream the output token-by-token depends on whether the provider has implemented proper streaming support.\n",
    "\n",
    "See which [integrations support token-by-token streaming here](/docs/integrations/chat/)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a76660e-7691-48b7-a2b4-2ccdff7875c3",
   "metadata": {},
   "source": [
    "## Sync streaming\n",
    "\n",
    "Below we use a `|` to help visualize the delimiter between tokens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "975c4f32-21f6-4a71-9091-f87b56347c33",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Here| is| a| |1| |verse| song| about| gol|dfish| on| the| moon|:|\n",
      "\n",
      "Floating| up| in| the| star|ry| night|,|\n",
      "Fins| a|-|gl|im|mer| in| the| pale| moon|light|.|\n",
      "Gol|dfish| swimming|,| peaceful| an|d free|,|\n",
      "Se|ren|ely| |drif|ting| across| the| lunar| sea|.|"
     ]
    }
   ],
   "source": [
    "from langchain_anthropic.chat_models import ChatAnthropic\n",
    "\n",
    "chat = ChatAnthropic(model=\"claude-3-haiku-20240307\")\n",
    "for chunk in chat.stream(\"Write me a 1 verse song about goldfish on the moon\"):\n",
    "    print(chunk.content, end=\"|\", flush=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5482d3a7-ee4f-40ba-b871-4d3f52603cd5",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Async Streaming"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "422f480c-df79-42e8-9bee-d0ebed31c557",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Here| is| a| |1| |verse| song| about| gol|dfish| on| the| moon|:|\n",
      "\n",
      "Floating| up| above| the| Earth|,|\n",
      "Gol|dfish| swim| in| alien| m|irth|.|\n",
      "In| their| bowl| of| lunar| dust|,|\n",
      "Gl|it|tering| scales| reflect| the| trust|\n",
      "Of| swimming| free| in| this| new| worl|d,|\n",
      "Where| their| aqu|atic| dream|'s| unf|ur|le|d.|"
     ]
    }
   ],
   "source": [
    "from langchain_anthropic.chat_models import ChatAnthropic\n",
    "\n",
    "chat = ChatAnthropic(model=\"claude-3-haiku-20240307\")\n",
    "async for chunk in chat.astream(\"Write me a 1 verse song about goldfish on the moon\"):\n",
    "    print(chunk.content, end=\"|\", flush=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c61e1309-3b6e-42fb-820a-2e4e3e6bc074",
   "metadata": {},
   "source": [
    "## Astream events\n",
    "\n",
    "Chat models also support the standard [astream events](https://api.python.langchain.com/en/latest/runnables/langchain_core.runnables.base.Runnable.html#langchain_core.runnables.base.Runnable.astream_events) method.\n",
    "\n",
    "This method is useful if you're streaming output from a larger LLM application that contains multiple steps (e.g., an LLM chain composed of a prompt, llm and parser)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "27bd1dfd-8ae2-49d6-b526-97180c81b5f4",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'event': 'on_chat_model_start', 'run_id': '08da631a-12a0-4f07-baee-fc9a175ad4ba', 'name': 'ChatAnthropic', 'tags': [], 'metadata': {}, 'data': {'input': 'Write me a 1 verse song about goldfish on the moon'}}\n",
      "{'event': 'on_chat_model_stream', 'run_id': '08da631a-12a0-4f07-baee-fc9a175ad4ba', 'tags': [], 'metadata': {}, 'name': 'ChatAnthropic', 'data': {'chunk': AIMessageChunk(content='Here', id='run-08da631a-12a0-4f07-baee-fc9a175ad4ba')}}\n",
      "{'event': 'on_chat_model_stream', 'run_id': '08da631a-12a0-4f07-baee-fc9a175ad4ba', 'tags': [], 'metadata': {}, 'name': 'ChatAnthropic', 'data': {'chunk': AIMessageChunk(content=\"'s\", id='run-08da631a-12a0-4f07-baee-fc9a175ad4ba')}}\n",
      "{'event': 'on_chat_model_stream', 'run_id': '08da631a-12a0-4f07-baee-fc9a175ad4ba', 'tags': [], 'metadata': {}, 'name': 'ChatAnthropic', 'data': {'chunk': AIMessageChunk(content=' a', id='run-08da631a-12a0-4f07-baee-fc9a175ad4ba')}}\n",
      "...Truncated\n"
     ]
    }
   ],
   "source": [
    "from langchain_anthropic.chat_models import ChatAnthropic\n",
    "\n",
    "chat = ChatAnthropic(model=\"claude-3-haiku-20240307\")\n",
    "idx = 0\n",
    "\n",
    "async for event in chat.astream_events(\n",
    "    \"Write me a 1 verse song about goldfish on the moon\", version=\"v1\"\n",
    "):\n",
    "    idx += 1\n",
    "    if idx >= 5:  # Truncate the output\n",
    "        print(\"...Truncated\")\n",
    "        break\n",
    "    print(event)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
