for OpenAI LLMs.You can also use it to count tokens when splitting documents with from langchain.text_splitter import CharacterTextSplitterCharacterTextSplitter.from_tiktoken_encoder(...)For a more detailed walkthrough of this, see this notebookDocument Loader‚ÄãSee a usage example.from langchain.document_loaders.chatgpt import ChatGPTLoaderRetriever‚ÄãSee a usage example.from langchain.retrievers import ChatGPTPluginRetrieverChain‚ÄãSee a usage example.from langchain.chains import OpenAIModerationChainPreviousMicrosoftNextActiveloop Deep LakeInstallation and SetupLLMChat modelText Embedding ModelTokenizerDocument LoaderRetrieverChainCommunityDiscordTwitterGitHubPythonJS/TSMoreHomepageBlogCopyright ¬© 2023 LangChain, Inc.
All functionality related to OpenAI
All functionality related to OpenAI ->: for OpenAI LLMs.You can also use it to count tokens when splitting documents with from langchain.text_splitter import CharacterTextSplitterCharacterTextSplitter.from_tiktoken_encoder(...)For a more detailed walkthrough of this, see this notebookDocument Loader‚ÄãSee a usage example.from langchain.document_loaders.chatgpt import ChatGPTLoaderRetriever‚ÄãSee a usage example.from langchain.retrievers import ChatGPTPluginRetrieverChain‚ÄãSee a usage example.from langchain.chains import OpenAIModerationChainPreviousMicrosoftNextActiveloop Deep LakeInstallation and SetupLLMChat modelText Embedding ModelTokenizerDocument LoaderRetrieverChainCommunityDiscordTwitterGitHubPythonJS/TSMoreHomepageBlogCopyright ¬© 2023 LangChain, Inc.