loading ..<span style="color: #800080; text-decoration-color: #800080">/../../../../tests/integration_tests/examples/</span><span style="color: #ff00ff; text-decoration-color: #ff00ff">example-non-utf8.txt</span></pre>The file example-non-utf8.txt uses a different encoding, so the load() function fails with a helpful message indicating which file failed decoding.With the default behavior of TextLoader any failure to load any of the documents will fail the whole loading process and no documents are loaded.B. Silent fail‚ÄãWe can pass the parameter silent_errors to the DirectoryLoader to skip the files which could not be loaded and continue the load process.loader = DirectoryLoader(path, glob="**/*.txt", loader_cls=TextLoader, silent_errors=True)docs = loader.load()    Error loading ../../../../../tests/integration_tests/examples/example-non-utf8.txtdoc_sources = [doc.metadata['source']  for doc in docs]doc_sources    ['../../../../../tests/integration_tests/examples/whatsapp_chat.txt',     '../../../../../tests/integration_tests/examples/example-utf8.txt']C. Auto detect encodings‚ÄãWe can also ask TextLoader to auto detect the file encoding before failing, by passing the autodetect_encoding to the loader class.text_loader_kwargs={'autodetect_encoding': True}loader = DirectoryLoader(path, glob="**/*.txt", loader_cls=TextLoader, loader_kwargs=text_loader_kwargs)docs = loader.load()doc_sources = [doc.metadata['source']  for doc in docs]doc_sources    ['../../../../../tests/integration_tests/examples/example-non-utf8.txt',     '../../../../../tests/integration_tests/examples/whatsapp_chat.txt',     '../../../../../tests/integration_tests/examples/example-utf8.txt']PreviousCSVNextHTMLShow a progress barUse multithreadingChange loader classAuto-detect file encodings with TextLoaderA. Default BehaviorB. Silent failC. Auto detect encodingsCommunityDiscordTwitterGitHubPythonJS/TSMoreHomepageBlogCopyright ¬© 2023 LangChain, Inc.
This covers how to load all documents in a directory.
This covers how to load all documents in a directory. ->: loading ..<span style="color: #800080; text-decoration-color: #800080">/../../../../tests/integration_tests/examples/</span><span style="color: #ff00ff; text-decoration-color: #ff00ff">example-non-utf8.txt</span></pre>The file example-non-utf8.txt uses a different encoding, so the load() function fails with a helpful message indicating which file failed decoding.With the default behavior of TextLoader any failure to load any of the documents will fail the whole loading process and no documents are loaded.B. Silent fail‚ÄãWe can pass the parameter silent_errors to the DirectoryLoader to skip the files which could not be loaded and continue the load process.loader = DirectoryLoader(path, glob="**/*.txt", loader_cls=TextLoader, silent_errors=True)docs = loader.load()    Error loading ../../../../../tests/integration_tests/examples/example-non-utf8.txtdoc_sources = [doc.metadata['source']  for doc in docs]doc_sources    ['../../../../../tests/integration_tests/examples/whatsapp_chat.txt',     '../../../../../tests/integration_tests/examples/example-utf8.txt']C. Auto detect encodings‚ÄãWe can also ask TextLoader to auto detect the file encoding before failing, by passing the autodetect_encoding to the loader class.text_loader_kwargs={'autodetect_encoding': True}loader = DirectoryLoader(path, glob="**/*.txt", loader_cls=TextLoader, loader_kwargs=text_loader_kwargs)docs = loader.load()doc_sources = [doc.metadata['source']  for doc in docs]doc_sources    ['../../../../../tests/integration_tests/examples/example-non-utf8.txt',     '../../../../../tests/integration_tests/examples/whatsapp_chat.txt',     '../../../../../tests/integration_tests/examples/example-utf8.txt']PreviousCSVNextHTMLShow a progress barUse multithreadingChange loader classAuto-detect file encodings with TextLoaderA. Default BehaviorB. Silent failC. Auto detect encodingsCommunityDiscordTwitterGitHubPythonJS/TSMoreHomepageBlogCopyright ¬© 2023 LangChain, Inc.