load returns a list, it will block until all documents are loaded. To have better control over this process, you can also you the lazy_load method which returns an iterator instead:docs_iterator = issues_loader.lazy_load()Keep in mind that by default the page content is empty and the metadata object contains all the information from the record. To create documents in a different, pass in a record_handler function when creating the loader:from langchain.docstore.document import Documentdef handle_record(record, id):    return Document(page_content=record.data["title"] + "\n" + (record.data["body"] or ""), metadata=record.data)issues_loader = AirbyteCDKLoader(source_class=SourceGithub, config=config, stream_name="issues", record_handler=handle_record)docs = issues_loader.load()Incremental loads‚ÄãSome streams allow incremental loading, this means the source keeps track of synced records and won't load them again. This is useful for sources that have a high volume of data and are updated frequently.To take advantage of this, store the last_state property of the loader and pass it in when creating the loader again. This will ensure that only new records are loaded.last_state = issues_loader.last_state # store safelyincremental_issue_loader = AirbyteCDKLoader(source_class=SourceGithub, config=config, stream_name="issues", state=last_state)new_docs = incremental_issue_loader.load()PreviousacreomNextAirbyte GongInstallationExampleIncremental loadsCommunityDiscordTwitterGitHubPythonJS/TSMoreHomepageBlogCopyright ¬© 2023 LangChain, Inc.
Airbyte is a data integration platform for ELT pipelines from APIs, databases & files to warehouses & lakes. It has the largest catalog of ELT connectors to data warehouses and databases.
Airbyte is a data integration platform for ELT pipelines from APIs, databases & files to warehouses & lakes. It has the largest catalog of ELT connectors to data warehouses and databases. ->: load returns a list, it will block until all documents are loaded. To have better control over this process, you can also you the lazy_load method which returns an iterator instead:docs_iterator = issues_loader.lazy_load()Keep in mind that by default the page content is empty and the metadata object contains all the information from the record. To create documents in a different, pass in a record_handler function when creating the loader:from langchain.docstore.document import Documentdef handle_record(record, id):    return Document(page_content=record.data["title"] + "\n" + (record.data["body"] or ""), metadata=record.data)issues_loader = AirbyteCDKLoader(source_class=SourceGithub, config=config, stream_name="issues", record_handler=handle_record)docs = issues_loader.load()Incremental loads‚ÄãSome streams allow incremental loading, this means the source keeps track of synced records and won't load them again. This is useful for sources that have a high volume of data and are updated frequently.To take advantage of this, store the last_state property of the loader and pass it in when creating the loader again. This will ensure that only new records are loaded.last_state = issues_loader.last_state # store safelyincremental_issue_loader = AirbyteCDKLoader(source_class=SourceGithub, config=config, stream_name="issues", state=last_state)new_docs = incremental_issue_loader.load()PreviousacreomNextAirbyte GongInstallationExampleIncremental loadsCommunityDiscordTwitterGitHubPythonJS/TSMoreHomepageBlogCopyright ¬© 2023 LangChain, Inc.