is a data integration platform for ELT pipelines from APIs, databases & files to warehouses & lakes. It has the largest catalog of ELT connectors to data warehouses and databases.A lot of source connectors are implemented using the Airbyte CDK. This loader allows to run any of these connectors and return the data as documents.Installation‚ÄãFirst, you need to install the airbyte-cdk python package.#!pip install airbyte-cdkThen, either install an existing connector from the Airbyte Github repository or create your own connector using the Airbyte CDK.For example, to install the Github connector, run#!pip install "source_github@git+https://github.com/airbytehq/airbyte.git@master#subdirectory=airbyte-integrations/connectors/source-github"Some sources are also published as regular packages on PyPIExample‚ÄãNow you can create an AirbyteCDKLoader based on the imported source. It takes a config object that's passed to the connector. You also have to pick the stream you want to retrieve records from by name (stream_name). Check the connectors documentation page and spec definition for more information on the config object and available streams. For the Github connectors these are:https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-github/source_github/spec.json.https://docs.airbyte.com/integrations/sources/github/from langchain.document_loaders.airbyte import AirbyteCDKLoaderfrom source_github.source import SourceGithub # plug in your own source hereconfig = {    # your github configuration    "credentials": {        "api_url": "api.github.com",        "personal_access_token": "<token>"    },    "repository": "<repo>",    "start_date": "<date from which to start retrieving records from in ISO format, e.g. 2020-10-20T00:00:00Z>"}issues_loader = AirbyteCDKLoader(source_class=SourceGithub, config=config, stream_name="issues")Now you can load documents the usual waydocs = issues_loader.load()As load returns a list, it will block until all
Airbyte is a data integration platform for ELT pipelines from APIs, databases & files to warehouses & lakes. It has the largest catalog of ELT connectors to data warehouses and databases.
Airbyte is a data integration platform for ELT pipelines from APIs, databases & files to warehouses & lakes. It has the largest catalog of ELT connectors to data warehouses and databases. ->: is a data integration platform for ELT pipelines from APIs, databases & files to warehouses & lakes. It has the largest catalog of ELT connectors to data warehouses and databases.A lot of source connectors are implemented using the Airbyte CDK. This loader allows to run any of these connectors and return the data as documents.Installation‚ÄãFirst, you need to install the airbyte-cdk python package.#!pip install airbyte-cdkThen, either install an existing connector from the Airbyte Github repository or create your own connector using the Airbyte CDK.For example, to install the Github connector, run#!pip install "source_github@git+https://github.com/airbytehq/airbyte.git@master#subdirectory=airbyte-integrations/connectors/source-github"Some sources are also published as regular packages on PyPIExample‚ÄãNow you can create an AirbyteCDKLoader based on the imported source. It takes a config object that's passed to the connector. You also have to pick the stream you want to retrieve records from by name (stream_name). Check the connectors documentation page and spec definition for more information on the config object and available streams. For the Github connectors these are:https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-github/source_github/spec.json.https://docs.airbyte.com/integrations/sources/github/from langchain.document_loaders.airbyte import AirbyteCDKLoaderfrom source_github.source import SourceGithub # plug in your own source hereconfig = {    # your github configuration    "credentials": {        "api_url": "api.github.com",        "personal_access_token": "<token>"    },    "repository": "<repo>",    "start_date": "<date from which to start retrieving records from in ISO format, e.g. 2020-10-20T00:00:00Z>"}issues_loader = AirbyteCDKLoader(source_class=SourceGithub, config=config, stream_name="issues")Now you can load documents the usual waydocs = issues_loader.load()As load returns a list, it will block until all