Skip to main contentü¶úÔ∏èüîó LangChainDocsUse casesIntegrationsAPICommunityChat our docsLangSmithJS/TS DocsSearchCTRLKProvidersAnthropicAWSGoogleMicrosoftOpenAIMoreComponentsLLMsChat modelsDocument loadersDocument transformersText embedding modelsVector storesRetrieversToolsAgents and toolkitsAINetworkAirbyte Question AnsweringAmadeusAzure Cognitive ServicesClickUp Langchain ToolkitCSVDocument ComparisonGithubGitlabGmailGoogle Drive toolJiraJSONMultiOnOffice365OpenAPINatural Language APIsPandas DataframePlayWright BrowserPowerBI DatasetPythonSpark DataframeSpark SQLSQL DatabaseVectorstoreXorbitsMemoryCallbacksChat loadersComponentsAgents and toolkitsSpark DataframeOn this pageSpark DataframeThis notebook shows how to use agents to interact with a Spark DataFrame and Spark Connect. It is mostly optimized for question answering.NOTE: this agent calls the Python agent under the hood, which executes LLM generated Python code - this can be bad if the LLM generated Python code is harmful. Use cautiously.import osos.environ["OPENAI_API_KEY"] = "...input your openai api key here..."Spark DataFrame example‚Äãfrom langchain.llms import OpenAIfrom pyspark.sql import SparkSessionfrom langchain.agents import create_spark_dataframe_agentspark = SparkSession.builder.getOrCreate()csv_file_path = "titanic.csv"df = spark.read.csv(csv_file_path, header=True, inferSchema=True)df.show()    23/05/15 20:33:10 WARN Utils: Your hostname, Mikes-Mac-mini.local resolves to a loopback address: 127.0.0.1; using 192.168.68.115 instead (on interface en1)    23/05/15 20:33:10 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address    Setting default log level to "WARN".    To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).    23/05/15 20:33:10 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
This notebook shows how to use agents to interact with a Spark DataFrame and Spark Connect. It is mostly optimized for question answering.
This notebook shows how to use agents to interact with a Spark DataFrame and Spark Connect. It is mostly optimized for question answering. ->: Skip to main contentü¶úÔ∏èüîó LangChainDocsUse casesIntegrationsAPICommunityChat our docsLangSmithJS/TS DocsSearchCTRLKProvidersAnthropicAWSGoogleMicrosoftOpenAIMoreComponentsLLMsChat modelsDocument loadersDocument transformersText embedding modelsVector storesRetrieversToolsAgents and toolkitsAINetworkAirbyte Question AnsweringAmadeusAzure Cognitive ServicesClickUp Langchain ToolkitCSVDocument ComparisonGithubGitlabGmailGoogle Drive toolJiraJSONMultiOnOffice365OpenAPINatural Language APIsPandas DataframePlayWright BrowserPowerBI DatasetPythonSpark DataframeSpark SQLSQL DatabaseVectorstoreXorbitsMemoryCallbacksChat loadersComponentsAgents and toolkitsSpark DataframeOn this pageSpark DataframeThis notebook shows how to use agents to interact with a Spark DataFrame and Spark Connect. It is mostly optimized for question answering.NOTE: this agent calls the Python agent under the hood, which executes LLM generated Python code - this can be bad if the LLM generated Python code is harmful. Use cautiously.import osos.environ["OPENAI_API_KEY"] = "...input your openai api key here..."Spark DataFrame example‚Äãfrom langchain.llms import OpenAIfrom pyspark.sql import SparkSessionfrom langchain.agents import create_spark_dataframe_agentspark = SparkSession.builder.getOrCreate()csv_file_path = "titanic.csv"df = spark.read.csv(csv_file_path, header=True, inferSchema=True)df.show()    23/05/15 20:33:10 WARN Utils: Your hostname, Mikes-Mac-mini.local resolves to a loopback address: 127.0.0.1; using 192.168.68.115 instead (on interface en1)    23/05/15 20:33:10 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address    Setting default log level to "WARN".    To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).    23/05/15 20:33:10 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable