Skip to main contentü¶úÔ∏èüîó LangChainDocsUse casesIntegrationsAPICommunityChat our docsLangSmithJS/TS DocsSearchCTRLKProvidersAnthropicAWSGoogleMicrosoftOpenAIMoreComponentsLLMsChat modelsDocument loadersDocument transformersText embedding modelsVector storesRetrieversToolsAgents and toolkitsAINetworkAirbyte Question AnsweringAmadeusAzure Cognitive ServicesClickUp Langchain ToolkitCSVDocument ComparisonGithubGitlabGmailGoogle Drive toolJiraJSONMultiOnOffice365OpenAPINatural Language APIsPandas DataframePlayWright BrowserPowerBI DatasetPythonSpark DataframeSpark SQLSQL DatabaseVectorstoreXorbitsMemoryCallbacksChat loadersComponentsAgents and toolkitsSpark SQLOn this pageSpark SQLThis notebook shows how to use agents to interact with Spark SQL. Similar to SQL Database Agent, it is designed to address general inquiries about Spark SQL and facilitate error recovery.NOTE: Note that, as this agent is in active development, all answers might not be correct. Additionally, it is not guaranteed that the agent won't perform DML statements on your Spark cluster given certain questions. Be careful running it on sensitive data!Initialization‚Äãfrom langchain.agents import create_spark_sql_agentfrom langchain.agents.agent_toolkits import SparkSQLToolkitfrom langchain.chat_models import ChatOpenAIfrom langchain.utilities.spark_sql import SparkSQLfrom pyspark.sql import SparkSessionspark = SparkSession.builder.getOrCreate()schema = "langchain_example"spark.sql(f"CREATE DATABASE IF NOT EXISTS {schema}")spark.sql(f"USE {schema}")csv_file_path = "titanic.csv"table = "titanic"spark.read.csv(csv_file_path, header=True, inferSchema=True).write.saveAsTable(table)spark.table(table).show()    Setting default log level to "WARN".    To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).    23/05/18 16:03:10 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
This notebook shows how to use agents to interact with Spark SQL. Similar to SQL Database Agent, it is designed to address general inquiries about Spark SQL and facilitate error recovery.
This notebook shows how to use agents to interact with Spark SQL. Similar to SQL Database Agent, it is designed to address general inquiries about Spark SQL and facilitate error recovery. ->: Skip to main contentü¶úÔ∏èüîó LangChainDocsUse casesIntegrationsAPICommunityChat our docsLangSmithJS/TS DocsSearchCTRLKProvidersAnthropicAWSGoogleMicrosoftOpenAIMoreComponentsLLMsChat modelsDocument loadersDocument transformersText embedding modelsVector storesRetrieversToolsAgents and toolkitsAINetworkAirbyte Question AnsweringAmadeusAzure Cognitive ServicesClickUp Langchain ToolkitCSVDocument ComparisonGithubGitlabGmailGoogle Drive toolJiraJSONMultiOnOffice365OpenAPINatural Language APIsPandas DataframePlayWright BrowserPowerBI DatasetPythonSpark DataframeSpark SQLSQL DatabaseVectorstoreXorbitsMemoryCallbacksChat loadersComponentsAgents and toolkitsSpark SQLOn this pageSpark SQLThis notebook shows how to use agents to interact with Spark SQL. Similar to SQL Database Agent, it is designed to address general inquiries about Spark SQL and facilitate error recovery.NOTE: Note that, as this agent is in active development, all answers might not be correct. Additionally, it is not guaranteed that the agent won't perform DML statements on your Spark cluster given certain questions. Be careful running it on sensitive data!Initialization‚Äãfrom langchain.agents import create_spark_sql_agentfrom langchain.agents.agent_toolkits import SparkSQLToolkitfrom langchain.chat_models import ChatOpenAIfrom langchain.utilities.spark_sql import SparkSQLfrom pyspark.sql import SparkSessionspark = SparkSession.builder.getOrCreate()schema = "langchain_example"spark.sql(f"CREATE DATABASE IF NOT EXISTS {schema}")spark.sql(f"USE {schema}")csv_file_path = "titanic.csv"table = "titanic"spark.read.csv(csv_file_path, header=True, inferSchema=True).write.saveAsTable(table)spark.table(table).show()    Setting default log level to "WARN".    To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).    23/05/18 16:03:10 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable