0|     3|Andersson, Mr. An...|  male|39.0|    1|    5|          347082| 31.275| null|       S|    |         15|       0|     3|Vestrom, Miss. Hu...|female|14.0|    0|    0|          350406| 7.8542| null|       S|    |         16|       1|     2|Hewlett, Mrs. (Ma...|female|55.0|    0|    0|          248706|   16.0| null|       S|    |         17|       0|     3|Rice, Master. Eugene|  male| 2.0|    4|    1|          382652| 29.125| null|       Q|    |         18|       1|     2|Williams, Mr. Cha...|  male|null|    0|    0|          244373|   13.0| null|       S|    |         19|       0|     3|Vander Planke, Mr...|female|31.0|    1|    0|          345763|   18.0| null|       S|    |         20|       1|     3|Masselmani, Mrs. ...|female|null|    0|    0|            2649|  7.225| null|       C|    +-----------+--------+------+--------------------+------+----+-----+-----+----------------+-------+-----+--------+    only showing top 20 rows    from langchain.agents import create_spark_dataframe_agentfrom langchain.llms import OpenAIimport osos.environ["OPENAI_API_KEY"] = "...input your openai api key here..."agent = create_spark_dataframe_agent(llm=OpenAI(temperature=0), df=df, verbose=True)agent.run(    """who bought the most expensive ticket?You can find all supported function types in https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/dataframe.html""")            > Entering new AgentExecutor chain...        Thought: I need to find the row with the highest fare    Action: python_repl_ast    Action Input: df.sort(df.Fare.desc()).first()    Observation: Row(PassengerId=259, Survived=1, Pclass=1, Name='Ward, Miss. Anna', Sex='female', Age=35.0, SibSp=0, Parch=0, Ticket='PC 17755', Fare=512.3292, Cabin=None, Embarked='C')    Thought: I now know the name of the person who bought the most expensive ticket    Final Answer: Miss. Anna Ward        > Finished chain.    'Miss. Anna Ward'spark.stop()PreviousPythonNextSpark SQLSpark DataFrame exampleSpark
This notebook shows how to use agents to interact with a Spark DataFrame and Spark Connect. It is mostly optimized for question answering.
This notebook shows how to use agents to interact with a Spark DataFrame and Spark Connect. It is mostly optimized for question answering. ->: 0|     3|Andersson, Mr. An...|  male|39.0|    1|    5|          347082| 31.275| null|       S|    |         15|       0|     3|Vestrom, Miss. Hu...|female|14.0|    0|    0|          350406| 7.8542| null|       S|    |         16|       1|     2|Hewlett, Mrs. (Ma...|female|55.0|    0|    0|          248706|   16.0| null|       S|    |         17|       0|     3|Rice, Master. Eugene|  male| 2.0|    4|    1|          382652| 29.125| null|       Q|    |         18|       1|     2|Williams, Mr. Cha...|  male|null|    0|    0|          244373|   13.0| null|       S|    |         19|       0|     3|Vander Planke, Mr...|female|31.0|    1|    0|          345763|   18.0| null|       S|    |         20|       1|     3|Masselmani, Mrs. ...|female|null|    0|    0|            2649|  7.225| null|       C|    +-----------+--------+------+--------------------+------+----+-----+-----+----------------+-------+-----+--------+    only showing top 20 rows    from langchain.agents import create_spark_dataframe_agentfrom langchain.llms import OpenAIimport osos.environ["OPENAI_API_KEY"] = "...input your openai api key here..."agent = create_spark_dataframe_agent(llm=OpenAI(temperature=0), df=df, verbose=True)agent.run(    """who bought the most expensive ticket?You can find all supported function types in https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/dataframe.html""")            > Entering new AgentExecutor chain...        Thought: I need to find the row with the highest fare    Action: python_repl_ast    Action Input: df.sort(df.Fare.desc()).first()    Observation: Row(PassengerId=259, Survived=1, Pclass=1, Name='Ward, Miss. Anna', Sex='female', Age=35.0, SibSp=0, Parch=0, Ticket='PC 17755', Fare=512.3292, Cabin=None, Embarked='C')    Thought: I now know the name of the person who bought the most expensive ticket    Final Answer: Miss. Anna Ward        > Finished chain.    'Miss. Anna Ward'spark.stop()PreviousPythonNextSpark SQLSpark DataFrame exampleSpark