0|     3|Andersson, Mr. An...|  male|39.0|    1|    5|          347082| 31.275| null|       S|    |         15|       0|     3|Vestrom, Miss. Hu...|female|14.0|    0|    0|          350406| 7.8542| null|       S|    |         16|       1|     2|Hewlett, Mrs. (Ma...|female|55.0|    0|    0|          248706|   16.0| null|       S|    |         17|       0|     3|Rice, Master. Eugene|  male| 2.0|    4|    1|          382652| 29.125| null|       Q|    |         18|       1|     2|Williams, Mr. Cha...|  male|null|    0|    0|          244373|   13.0| null|       S|    |         19|       0|     3|Vander Planke, Mr...|female|31.0|    1|    0|          345763|   18.0| null|       S|    |         20|       1|     3|Masselmani, Mrs. ...|female|null|    0|    0|            2649|  7.225| null|       C|    +-----------+--------+------+--------------------+------+----+-----+-----+----------------+-------+-----+--------+    only showing top 20 rows    agent = create_spark_dataframe_agent(llm=OpenAI(temperature=0), df=df, verbose=True)agent.run("how many rows are there?")            > Entering new AgentExecutor chain...    Thought: I need to find out how many rows are in the dataframe    Action: python_repl_ast    Action Input: df.count()    Observation: 891    Thought: I now know the final answer    Final Answer: There are 891 rows in the dataframe.        > Finished chain.    'There are 891 rows in the dataframe.'agent.run("how many people have more than 3 siblings")            > Entering new AgentExecutor chain...    Thought: I need to find out how many people have more than 3 siblings    Action: python_repl_ast    Action Input: df.filter(df.SibSp > 3).count()    Observation: 30    Thought: I now know the final answer    Final Answer: 30 people have more than 3 siblings.        > Finished chain.    '30 people have more than 3 siblings.'agent.run("whats the square root of the average age?")            > Entering new AgentExecutor chain...    Thought: I need to get the
This notebook shows how to use agents to interact with a Spark DataFrame and Spark Connect. It is mostly optimized for question answering.
This notebook shows how to use agents to interact with a Spark DataFrame and Spark Connect. It is mostly optimized for question answering. ->: 0|     3|Andersson, Mr. An...|  male|39.0|    1|    5|          347082| 31.275| null|       S|    |         15|       0|     3|Vestrom, Miss. Hu...|female|14.0|    0|    0|          350406| 7.8542| null|       S|    |         16|       1|     2|Hewlett, Mrs. (Ma...|female|55.0|    0|    0|          248706|   16.0| null|       S|    |         17|       0|     3|Rice, Master. Eugene|  male| 2.0|    4|    1|          382652| 29.125| null|       Q|    |         18|       1|     2|Williams, Mr. Cha...|  male|null|    0|    0|          244373|   13.0| null|       S|    |         19|       0|     3|Vander Planke, Mr...|female|31.0|    1|    0|          345763|   18.0| null|       S|    |         20|       1|     3|Masselmani, Mrs. ...|female|null|    0|    0|            2649|  7.225| null|       C|    +-----------+--------+------+--------------------+------+----+-----+-----+----------------+-------+-----+--------+    only showing top 20 rows    agent = create_spark_dataframe_agent(llm=OpenAI(temperature=0), df=df, verbose=True)agent.run("how many rows are there?")            > Entering new AgentExecutor chain...    Thought: I need to find out how many rows are in the dataframe    Action: python_repl_ast    Action Input: df.count()    Observation: 891    Thought: I now know the final answer    Final Answer: There are 891 rows in the dataframe.        > Finished chain.    'There are 891 rows in the dataframe.'agent.run("how many people have more than 3 siblings")            > Entering new AgentExecutor chain...    Thought: I need to find out how many people have more than 3 siblings    Action: python_repl_ast    Action Input: df.filter(df.SibSp > 3).count()    Observation: 30    Thought: I now know the final answer    Final Answer: 30 people have more than 3 siblings.        > Finished chain.    '30 people have more than 3 siblings.'agent.run("whats the square root of the average age?")            > Entering new AgentExecutor chain...    Thought: I need to get the