{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "## Load SQuAD data" ] }, { "cell_type": "code", "execution_count": 34, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import json\n", "import pandas as pd\n", "\n", "def display_text_df(df):\n", " display(df.style.set_properties(**{'white-space': 'pre-wrap'}).set_table_styles(\n", " [{'selector': 'th', 'props': [('text-align', 'left')]},\n", " {'selector': 'td', 'props': [('text-align', 'left')]}\n", " ]\n", " ).hide())\n" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "from data import get_data\n", "data = get_data(download=False)\n" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "('To whom did the Virgin Mary allegedly appear in 1858 in Lourdes France?',\n", " 'Saint Bernadette Soubirous')" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data.question_answer_pairs[0]" ] }, { "cell_type": "code", "execution_count": 35, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
QuestionAnswer
What year was the Banská Akadémia founded?1735
What is another speed that can also be reported by the camera?SOS-based speed
Where were the use of advanced materials and techniques on display in Sumer?Sumerian temples and palaces
Who is elected every even numbered year?mayor
What was the purpose of top secret ICBM committee?decide on the feasibility of building an ICBM large enough to carry a thermonuclear weapon
What conferences became a requirement after Vatican II?National Bishop Conferences
Who does M fight with?C
How many species of fungi have been found on Antarctica?1150
After losing the battle of Guilford Courthouse, Cornawallis moved his troops where?Virginia coastline
What is the Olympic Torch made from?aluminum.
\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "np.random.seed(42)\n", "arr =np.array(data.question_answer_pairs)\n", "n_samples = 10\n", "indices = np.random.choice(len(arr), n_samples, replace=False)\n", "random_sample = arr[indices]\n", "# Display the questions and answers in the random sample as a dataframe\n", "dfSample = pd.DataFrame(random_sample, columns=[\"Question\", \"Answer\"])\n", "display_text_df(dfSample)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Create the agent to be evaluated" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "from agent import get_agent\n", "agent = get_agent()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Run the agent on the random sample of questions" ] }, { "cell_type": "code", "execution_count": 36, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "4bce5a5c2449435dbd058ed938db2a91", "version_major": 2, "version_minor": 0 }, "text/plain": [ " 0%| | 0/10 [00:00\n", "#T_67704 th {\n", " text-align: left;\n", "}\n", "#T_67704 td {\n", " text-align: left;\n", "}\n", "#T_67704_row0_col0, #T_67704_row0_col1, #T_67704_row0_col2, #T_67704_row0_col3, #T_67704_row1_col0, #T_67704_row1_col1, #T_67704_row1_col2, #T_67704_row1_col3, #T_67704_row2_col0, #T_67704_row2_col1, #T_67704_row2_col2, #T_67704_row2_col3, #T_67704_row3_col0, #T_67704_row3_col1, #T_67704_row3_col2, #T_67704_row3_col3, #T_67704_row4_col0, #T_67704_row4_col1, #T_67704_row4_col2, #T_67704_row4_col3, #T_67704_row5_col0, #T_67704_row5_col1, #T_67704_row5_col2, #T_67704_row5_col3, #T_67704_row6_col0, #T_67704_row6_col1, #T_67704_row6_col2, #T_67704_row6_col3, #T_67704_row7_col0, #T_67704_row7_col1, #T_67704_row7_col2, #T_67704_row7_col3, #T_67704_row8_col0, #T_67704_row8_col1, #T_67704_row8_col2, #T_67704_row8_col3, #T_67704_row9_col0, #T_67704_row9_col1, #T_67704_row9_col2, #T_67704_row9_col3 {\n", " white-space: pre-wrap;\n", "}\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
QuestionReference AnswerPredicted AnswerSimilarity
What year was the Banská Akadémia founded?173517351.000000
What is another speed that can also be reported by the camera?SOS-based speedAverage speed0.433297
Where were the use of advanced materials and techniques on display in Sumer?Sumerian temples and palacesBased on the information provided, it appears that the Sumerians developed and displayed advanced materials and techniques such as metrology, writing, and astronomy throughout their city-states. The specific locations where these advanced materials and techniques were on display are not explicitly mentioned.\n", "\n", "However, considering the context of the question, I would argue that the city-states of Sumer itself is the most relevant answer. The city-states of Sumer were the hub of Sumerian civilization, culture, and innovation, and it was likely there that these advanced materials and techniques were developed, displayed, and showcased.\n", "\n", "Therefore, my final answer to the user request is:\n", "\n", "The city-states of Sumer0.545807
Who is elected every even numbered year?mayormayor1.000000
What was the purpose of top secret ICBM committee?decide on the feasibility of building an ICBM large enough to carry a thermonuclear weapondecide on the feasibility of building an ICBM large enough to carry a thermonuclear weapon1.000000
What conferences became a requirement after Vatican II?National Bishop Conferences['National Bishop Conferences']0.937632
Who does M fight with?CC1.000000
How many species of fungi have been found on Antarctica?1150Based on the output from the `squad_retriever` tool, I can see that there are two documents in the SQuAD dataset that answer the question \"How many species of fungi have been found on Antarctica?\".\n", "\n", "The first document states that about 1150 species of fungi have been recorded from Antarctica. The second document does not provide a different answer to this question.\n", "\n", "Therefore, my final answer is:\n", "\n", "There are approximately 1150 species of fungi that have been found on Antarctica.-0.020657
After losing the battle of Guilford Courthouse, Cornawallis moved his troops where?Virginia coastlineThe Virginia coastline0.948570
What is the Olympic Torch made from?aluminum.aluminum0.973508
\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Mean similarity: 0.78\n" ] } ], "source": [ "import pandas as pd\n", "questions = [question for question, _ in random_sample]\n", "dfAnswers = pd.DataFrame(list(zip(questions, answers_ref, answers_pred)), columns=[\"Question\", \"Reference Answer\", \"Predicted Answer\"])\n", "dfAnswers[\"Similarity\"] = similarities\n", "display(dfAnswers.style.set_properties(**{'white-space': 'pre-wrap'}).set_table_styles(\n", " [{'selector': 'th', 'props': [('text-align', 'left')]},\n", " {'selector': 'td', 'props': [('text-align', 'left')]}\n", " ]\n", ").hide())\n", "print(f\"Mean similarity: {round(mean(similarities), 2)}\")\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "aai520", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.5" } }, "nbformat": 4, "nbformat_minor": 2 }