Upload folder using huggingface_hub

README.md

@@ -12,6 +12,8 @@ python_version: 3.11.9
 
 The project is built using Transformers Agents 2.0, and uses the Stanford SQuAD dataset for training. The chatbot is designed to answer questions about the dataset, while also incorporating conversational context and various tools to provide a more natural and engaging conversational experience.
 
+At the time of writing, the project is available on [Hugging Face Spaces](https://huggingface.co/spaces/kaiokendall/SQuAD_Agent_Experiment).
+
 ## Getting Started
 
 1. Install dependencies:
@@ -23,13 +25,16 @@ pip install -r pre-requirements.txt
 pip install -r requirements.txt
 ```
 
-1. Set up required keys:
+2. Set up required keys:
+
+Create a `.env` file and set the following environment variables:
 
 ```bash
 HF_TOKEN=<your token>
+OPENAI_API_KEY=<your key>
 ```
 
-1. Run the app:
+3. Run the app:
 
 ```bash
 python app.py
@@ -39,37 +44,37 @@ python app.py
 
 1. SQuAD Dataset: The dataset used for training the chatbot is the Stanford SQuAD dataset, which contains over 100,000 questions and answers extracted from 500+ articles.
 2. RAG: RAG is a technique used to improve the accuracy of chatbots by using a custom knowledge base. In this project, the Stanford SQuAD dataset is used as the knowledge base.
-3. Llama 3.1: Llama 3.1 is a large language model used to generate responses to user questions. It is used in this project to generate responses to user questions, while also incorporating conversational context.
-4. Transformers Agents 2.0: Transformers Agents 2.0 is a framework for building conversational AI systems. It is used in this project to build the chatbot.
-5. Created a SquadRetrieverTool to integrate a fine-tuned BERT model into the agent, along with a TextToImageTool for a playful way to engage with the question-answering agent.
+3. Transformers Agents 2.0: Transformers Agents 2.0 is a framework for building conversational AI systems. It is used in this project to build the chatbot.
+4. Created a SquadRetrieverTool to integrate a fine-tuned BERT model into the agent, along with a TextToImageTool for a playful way to engage with the question-answering agent.
+5. Gradio: Gradio is used to create the chatbot interface, in `app.py`.
 
 ## Evaluation
 
-* [Agent Reasoning Benchmark](https://github.com/aymeric-roucher/agent_reasoning_benchmark)
-* [Hugging Face Blog: Open Source LLMs as Agents](https://huggingface.co/blog/open-source-llms-as-agents)
-* [Benchmarking Transformers Agents](https://github.com/aymeric-roucher/agent_reasoning_benchmark/blob/main/benchmark_transformers_agents.ipynb)
+SemScore is used in this project to evaluate the chatbot's responses in the notebook `benchmarking.ipynb`.
 
-## Results
+See [SemScore: Automated Evaluation of Instruction-Tuned LLMs based on Semantic Textual Similarity](https://doi.org/10.48550/arXiv.2401.17072)
 
-TBD
+In this experiment, the agent is evaluated with 3 different system prompting approaches:
 
-## Limitations
+1. The default prompting approach, which is just the default system prompt used in Hugging Face Transformers Agents 2.0, with only an example of using the `squad_retriever` tool added.
+2. A succinct prompting approach, which guides the agent to be concise if possible while still answering the question.
+3. A focused prompting approach, which reframes the entire chatbots purpose to focus more on the specific task of answering questions about the SQuAD dataset, while still being open to exploring other topics.
+
+## Results
 
-TBD
 
-## Related Research
 
-* [Retro: A Generalist Agent for Science](https://arxiv.org/abs/2112.04426)
-* [RETRO-pytorch](https://github.com/lucidrains/RETRO-pytorch)
-* [Why isn't Retro mainstream? State-of-the-art within reach](https://www.reddit.com/r/MachineLearning/comments/1cffgkt/d_why_isnt_retro_mainstream_stateoftheart_within/)
+## Limitations
 
-TBD
+* This experiment is not designed for multiple users. While it has in-session memory, simply refreshing the browser will reset the chat history, which is convenient for experimentation.
+* Some of the agent's underlying engines, models, and tools use keys that have usage limits, so the app may not work if those limits have been reached.
+  * It is recommended to clone the repo and run the code using your own keys, to avoid running into those limits.
 
 ## Acknowledgments
 
-* [Agents 2.0](https://github.com/huggingface/transformers/tree/main/src/transformers/agents)
+* [Hugging Face Transformers Agents 2.0](https://huggingface.co/docs/transformers/en/main_classes/agent)
 * [SemScore: Automated Evaluation of Instruction-Tuned LLMs based on Semantic Textual Similarity](https://arxiv.org/abs/2401.17072)
+* `semscore.py` from [geronimi73/semscore](https://github.com/geronimi73/semscore/blob/main/semscore.py)
 * [SemScore](https://huggingface.co/blog/g-ronimo/semscore)
 * [Stanford SQuAD](https://rajpurkar.github.io/SQuAD-explorer/)
-* [llama 3.1](https://github.com/meta-llama/Meta-Llama)
 * [Gradio](https://www.gradio.app/)

agent.py

@@ -3,6 +3,7 @@ from prompts import *
 from tools.squad_tools import SquadRetrieverTool, SquadQueryTool
 from transformers.agents.llm_engine import MessageRole, get_clean_message_list
 from openai import OpenAI
+from prompts import FOCUSED_SQUAD_REACT_CODE_SYSTEM_PROMPT
 
 DEFAULT_TASK_SOLVING_TOOLBOX = [SquadRetrieverTool()] # , SquadQueryTool()
 
@@ -30,7 +31,7 @@ class OpenAIModel:
 
 def get_agent(
     model_name=None,
-    system_prompt=DEFAULT_SQUAD_REACT_CODE_SYSTEM_PROMPT,
+    system_prompt=FOCUSED_SQUAD_REACT_CODE_SYSTEM_PROMPT,
     toolbox=DEFAULT_TASK_SOLVING_TOOLBOX,
     use_openai=True,
     openai_model_name="gpt-4o-mini-2024-07-18",

app.py

@@ -40,6 +40,12 @@ model_name = (
     else "http://localhost:1234/v1"
 )
 
+"""
+The ImageQuestionAnsweringTool from Transformers Agents 2.0 has a bug where 
+it said it accepts the path to an image, but it does not. 
+This class uses the adapter pattern to fix the issue, in a way that may be 
+compatible with future versions of the tool even if the bug is fixed.
+"""
 class FixImageQuestionAnsweringTool(ImageQuestionAnsweringTool):
     def __init__(self, *args, **kwargs):
         super().__init__(*args, **kwargs)
@@ -49,6 +55,13 @@ class FixImageQuestionAnsweringTool(ImageQuestionAnsweringTool):
             image = Image.open(image)
         return super().encode(image, question)
 
+"""
+The app version of the agent has access to additional tools that are not available
+during benchmarking. We chose this approach to focus benchmarking on the agent's
+ability to solve questions about the SQuAD dataset, without the help of general 
+knowledge available on the web.  For the purposes of the project, the demo 
+app has access to additional tools to provide a more interactive and engaging experience.
+"""
 ADDITIONAL_TOOLS = [
     DuckDuckGoSearchTool(),
     VisitWebpageTool(),
@@ -62,7 +75,7 @@ ADDITIONAL_TOOLS = [
 # Add image tools to the default task solving toolbox, for a more visually interactive experience
 TASK_SOLVING_TOOLBOX = DEFAULT_TASK_SOLVING_TOOLBOX + ADDITIONAL_TOOLS
 
-# system_prompt = DEFAULT_SQUAD_REACT_CODE_SYSTEM_PROMPT
+# Using the focused prompt, which was the top-performing prompt during benchmarking
 system_prompt = FOCUSED_SQUAD_REACT_CODE_SYSTEM_PROMPT
 
 agent = get_agent(
@@ -72,9 +85,6 @@ agent = get_agent(
     use_openai=True,  # Use OpenAI instead of a local or HF model as the base LLM engine
 )
 
-app = None
-
-
 def append_example_message(x: gr.SelectData, messages):
     if x.value["text"] is not None:
         message = x.value["text"]
@@ -197,7 +207,7 @@ with gr.Blocks(
                     "text": "What is on top of the Notre Dame building?",
                 },
                 {
-                    "text": "Tell me what's on top of the Notre Dame building, and draw a picture of it.",
+                    "text": "What is the Olympic Torch made of?",
                 },
                 {
                     "text": "Draw a picture of whatever is on top of the Notre Dame building.",

test_bots.py

@@ -1,53 +1,39 @@
-import pytest
 from deepeval import assert_test
 from deepeval.metrics import AnswerRelevancyMetric
 from deepeval.test_case import LLMTestCase
-import pandas as pd
-import os
 from agent import get_agent
-from semscore import EmbeddingModelWrapper
+from prompts import FOCUSED_SQUAD_REACT_CODE_SYSTEM_PROMPT
 import logging
-from tqdm import tqdm
-from transformers.agents import agent_types
 
-def test_case():
-    answer_relevancy_metric = AnswerRelevancyMetric(threshold=0.5)
-    test_case = LLMTestCase(
-        input="What if these shoes don't fit?",
-        # Replace this with the actual output from your LLM application
-        actual_output="We offer a 30-day full refund at no extra costs.",
-        retrieval_context=["All customers are eligible for a 30 day full refund at no extra costs."]
-    )
-    assert_test(test_case, [answer_relevancy_metric])
-
-
-def test_default_agent():
-    SAMPLES_DIR = "samples"
-    os.makedirs(SAMPLES_DIR, exist_ok=True)
-    dfSample = pd.read_pickle(os.path.join(SAMPLES_DIR, f"samples.pkl"))
-    agent = get_agent()
-    # Suppress logging from the agent, which can be quite verbose
+"""
+Test the chatbot's ability to carry out multi-turn conversations, 
+adapt to context, and handle a variety of topics.
+"""
+def test_chatbot_goals():
+    user_messages = [
+        "What is on top of the Notre Dame building?",
+        "When did the United States purchase Alaska from Russia?",
+        "What year did Bern join the Swiss Confederacy?",
+        "Are there any other statues nearby the first one you mentioned?",
+    ]
+    minimum_acceptable_answers = [
+        "golden statue of the Virgin Mary",
+        "1867",
+        "1353",
+        "copper statue of Christ"
+    ]
+    agent = get_agent(system_prompt=FOCUSED_SQUAD_REACT_CODE_SYSTEM_PROMPT)
     agent.logger.setLevel(logging.CRITICAL)
-    answers_ref = []
-    answers_pred = []
-    for title, context, question, answer, synthesized_question in tqdm(dfSample.values):
-        class Output:
-            output: agent_types.AgentType | str = None
-
-        prompt = synthesized_question
-        answers_ref.append(answer)
-        final_answer = agent.run(prompt, stream=False, reset=True)
-        answers_pred.append(final_answer)
-
-    answers_ref = [str(answer) for answer in answers_ref]
-    answers_pred = [str(answer) for answer in answers_pred]
-
-    em = EmbeddingModelWrapper()
-    similarities = em.get_similarities(
-        em.get_embeddings( answers_pred ),
-        em.get_embeddings( answers_ref ),
-    )
-    mean_similarity = similarities.mean()
-
-    assert(mean_similarity >= 0.5, f"Mean similarity is too low: {mean_similarity}")
-        
+    for i, (user_message, minimum_acceptable_answer) in enumerate(zip(user_messages, minimum_acceptable_answers)):
+        answer_relevancy_metric = AnswerRelevancyMetric(threshold=0.5)
+        reset = (i == 0) # Reset the agent for the first message only
+        print(f"Running with reset={reset}")
+        answer = agent.run(user_message, stream=False, reset=reset)
+        print(f"User message: {user_message}")
+        print(f"Minimum acceptable answer: {minimum_acceptable_answer}")
+        print(f"Answer: {answer}")
+        test_case = LLMTestCase(
+            input=user_message,
+            actual_output=answer,
+        )
+        assert_test(test_case, [answer_relevancy_metric])