🏆 Mistral AI Robotic Hackathon – 2nd Place: “LeCopain” 🤖

Building an Interactive Guess Who? Robot with Mistral AI and SO-100 Arms

🧠 Genesis of the Project

From April 11th to 13th, the Mistral AI Robotic Hackathon challenged participants to combine the power of Mistral AI models with SO-100 robotic arms to build real-world interactive systems.

Our team secured 2nd place with “LeCopain” — a friendly robot capable of playing the classic board game Guess Who? by combining natural language processing, speech interfaces, and robotic control into a seamless, interactive experience.

The concept emerged naturally during brainstorming: what if a robot could play games with us? Board games offer a rich environment blending perception, reasoning, dialogue, and action — the ideal playground for testing both LLMs and robot policies. Guess Who? stood out as a simple yet cognitively rich challenge requiring multimodal understanding, reasoning, and physical interaction.

🎲 The Guess Who? Game – A Quick Recap

Guess Who? is a two-player guessing game where each player selects a mystery character from a pool of options. Players take turns asking yes/no questions (e.g., “Does your animal have fur?”), eliminating candidates based on the answers until only one remains. In our version, we used animals as the characters, making it more visually and semantically engaging for a robotic system.

🛠️ Project Overview

System Pipeline

To bring LeCopain to life, we setup a modular pipeline that combines speech interaction, language understanding, and robotic control. Here’s a breakdown of each component:

1. 🧑‍🎤 User Interface & Avatar: We built an engaging and accessible user interface:

• The robot is plugged to a computer equipped with a microphone and speaker to enable natural voice interactions.
• A custom 2D animated avatar (mouth-sync, eye movement) provides visual feedback, enhancing the sense of personality and presence.

While we planned to implement emotional expressions, this remains an area for future development. This multimodal design ensures the game is enjoyable and accessible to people of all ages and abilities.

2. 🗣️ Speech-to-Text and Text-to-Speech: We used Whisper by OpenAI for real-time speech-to-text, enabling the robot to understand user questions. To respond, we added text-to-speech synthesis, making the robot capable of holding full spoken conversations. This setup not only mimics natural dialogue but also makes the game accessible for players with limited mobility.

3. 🧠 Cognitive Model: Reasoning with Mistral AI At the heart of LeCopain’s intelligence lies Mistral AI’s open-source small model, used to interpret user input and decide which animals to eliminate. We implemented two modes:

   • Image-Based Prompting: Feeding an image of the full board with the user’s question.
   • Text-Based Prompting: Using a structured list of animal traits with the user’s question. The model was prompted to return a structured JSON output containing a list of animals to eliminate, making downstream control decisions simpler.

ACT conditioning with grid ID 4. 🤖 Robotic Control with LeRobot + Custom ACT Policy To execute physical actions (e.g., flipping cells), we used the LeRobot library from Hugging Face and developed a custom ACT policy: - We added a lightweight conditioning layer to the policy's input, allowing it to receive the grid position of the animal to eliminate. - This enabled us to train a single policy model that generalizes to all possible board actions.

📊 Data Collection

To train the ACT policy, we extended LeRobotDataset to include grid ID input and collected 96 trajectories, covering each grid cell from left-to-right, top-to-bottom.

This dataset proved sufficient for a smooth demo experience, though further training will be needed for robust deployment in more complex environments.

Our different datasets we realeased at the end of the Hackathon at the following page (team 2 datasets): Guess Who datasets

🎉 Final Thoughts

Our project, LeCopain, demonstrates the powerful potential of integrating robotics, LLMs, and voice interaction to create truly engaging experiences.

From cognitive reasoning to physical execution, we showed how these technologies can come together to power interactive, playful, and meaningful robotics applications. This prototype could evolve into tools for education, therapy, or assistive entertainment.

Special thanks to Mistral AI and all the other partners of this incredible event!

🎥 Demo Video

📺 See LeCopain in action:

💻 Code Release & Call for Contributors

We’ve open-sourced the project and are excited to keep building this idea with the community. If you're passionate about AI, robotics, games, or accessibility, we’d love to collaborate!

👉 Github repository (We are currently cleaning up the code but will accept soon new contributors to the project)

📧 Interested in contributing or collaborating? Reach out at [email protected]