𝗔𝗿𝗲 𝗔𝗴𝗲𝗻𝘁𝘀 𝗰𝗮𝗽𝗮𝗯𝗹𝗲 𝗲𝗻𝗼𝘂𝗴𝗵 𝗳𝗼𝗿 𝗗𝗮𝘁𝗮 𝗦𝗰𝗶𝗲𝗻𝗰𝗲? ⇒ 𝗠𝗲𝗮𝘀𝘂𝗿𝗲 𝘁𝗵𝗲𝗶𝗿 𝗽𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲 𝘄𝗶𝘁𝗵 𝗗𝗦𝗕𝗲𝗻𝗰𝗵 📊
text
𝗔𝗿𝗲 𝗔𝗴𝗲𝗻𝘁𝘀 𝗰𝗮𝗽𝗮𝗯𝗹𝗲 𝗲𝗻𝗼𝘂𝗴𝗵 𝗳𝗼𝗿 𝗗𝗮𝘁𝗮 𝗦𝗰𝗶𝗲𝗻𝗰𝗲? ⇒ 𝗠𝗲𝗮𝘀𝘂𝗿𝗲 𝘁𝗵𝗲𝗶𝗿 𝗽𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲 𝘄𝗶𝘁𝗵 𝗗𝗦𝗕𝗲𝗻𝗰𝗵 📊
new_line
new_line
A team from Tencent AI wanted to evaluate agentic systems on data science (DS) tasks : but they noticed that existing agentic benchmarks were severely limited in several aspects: they were limited to text and did not include tables or images, were only specific to certain packages, only performed exact match evaluation…
text
A team from Tencent AI wanted to evaluate agentic systems on data science (DS) tasks : but they noticed that existing agentic benchmarks were severely limited in several aspects: they were limited to text and did not include tables or images, were only specific to certain packages, only performed exact match evaluation…
new_line
new_line
➡️ So they set out to build a much more exhaustive approach, to finally make the definitive DS agent benchmark.
text
➡️ So they set out to build a much more exhaustive approach, to finally make the definitive DS agent benchmark.
new_line
new_line
𝗧𝗵𝗲 𝗗𝗦𝗕𝗲𝗻𝗰𝗵 𝗱𝗮𝘁𝗮𝘀𝗲𝘁
text
𝗧𝗵𝗲 𝗗𝗦𝗕𝗲𝗻𝗰𝗵 𝗱𝗮𝘁𝗮𝘀𝗲𝘁
new_line
▪️DS bench has 466 data analysis tasks and 74 data modelling tasks
text
▪️DS bench has 466 data analysis tasks and 74 data modelling tasks
new_line
▪️The tasks are sourced from ModelOff and Kaggle, the platforms hosting the most popular data science competitions
text
▪️The tasks are sourced from ModelOff and Kaggle, the platforms hosting the most popular data science competitions
new_line
▪️Difference with previous DS benchmarks:
text
▪️Difference with previous DS benchmarks:
new_line
 ❶ This benchmark leverages various modalities on top of text: images, Excel files, tables
text
 ❶ This benchmark leverages various modalities on top of text: images, Excel files, tables
new_line
 ❷ Complex tables: sometimes several tables should be leveraged to answer one question
text
 ❷ Complex tables: sometimes several tables should be leveraged to answer one question
new_line
 ❸ The context is richer, with longer descriptions.
text
 ❸ The context is richer, with longer descriptions.
new_line
▪️ Evaluation metrics : the benchmark is scored with an LLM as a judge, using a specific prompt.
text
▪️ Evaluation metrics : the benchmark is scored with an LLM as a judge, using a specific prompt.
new_line
new_line
𝗜𝗻𝘀𝗶𝗴𝗵𝘁𝘀 𝗳𝗿𝗼𝗺 𝗲𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗻𝗴 𝗮𝗴𝗲𝗻𝘁𝘀
text
𝗜𝗻𝘀𝗶𝗴𝗵𝘁𝘀 𝗳𝗿𝗼𝗺 𝗲𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗻𝗴 𝗮𝗴𝗲𝗻𝘁𝘀
new_line
▪️ Their evaluation confirms that using LLMs in an agent setup, for instance by allowing them to run a single step of code execution, is more costly (especially with multi-turn frameworks like autogen) but also much more performant than the vanilla LLM.
text
▪️ Their evaluation confirms that using LLMs in an agent setup, for instance by allowing them to run a single step of code execution, is more costly (especially with multi-turn frameworks like autogen) but also much more performant than the vanilla LLM.
new_line
▪️ The sets of tasks solved by different models (like GPT-3.5 vs Llama-3-8B) has quite low overlap, which suggests that different models tend to try very different approches.
text
▪️ The sets of tasks solved by different models (like GPT-3.5 vs Llama-3-8B) has quite low overlap, which suggests that different models tend to try very different approches.
new_line
new_line
This new benchmark is really welcome, can't wait to try transformers agents on it! 🤗
text
This new benchmark is really welcome, can't wait to try transformers agents on it! 🤗
new_line
new_line
Read their full paper 👉 
text
Read their full paper 👉 
DSBench: How Far Are Data Science Agents to Becoming Data Science
  Experts? (2409.07703)
https://huggingface.co/papers/2409.07703
2409.07703
paper
resource
https://huggingface.co/papers/2409.07703
𝗔𝗿𝗲 𝗔𝗴𝗲𝗻𝘁𝘀 𝗰𝗮𝗽𝗮𝗯𝗹𝗲 𝗲𝗻𝗼𝘂𝗴𝗵 𝗳𝗼𝗿 𝗗𝗮𝘁𝗮 𝗦𝗰𝗶𝗲𝗻𝗰𝗲? ⇒ 𝗠𝗲𝗮𝘀𝘂𝗿𝗲 𝘁𝗵𝗲𝗶𝗿 𝗽𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲 𝘄𝗶𝘁𝗵 𝗗𝗦𝗕𝗲𝗻𝗰𝗵 📊

A team from Tencent AI wanted to evaluate agentic systems on data science (DS) tasks : but they noticed that existing agentic benchmarks were severely limited in several aspects: they were limited to text and did not include tables or images, were only specific to certain packages, only performed exact match evaluation…

➡️ So they set out to build a much more exhaustive approach, to finally make the definitive DS agent benchmark.

𝗧𝗵𝗲 𝗗𝗦𝗕𝗲𝗻𝗰𝗵 𝗱𝗮𝘁𝗮𝘀𝗲𝘁
▪️DS bench has 466 data analysis tasks and 74 data modelling tasks
▪️The tasks are sourced from ModelOff and Kaggle, the platforms hosting the most popular data science competitions
▪️Difference with previous DS benchmarks:
 ❶ This benchmark leverages various modalities on top of text: images, Excel files, tables
 ❷ Complex tables: sometimes several tables should be leveraged to answer one question
 ❸ The context is richer, with longer descriptions.
▪️ Evaluation metrics : the benchmark is scored with an LLM as a judge, using a specific prompt.

𝗜𝗻𝘀𝗶𝗴𝗵𝘁𝘀 𝗳𝗿𝗼𝗺 𝗲𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗻𝗴 𝗮𝗴𝗲𝗻𝘁𝘀
▪️ Their evaluation confirms that using LLMs in an agent setup, for instance by allowing them to run a single step of code execution, is more costly (especially with multi-turn frameworks like autogen) but also much more performant than the vanilla LLM.
▪️ The sets of tasks solved by different models (like GPT-3.5 vs Llama-3-8B) has quite low overlap, which suggests that different models tend to try very different approches.

This new benchmark is really welcome, can't wait to try transformers agents on it! 🤗

Read their full paper 👉 https://huggingface.co/papers/2409.07703
👀
John6666 louisbrulenaudet asoria
👍
Tonic FalconNet liqiang888