content.raw: 𝗔𝗿𝗲 𝗔𝗴𝗲𝗻𝘁𝘀 𝗰𝗮𝗽𝗮𝗯𝗹𝗲 𝗲𝗻𝗼𝘂𝗴𝗵 𝗳𝗼𝗿 𝗗𝗮𝘁𝗮 𝗦𝗰𝗶𝗲𝗻𝗰𝗲? ⇒ 𝗠𝗲𝗮𝘀𝘂𝗿𝗲 𝘁𝗵𝗲𝗶𝗿 𝗽𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲 𝘄𝗶𝘁𝗵 𝗗𝗦𝗕𝗲𝗻𝗰𝗵 📊
content.type: text
content.value: 𝗔𝗿𝗲 𝗔𝗴𝗲𝗻𝘁𝘀 𝗰𝗮𝗽𝗮𝗯𝗹𝗲 𝗲𝗻𝗼𝘂𝗴𝗵 𝗳𝗼𝗿 𝗗𝗮𝘁𝗮 𝗦𝗰𝗶𝗲𝗻𝗰𝗲? ⇒ 𝗠𝗲𝗮𝘀𝘂𝗿𝗲 𝘁𝗵𝗲𝗶𝗿 𝗽𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲 𝘄𝗶𝘁𝗵 𝗗𝗦𝗕𝗲𝗻𝗰𝗵 📊
content.type: new_line
content.type: new_line
content.raw: A team from Tencent AI wanted to evaluate agentic systems on data science (DS) tasks : but they noticed that existing agentic benchmarks were severely limited in several aspects: they were limited to text and did not include tables or images, were only specific to certain packages, only performed exact match evaluation…
content.type: text
content.value: A team from Tencent AI wanted to evaluate agentic systems on data science (DS) tasks : but they noticed that existing agentic benchmarks were severely limited in several aspects: they were limited to text and did not include tables or images, were only specific to certain packages, only performed exact match evaluation…
content.type: new_line
content.type: new_line
content.raw: ➡️ So they set out to build a much more exhaustive approach, to finally make the definitive DS agent benchmark.
content.type: text
content.value: ➡️ So they set out to build a much more exhaustive approach, to finally make the definitive DS agent benchmark.
content.type: new_line
content.type: new_line
content.raw: 𝗧𝗵𝗲 𝗗𝗦𝗕𝗲𝗻𝗰𝗵 𝗱𝗮𝘁𝗮𝘀𝗲𝘁
content.type: text
content.value: 𝗧𝗵𝗲 𝗗𝗦𝗕𝗲𝗻𝗰𝗵 𝗱𝗮𝘁𝗮𝘀𝗲𝘁
content.type: new_line
content.raw: ▪️DS bench has 466 data analysis tasks and 74 data modelling tasks
content.type: text
content.value: ▪️DS bench has 466 data analysis tasks and 74 data modelling tasks
content.type: new_line
content.raw: ▪️The tasks are sourced from ModelOff and Kaggle, the platforms hosting the most popular data science competitions
content.type: text
content.value: ▪️The tasks are sourced from ModelOff and Kaggle, the platforms hosting the most popular data science competitions
content.type: new_line
content.raw: ▪️Difference with previous DS benchmarks:
content.type: text
content.value: ▪️Difference with previous DS benchmarks:
content.type: new_line
content.raw:  ❶ This benchmark leverages various modalities on top of text: images, Excel files, tables
content.type: text
content.value:  ❶ This benchmark leverages various modalities on top of text: images, Excel files, tables
content.type: new_line
content.raw:  ❷ Complex tables: sometimes several tables should be leveraged to answer one question
content.type: text
content.value:  ❷ Complex tables: sometimes several tables should be leveraged to answer one question
content.type: new_line
content.raw:  ❸ The context is richer, with longer descriptions.
content.type: text
content.value:  ❸ The context is richer, with longer descriptions.
content.type: new_line
content.raw: ▪️ Evaluation metrics : the benchmark is scored with an LLM as a judge, using a specific prompt.
content.type: text
content.value: ▪️ Evaluation metrics : the benchmark is scored with an LLM as a judge, using a specific prompt.
content.type: new_line
content.type: new_line
content.raw: 𝗜𝗻𝘀𝗶𝗴𝗵𝘁𝘀 𝗳𝗿𝗼𝗺 𝗲𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗻𝗴 𝗮𝗴𝗲𝗻𝘁𝘀
content.type: text
content.value: 𝗜𝗻𝘀𝗶𝗴𝗵𝘁𝘀 𝗳𝗿𝗼𝗺 𝗲𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗻𝗴 𝗮𝗴𝗲𝗻𝘁𝘀
content.type: new_line
content.raw: ▪️ Their evaluation confirms that using LLMs in an agent setup, for instance by allowing them to run a single step of code execution, is more costly (especially with multi-turn frameworks like autogen) but also much more performant than the vanilla LLM.
content.type: text
content.value: ▪️ Their evaluation confirms that using LLMs in an agent setup, for instance by allowing them to run a single step of code execution, is more costly (especially with multi-turn frameworks like autogen) but also much more performant than the vanilla LLM.
content.type: new_line
content.raw: ▪️ The sets of tasks solved by different models (like GPT-3.5 vs Llama-3-8B) has quite low overlap, which suggests that different models tend to try very different approches.
content.type: text
content.value: ▪️ The sets of tasks solved by different models (like GPT-3.5 vs Llama-3-8B) has quite low overlap, which suggests that different models tend to try very different approches.
content.type: new_line
content.type: new_line
content.raw: This new benchmark is really welcome, can't wait to try transformers agents on it! 🤗
content.type: text
content.value: This new benchmark is really welcome, can't wait to try transformers agents on it! 🤗
content.type: new_line
content.type: new_line
content.raw: Read their full paper 👉 
content.type: text
content.value: Read their full paper 👉 
content.label: DSBench: How Far Are Data Science Agents to Becoming Data Science
    Experts? (2409.07703)
content.raw: https://huggingface.co/papers/2409.07703
content.id: 2409.07703
content.type: paper
content.type: resource
content.url: https://huggingface.co/papers/2409.07703
rawContent: 𝗔𝗿𝗲 𝗔𝗴𝗲𝗻𝘁𝘀 𝗰𝗮𝗽𝗮𝗯𝗹𝗲 𝗲𝗻𝗼𝘂𝗴𝗵 𝗳𝗼𝗿 𝗗𝗮𝘁𝗮 𝗦𝗰𝗶𝗲𝗻𝗰𝗲? ⇒ 𝗠𝗲𝗮𝘀𝘂𝗿𝗲 𝘁𝗵𝗲𝗶𝗿 𝗽𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲 𝘄𝗶𝘁𝗵 𝗗𝗦𝗕𝗲𝗻𝗰𝗵 📊
  
  A team from Tencent AI wanted to evaluate agentic systems on data science (DS) tasks : but they noticed that existing agentic benchmarks were severely limited in several aspects: they were limited to text and did not include tables or images, were only specific to certain packages, only performed exact match evaluation…
  
  ➡️ So they set out to build a much more exhaustive approach, to finally make the definitive DS agent benchmark.
  
  𝗧𝗵𝗲 𝗗𝗦𝗕𝗲𝗻𝗰𝗵 𝗱𝗮𝘁𝗮𝘀𝗲𝘁
  ▪️DS bench has 466 data analysis tasks and 74 data modelling tasks
  ▪️The tasks are sourced from ModelOff and Kaggle, the platforms hosting the most popular data science competitions
  ▪️Difference with previous DS benchmarks:
   ❶ This benchmark leverages various modalities on top of text: images, Excel files, tables
   ❷ Complex tables: sometimes several tables should be leveraged to answer one question
   ❸ The context is richer, with longer descriptions.
  ▪️ Evaluation metrics : the benchmark is scored with an LLM as a judge, using a specific prompt.
  
  𝗜𝗻𝘀𝗶𝗴𝗵𝘁𝘀 𝗳𝗿𝗼𝗺 𝗲𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗻𝗴 𝗮𝗴𝗲𝗻𝘁𝘀
  ▪️ Their evaluation confirms that using LLMs in an agent setup, for instance by allowing them to run a single step of code execution, is more costly (especially with multi-turn frameworks like autogen) but also much more performant than the vanilla LLM.
  ▪️ The sets of tasks solved by different models (like GPT-3.5 vs Llama-3-8B) has quite low overlap, which suggests that different models tend to try very different approches.
  
  This new benchmark is really welcome, can't wait to try transformers agents on it! 🤗
  
  Read their full paper 👉 https://huggingface.co/papers/2409.07703
reactions.reaction: 👀
reactions.users: John6666 louisbrulenaudet asoria
reactions.reaction: 👍
reactions.users: Tonic FalconNet liqiang888