content.raw: 𝗬𝗼𝘂 𝗱𝗼𝗻'𝘁 𝗻𝗲𝗲𝗱 "𝗳𝘂𝗻𝗰𝘁𝗶𝗼𝗻 𝗰𝗮𝗹𝗹𝗶𝗻𝗴 𝗳𝗶𝗻𝗲-𝘁𝘂𝗻𝗶𝗻𝗴" 𝘁𝗼 𝗯𝘂𝗶𝗹𝗱 𝗴𝗼𝗼𝗱 𝗮𝗴𝗲𝗻𝘁𝘀 ⛔
content.type: text
content.value: 𝗬𝗼𝘂 𝗱𝗼𝗻'𝘁 𝗻𝗲𝗲𝗱 "𝗳𝘂𝗻𝗰𝘁𝗶𝗼𝗻 𝗰𝗮𝗹𝗹𝗶𝗻𝗴 𝗳𝗶𝗻𝗲-𝘁𝘂𝗻𝗶𝗻𝗴" 𝘁𝗼 𝗯𝘂𝗶𝗹𝗱 𝗴𝗼𝗼𝗱 𝗮𝗴𝗲𝗻𝘁𝘀 ⛔
content.type: new_line
content.type: new_line
content.raw: It's trendy to share models "fine-tuned for function calling"; but from my observations, this fine-tuning is not necessary or sufficient to build good agent systems.
content.type: text
content.value: It's trendy to share models "fine-tuned for function calling"; but from my observations, this fine-tuning is not necessary or sufficient to build good agent systems.
content.type: new_line
content.raw: To name only a few:
content.type: text
content.value: To name only a few:
content.type: new_line
content.raw: 🐦‍⬛ Nexusflow/𝗡𝗲𝘅𝘂𝘀𝗥𝗮𝘃𝗲𝗻-𝗩𝟮-𝟭𝟯𝗕
content.type: text
content.value: 🐦‍⬛ Nexusflow/𝗡𝗲𝘅𝘂𝘀𝗥𝗮𝘃𝗲𝗻-𝗩𝟮-𝟭𝟯𝗕
content.type: new_line
content.raw: ⌘ CohereForAI/𝗰𝟰𝗮𝗶-𝗰𝗼𝗺𝗺𝗮𝗻𝗱-𝗿-𝗽𝗹𝘂𝘀
content.type: text
content.value: ⌘ CohereForAI/𝗰𝟰𝗮𝗶-𝗰𝗼𝗺𝗺𝗮𝗻𝗱-𝗿-𝗽𝗹𝘂𝘀
content.type: new_line
content.raw: ⛵️ mistralai/𝗠𝗶𝘅𝘁𝗿𝗮𝗹-𝟴𝘅𝟮𝟮𝗕-𝗜𝗻𝘀𝘁𝗿𝘂𝗰𝘁-𝘃𝟬.𝟭
content.type: text
content.value: ⛵️ mistralai/𝗠𝗶𝘅𝘁𝗿𝗮𝗹-𝟴𝘅𝟮𝟮𝗕-𝗜𝗻𝘀𝘁𝗿𝘂𝗰𝘁-𝘃𝟬.𝟭
content.type: new_line
content.raw: "Fine-tuned for function-calling" generally means "fine-tuned to generate function calls in correct JSON for extremely simple tasks". In other terms, it means "improve the formatting of the tool calls".
content.type: text
content.value: "Fine-tuned for function-calling" generally means "fine-tuned to generate function calls in correct JSON for extremely simple tasks". In other terms, it means "improve the formatting of the tool calls".
content.type: new_line
content.type: new_line
content.raw: Yet I discovered two things while improving Transformers Agents:
content.type: text
content.value: Yet I discovered two things while improving Transformers Agents:
content.type: new_line
content.raw: 🧐 Even when used as JSON agents, these fine-tuned models don't perform very well
content.type: text
content.value: 🧐 Even when used as JSON agents, these fine-tuned models don't perform very well
content.type: new_line
content.raw: 🏅 𝙂𝙤𝙤𝙙 𝙗𝙖𝙨𝙚 𝙢𝙤𝙙𝙚𝙡𝙨 𝙥𝙚𝙧𝙛𝙤𝙧𝙢 𝙗𝙚𝙩𝙩𝙚𝙧 𝙬𝙞𝙩𝙝𝙤𝙪𝙩 𝙖𝙣𝙮 𝙛𝙞𝙣𝙚-𝙩𝙪𝙣𝙞𝙣𝙜, 𝙟𝙪𝙨𝙩 𝙥𝙡𝙖𝙞𝙣 𝙥𝙧𝙤𝙢𝙥𝙩𝙞𝙣𝙜. (Llama-3-70B-Instruct, GPT-4o, Claude-3.5-Sonnet) 
content.type: text
content.value: 🏅 𝙂𝙤𝙤𝙙 𝙗𝙖𝙨𝙚 𝙢𝙤𝙙𝙚𝙡𝙨 𝙥𝙚𝙧𝙛𝙤𝙧𝙢 𝙗𝙚𝙩𝙩𝙚𝙧 𝙬𝙞𝙩𝙝𝙤𝙪𝙩 𝙖𝙣𝙮 𝙛𝙞𝙣𝙚-𝙩𝙪𝙣𝙞𝙣𝙜, 𝙟𝙪𝙨𝙩 𝙥𝙡𝙖𝙞𝙣 𝙥𝙧𝙤𝙢𝙥𝙩𝙞𝙣𝙜. (Llama-3-70B-Instruct, GPT-4o, Claude-3.5-Sonnet) 
content.type: new_line
content.type: new_line
content.raw: 👇 The graph below shows the count of errors for my GPT-4o validation run on the GAIA benchmark: 𝙰𝚐𝚎𝚗𝚝𝙿𝚊𝚛𝚜𝚒𝚗𝚐𝙴𝚛𝚛𝚘𝚛 and 𝙰𝚐𝚎𝚗𝚝𝙴𝚡𝚎𝚌𝚞𝚝𝚒𝚘𝚗𝙴𝚛𝚛𝚘𝚛 are the ones caused by incorrect formatting.
content.type: text
content.value: 👇 The graph below shows the count of errors for my GPT-4o validation run on the GAIA benchmark: 𝙰𝚐𝚎𝚗𝚝𝙿𝚊𝚛𝚜𝚒𝚗𝚐𝙴𝚛𝚛𝚘𝚛 and 𝙰𝚐𝚎𝚗𝚝𝙴𝚡𝚎𝚌𝚞𝚝𝚒𝚘𝚗𝙴𝚛𝚛𝚘𝚛 are the ones caused by incorrect formatting.
content.type: new_line
content.raw: ➤ As you can see, their count is already close to 0!
content.type: text
content.value: ➤ As you can see, their count is already close to 0!
content.type: new_line
content.raw: And given that GPT-4o is certainly not fine-tuned for our Code tool calling format, this shows that "function calling fine-tuning" is not necessary!
content.type: text
content.value: And given that GPT-4o is certainly not fine-tuned for our Code tool calling format, this shows that "function calling fine-tuning" is not necessary!
content.type: new_line
content.type: new_line
content.raw: The hardest thing to get right in an agent is still to 𝙥𝙡𝙖𝙣 𝙜𝙤𝙤𝙙 𝙩𝙖𝙨𝙠-𝙨𝙤𝙡𝙫𝙞𝙣𝙜 𝙩𝙧𝙖𝙟𝙚𝙘𝙩𝙤𝙧𝙞𝙚𝙨 𝙤𝙫𝙚𝙧 𝙨𝙚𝙫𝙚𝙧𝙖𝙡 𝙨𝙩𝙚𝙥𝙨.
content.type: text
content.value: The hardest thing to get right in an agent is still to 𝙥𝙡𝙖𝙣 𝙜𝙤𝙤𝙙 𝙩𝙖𝙨𝙠-𝙨𝙤𝙡𝙫𝙞𝙣𝙜 𝙩𝙧𝙖𝙟𝙚𝙘𝙩𝙤𝙧𝙞𝙚𝙨 𝙤𝙫𝙚𝙧 𝙨𝙚𝙫𝙚𝙧𝙖𝙡 𝙨𝙩𝙚𝙥𝙨.
content.type: new_line
content.raw: To improve this, we could:
content.type: text
content.value: To improve this, we could:
content.type: new_line
content.raw: - Use more powerful base models
content.type: text
content.value: - Use more powerful base models
content.type: new_line
content.raw: - Make tool calling datasets with complex solving trajectories
content.type: text
content.value: - Make tool calling datasets with complex solving trajectories
content.type: new_line
content.raw: - Use RL! cc 
content.type: text
content.value: - Use RL! cc 
content.raw: @lvwerra
content.type: mention
content.user: lvwerra
rawContent: 𝗬𝗼𝘂 𝗱𝗼𝗻'𝘁 𝗻𝗲𝗲𝗱 "𝗳𝘂𝗻𝗰𝘁𝗶𝗼𝗻 𝗰𝗮𝗹𝗹𝗶𝗻𝗴 𝗳𝗶𝗻𝗲-𝘁𝘂𝗻𝗶𝗻𝗴" 𝘁𝗼 𝗯𝘂𝗶𝗹𝗱 𝗴𝗼𝗼𝗱 𝗮𝗴𝗲𝗻𝘁𝘀 ⛔
  
  It's trendy to share models "fine-tuned for function calling"; but from my observations, this fine-tuning is not necessary or sufficient to build good agent systems.
  To name only a few:
  🐦‍⬛ Nexusflow/𝗡𝗲𝘅𝘂𝘀𝗥𝗮𝘃𝗲𝗻-𝗩𝟮-𝟭𝟯𝗕
  ⌘ CohereForAI/𝗰𝟰𝗮𝗶-𝗰𝗼𝗺𝗺𝗮𝗻𝗱-𝗿-𝗽𝗹𝘂𝘀
  ⛵️ mistralai/𝗠𝗶𝘅𝘁𝗿𝗮𝗹-𝟴𝘅𝟮𝟮𝗕-𝗜𝗻𝘀𝘁𝗿𝘂𝗰𝘁-𝘃𝟬.𝟭
  "Fine-tuned for function-calling" generally means "fine-tuned to generate function calls in correct JSON for extremely simple tasks". In other terms, it means "improve the formatting of the tool calls".
  
  Yet I discovered two things while improving Transformers Agents:
  🧐 Even when used as JSON agents, these fine-tuned models don't perform very well
  🏅 𝙂𝙤𝙤𝙙 𝙗𝙖𝙨𝙚 𝙢𝙤𝙙𝙚𝙡𝙨 𝙥𝙚𝙧𝙛𝙤𝙧𝙢 𝙗𝙚𝙩𝙩𝙚𝙧 𝙬𝙞𝙩𝙝𝙤𝙪𝙩 𝙖𝙣𝙮 𝙛𝙞𝙣𝙚-𝙩𝙪𝙣𝙞𝙣𝙜, 𝙟𝙪𝙨𝙩 𝙥𝙡𝙖𝙞𝙣 𝙥𝙧𝙤𝙢𝙥𝙩𝙞𝙣𝙜. (Llama-3-70B-Instruct, GPT-4o, Claude-3.5-Sonnet) 
  
  👇 The graph below shows the count of errors for my GPT-4o validation run on the GAIA benchmark: 𝙰𝚐𝚎𝚗𝚝𝙿𝚊𝚛𝚜𝚒𝚗𝚐𝙴𝚛𝚛𝚘𝚛 and 𝙰𝚐𝚎𝚗𝚝𝙴𝚡𝚎𝚌𝚞𝚝𝚒𝚘𝚗𝙴𝚛𝚛𝚘𝚛 are the ones caused by incorrect formatting.
  ➤ As you can see, their count is already close to 0!
  And given that GPT-4o is certainly not fine-tuned for our Code tool calling format, this shows that "function calling fine-tuning" is not necessary!
  
  The hardest thing to get right in an agent is still to 𝙥𝙡𝙖𝙣 𝙜𝙤𝙤𝙙 𝙩𝙖𝙨𝙠-𝙨𝙤𝙡𝙫𝙞𝙣𝙜 𝙩𝙧𝙖𝙟𝙚𝙘𝙩𝙤𝙧𝙞𝙚𝙨 𝙤𝙫𝙚𝙧 𝙨𝙚𝙫𝙚𝙧𝙖𝙡 𝙨𝙩𝙚𝙥𝙨.
  To improve this, we could:
  - Use more powerful base models
  - Make tool calling datasets with complex solving trajectories
  - Use RL! cc @lvwerra
reactions.reaction: 🔥
reactions.users: lvwerra merterbak AnkitAI louisbrulenaudet osanseviero Joseph717171 JackCloudman GPT007
reactions.reaction: 😎
reactions.users: GPT007 osanseviero Joseph717171
commentators._id: 63d10d4e8eaa4831005e92b5
commentators.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63d10d4e8eaa4831005e92b5/7p7-OmWM6PqqCs7ZStPGD.jpeg
commentators.fullname: Aymeric Roucher
commentators.name: m-ric
commentators.type: user
commentators._id: 642f57bdaa1dd0ebdf3e4aeb
commentators.avatarUrl: /avatars/d1b8a9a55b8a3d5e8abac51953027b07.svg
commentators.fullname: GX Kok
commentators.name: gxkok
commentators.type: user