content.raw: 🚀 𝗪𝗵𝗲𝗿𝗲 𝘀𝗰𝗮𝗹𝗶𝗻𝗴 𝗹𝗮𝘄𝘀 𝗮𝗿𝗲 𝘁𝗮𝗸𝗶𝗻𝗴 𝘂𝘀 : 𝗯𝘆 𝟮𝟬𝟮𝟴, 𝗔𝗜 𝗖𝗹𝘂𝘀𝘁𝗲𝗿𝘀 𝘄𝗶𝗹𝗹 𝗿𝗲𝗮𝗰𝗵 𝘁𝗵𝗲 𝗽𝗼𝘄𝗲𝗿 𝗰𝗼𝗻𝘀𝘂𝗺𝗽𝘁𝗶𝗼𝗻 𝗼𝗳 𝗲𝗻𝘁𝗶𝗿𝗲 𝗰𝗼𝘂𝗻𝘁𝗿𝗶𝗲𝘀
content.type: text
content.value: 🚀 𝗪𝗵𝗲𝗿𝗲 𝘀𝗰𝗮𝗹𝗶𝗻𝗴 𝗹𝗮𝘄𝘀 𝗮𝗿𝗲 𝘁𝗮𝗸𝗶𝗻𝗴 𝘂𝘀 : 𝗯𝘆 𝟮𝟬𝟮𝟴, 𝗔𝗜 𝗖𝗹𝘂𝘀𝘁𝗲𝗿𝘀 𝘄𝗶𝗹𝗹 𝗿𝗲𝗮𝗰𝗵 𝘁𝗵𝗲 𝗽𝗼𝘄𝗲𝗿 𝗰𝗼𝗻𝘀𝘂𝗺𝗽𝘁𝗶𝗼𝗻 𝗼𝗳 𝗲𝗻𝘁𝗶𝗿𝗲 𝗰𝗼𝘂𝗻𝘁𝗿𝗶𝗲𝘀
content.type: new_line
content.type: new_line
content.raw: Reminder : “Scaling laws” are empirical laws saying that  if you keep multiplying your compute by x10, your models will mechanically keep getting better and better.
content.type: text
content.value: Reminder : “Scaling laws” are empirical laws saying that  if you keep multiplying your compute by x10, your models will mechanically keep getting better and better.
content.type: new_line
content.type: new_line
content.raw: To give you an idea, GPT-3 can barely write sentences, and GPT-4, which only used x15 its amount of compute, already sounds much smarter than some of my friends (although it's not really - or at least I haven't tested them side-by side). So you can imagine how far a x100 over GPT-4 can take us.
content.type: text
content.value: To give you an idea, GPT-3 can barely write sentences, and GPT-4, which only used x15 its amount of compute, already sounds much smarter than some of my friends (although it's not really - or at least I haven't tested them side-by side). So you can imagine how far a x100 over GPT-4 can take us.
content.type: new_line
content.type: new_line
content.raw: 🏎️ As a result, tech titans are racing to build the biggest models, and for this they need gigantic training clusters.
content.type: text
content.value: 🏎️ As a result, tech titans are racing to build the biggest models, and for this they need gigantic training clusters.
content.type: new_line
content.type: new_line
content.raw: The picture below shows the growth of training compute: it is increasing at a steady exponential rate of a x10 every 2 years. So let’s take this progress a bit further:
content.type: text
content.value: The picture below shows the growth of training compute: it is increasing at a steady exponential rate of a x10 every 2 years. So let’s take this progress a bit further:
content.type: new_line
content.raw: - 2022: starting training for GPT-4 : 10^26 FLOPs, cost of $100M
content.type: text
content.value: - 2022: starting training for GPT-4 : 10^26 FLOPs, cost of $100M
content.type: new_line
content.raw: - 2024: today, companies start training on much larger clusters like the “super AI cluster” of Elon Musk’s xAI, 10^27 FLOPS, $1B
content.type: text
content.value: - 2024: today, companies start training on much larger clusters like the “super AI cluster” of Elon Musk’s xAI, 10^27 FLOPS, $1B
content.type: new_line
content.raw: - 2026 : by then clusters will require 1GW, i.e. around the full power generated by a nuclear reactor
content.type: text
content.value: - 2026 : by then clusters will require 1GW, i.e. around the full power generated by a nuclear reactor
content.type: new_line
content.raw: - 2028: we reach cluster prices in the 100 billion dollars, using 10GW, more than the most powerful power stations currently in use in the US. This last size seems crazy, but Microsoft and OpenAI already are planning one.
content.type: text
content.value: - 2028: we reach cluster prices in the 100 billion dollars, using 10GW, more than the most powerful power stations currently in use in the US. This last size seems crazy, but Microsoft and OpenAI already are planning one.
content.type: new_line
content.type: new_line
content.raw: Will AI clusters effectively reach these crazy sizes where the consume as much as entire countries? 
content.type: text
content.value: Will AI clusters effectively reach these crazy sizes where the consume as much as entire countries? 
content.type: new_line
content.raw: ➡️ Three key ingredients of training might be a roadblock to scaling up :
content.type: text
content.value: ➡️ Three key ingredients of training might be a roadblock to scaling up :
content.type: new_line
content.raw: 💸 Money: but it’s very unlikely, given the potential market size for AGI, that investors lose interest.
content.type: text
content.value: 💸 Money: but it’s very unlikely, given the potential market size for AGI, that investors lose interest.
content.type: new_line
content.raw: ⚡️ Energy supply at a specific location
content.type: text
content.value: ⚡️ Energy supply at a specific location
content.type: new_line
content.raw: 📚 Training data: we’re already using 15 trillion tokens for Llama-3.1 when Internet has something like 60 trillion.
content.type: text
content.value: 📚 Training data: we’re already using 15 trillion tokens for Llama-3.1 when Internet has something like 60 trillion.
content.type: new_line
content.type: new_line
content.raw: 🤔 I’d be curious to hear your thoughts: do you think we’ll race all the way there?
content.type: text
content.value: 🤔 I’d be curious to hear your thoughts: do you think we’ll race all the way there?
rawContent: 🚀 𝗪𝗵𝗲𝗿𝗲 𝘀𝗰𝗮𝗹𝗶𝗻𝗴 𝗹𝗮𝘄𝘀 𝗮𝗿𝗲 𝘁𝗮𝗸𝗶𝗻𝗴 𝘂𝘀 : 𝗯𝘆 𝟮𝟬𝟮𝟴, 𝗔𝗜 𝗖𝗹𝘂𝘀𝘁𝗲𝗿𝘀 𝘄𝗶𝗹𝗹 𝗿𝗲𝗮𝗰𝗵 𝘁𝗵𝗲 𝗽𝗼𝘄𝗲𝗿 𝗰𝗼𝗻𝘀𝘂𝗺𝗽𝘁𝗶𝗼𝗻 𝗼𝗳 𝗲𝗻𝘁𝗶𝗿𝗲 𝗰𝗼𝘂𝗻𝘁𝗿𝗶𝗲𝘀
  
  Reminder : “Scaling laws” are empirical laws saying that  if you keep multiplying your compute by x10, your models will mechanically keep getting better and better.
  
  To give you an idea, GPT-3 can barely write sentences, and GPT-4, which only used x15 its amount of compute, already sounds much smarter than some of my friends (although it's not really - or at least I haven't tested them side-by side). So you can imagine how far a x100 over GPT-4 can take us.
  
  🏎️ As a result, tech titans are racing to build the biggest models, and for this they need gigantic training clusters.
  
  The picture below shows the growth of training compute: it is increasing at a steady exponential rate of a x10 every 2 years. So let’s take this progress a bit further:
  - 2022: starting training for GPT-4 : 10^26 FLOPs, cost of $100M
  - 2024: today, companies start training on much larger clusters like the “super AI cluster” of Elon Musk’s xAI, 10^27 FLOPS, $1B
  - 2026 : by then clusters will require 1GW, i.e. around the full power generated by a nuclear reactor
  - 2028: we reach cluster prices in the 100 billion dollars, using 10GW, more than the most powerful power stations currently in use in the US. This last size seems crazy, but Microsoft and OpenAI already are planning one.
  
  Will AI clusters effectively reach these crazy sizes where the consume as much as entire countries? 
  ➡️ Three key ingredients of training might be a roadblock to scaling up :
  💸 Money: but it’s very unlikely, given the potential market size for AGI, that investors lose interest.
  ⚡️ Energy supply at a specific location
  📚 Training data: we’re already using 15 trillion tokens for Llama-3.1 when Internet has something like 60 trillion.
  
  🤔 I’d be curious to hear your thoughts: do you think we’ll race all the way there?
reactions.reaction: 👀
reactions.users: John6666 Kaoeiri
reactions.reaction: 👍
reactions.users: lamhieu Kaoeiri
commentators._id: 600ae38cc92b79f54efd4556
commentators.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/600ae38cc92b79f54efd4556/cSqRIslYl5L3I4WK3a31f.png
commentators.fullname: Hieu Lam
commentators.name: lamhieu
commentators.type: user
commentators._id: 65921e3dcdc0c4099b83daa3
commentators.avatarUrl: /avatars/ac25f29292cca71ab6d509ea781e7943.svg
commentators.fullname: Shareef Taylor
commentators.name: MANOFAi94
commentators.type: user
commentators._id: 662162fd296b3d40f15367a4
commentators.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/662162fd296b3d40f15367a4/jM74dtHuAGI6UlLGT7A9s.jpeg
commentators.fullname: Stephen Genusa
commentators.name: StephenGenusa
commentators.type: user