The Inference Cost of Search Disruption – Large Language Model Cost Analysis
Few things to note:<p>- Google qps is closer to 100k then 320k [1]<p>- Not every query has to run on LLM. Probably only 10% would benefit from it<p>- This means 10,000 queries per second, each needing 5 A100s to run, so 50,000 A100s are sufficient. Cost for that is $500MM, quadruple that to $2B with CPU&#x2F;RAM&#x2F;storage&#x2F;network. That is peanuts for Google.<p>- Let&#x27;s say AI unlocks new, never seen before queries (&#x27;do more people live in madrid or tel aviv&#x27;) and those are 10x in volume. So capex is now $20B, still peanuts.<p>- Latency, not cost, is a bigger issue. This should be addressed soon by H100 and newer chips.<p>- The main issue for the consumer remains the business model. Who will pay for LLMs (user or the advertiser) and will Google and Microsoft stuff ads into LLM responses?<p>[1] <a href="https:&#x2F;&#x2F;www.internetlivestats.com&#x2F;one-second&#x2F;#google-band" rel="nofollow">https:&#x2F;&#x2F;www.internetlivestats.com&#x2F;one-second&#x2F;#google-band</a>