Burnt $72k testing Firebase and Cloud Run and almost went bankrupt
Disclaimer I work for another cloud (not AWS), opinions are entirely my own.  I try to avoid posting in a negative fashion about clouds, but holy crap this blog post...<p>AWS has this principle of Customer Obsession that enters in to lots of discussions, design decisions etc.  &quot;What is the customer experience of $foo?&quot;.  Along with asking the positive, you ask the negative too, and explore the customer impact of shit going wrong.  What does the worst experience look like, what is the impact for the customer, how might you mitigate that or make it so you can at least make it up to customers quickly, if you really can&#x27;t avoid it.<p>I find it hard to fathom Sudeep&#x27;s attitude here.  So much of this article is ringing large alarm bells.  These are not the things I&#x27;d want to see from a cloud provider as a customer.<p>Is this Stockholm Syndrome?  Too much drinking of Kool-Aid as an ex-googler?  Unfamiliarity with how other cloud providers operate?<p>(from part 1)
&gt; Automatic Upgrade of Firebase Account to Paid Account<p>This is what I mean when I say look at negative vs positive use cases.  I&#x27;m guessing some combination of customers having a lousy experience running in to Free Tier limits, and staff spending too long having to bump up accounts.  So they implemented an automatic upgrade  (What, then, is the point of a free tier?  No room to experiment, no room to try it and see)<p>This is precisely the sort of thing that customer obsession principle is supposed to aid in.  Automated upgrade certainly solves the staffing time spent bumping up accounts, and it helps customers that used to have to request limits being increased, but it massively fails in the negative customer experience side of the equation here.  Someone, somewhere, should have asked the question &quot;What if the customer has made a mistake&quot;.<p>Instead, make it easy and quick for anyone to click a button and get their account changed from Free to Paid, without staff engagement.  Give customers easy agency to control their experience.<p>&gt; Billing “Limits” don’t exist. Budgets are at least a day late.<p>That&#x27;s insane.  Clouds are about speed and dynamic scalability.  Mistakes can ramp up the bill an crazy amount in a short period of time, as Sudeep found out.<p>How is a 24 hour delay in billing sync and budget warnings even remotely acceptable to them &#x2F; Sudeep &#x2F; customers?<p>Sure it&#x27;s probably fine for the 90% cases, but that&#x27;s crippling for the 10% and even if you decide you really don&#x27;t give a crap about your customers, you don&#x27;t want the bad press that 10% will likely give you.<p>Picture what financial damage someone might do if they compromised some of your credentials somehow?  You screwed up, credentials got leaked, and you won&#x27;t necessarily know for a <i>day</i> that something has gone wrong, nor will your restrictions kick in?!<p>Billing is the single highest TPS service in any cloud, with Identity often a close second (billing gets requests for every transaction, <i>and</i> internal requests related to ongoing charges).  You need to handle a high rate of requests, with low latency both in request&#x2F;response and processing data received.  It&#x27;s a hard engineering problem, and cloud platforms try to get some of the smartest engineers working on it.  An organisation of Google&#x27;s caliber has more than enough smart engineers to be working on these kinds of hard problems, even by temporary secondment.<p>Quota &#x2F; Limits in a fast changing cloud environment need to be dynamic and responsive.<p>&gt;I knew how to put the case for Google team when they would come back to work in 2 days.<p>How is 2 days even remotely acceptable?  Maybe it&#x27;s just how it&#x27;s written, but it reads like this is just accepted as the way things are.  Why would you even have to carefully work to present your case?<p>Where are the 24x7 response people with the ability to forgive bills?  $72k is chump change for a cloud provider, and especially for a company of the scale of Google.  Give your support agents the tools and authority they need to make reasonable decisions, with some appropriate kind of oversight process, and stick in feedback mechanisms so product managers know what problems customers are having.<p>It&#x27;s not like that would actually have cost them $72k in direct running costs either.  That <i>should</i> have been a near instant no-brainer.  Forgive, move on, and reap the benefits of good customer good will.  That good will will earn you way more profit than forgiving it would have cost.  You&#x27;re investing in their continuing business.  Sometimes those investments will fail, but most of the time they&#x27;ll succeed.<p>&gt;In our case, it differed by 86,585,365.85 %, or 86 million percentage points. Even when the bill was notified to us, Firebase Console dashboard still said 42,000 read+writes for the month (below the daily limit).<p>So it&#x27;s just fake observability?  What&#x27;s the point?  24 hours delay here is nuts, almost to the point of being useless.  It can be hard to calculate these figures out yourselves.  A fast feedback cycle is critical.  As Sudeep here found out, 24 hours is a great way to have zero clue what&#x27;s going on until it&#x27;s too late.  Is there really no other way to get this information more up-to-date?<p>Moving on to part 2:
&gt;I had a team of ~7 engineers&#x2F;interns at this time, and it would take Google about 10 days to get back to us on this incident.<p>Why is a 10 day response time from Google considered even remotely acceptable for a cloud provider?  Your entire platform is down, you&#x27;re working out ways around this situation, stressing about potential bankruptcy, and it&#x27;s just cool with you that it took 10 days for them to make a business&#x2F;life changing decision over what amounts to chump change?<p>These kinds of mistakes happen with clouds, AWS is famous for waving these shock bills from mistakes and it never takes 10 days to get it done.<p>Billing should be the easiest and most obvious thing.  If your cloud provider is creating complicated billing structures, that&#x27;s a problem the cloud provider should be solving, not expecting customers to unravel the mysteries.<p>Companies being spun up to help people navigate your billing should be an alarm call, not something to celebrate or for customers to consider normal.<p>&gt; Fail fast, learn fast with Cloud is a bad idea<p>It shouldn&#x27;t be.  With near immediate feedback you&#x27;d have known straight away that shit was bad, and cut the experiment out before it cost you an arm and a leg.<p>&gt; While creating a Cloud Run service, we chose default values in the service. The max-instances is preset to 1000, and concurrency set to 80 ...... Same goes with Cloud Run! With Concurrency == 60, max_containers == 1000 and each Request taking 400ms, number of requests Cloud Run can handle 9 million requests per minute!<p>Why are the default values that high on a service?  That seems like you&#x27;re asking customers to shoot themselves in the foot.  Where was the look at the negative customer experience side of the equation?  Make it easy for customers to do the right thing.<p>Then the bit that really bugs the crap out of me:
&gt; Thank you Google!<p>He&#x27;s thanking Google for having had an absolutely shitty experience on their platform: 10 days of stress from needless delays in forgiving a trivially small bill, dealings with multiple lawyers, investigating bankruptcy, risk of missing product launch date, working around the clock to dig themselves out of hell...