Being on call sucks
So I&#x27;ve been oncall at two major companies (Google and Facebook) and, at least in my experience, this covered both ends of the spectrum. Basically, Google gets it mostly right and Facebook gets it mostly wrong.<p>At Google, a new service has to be supported by the team that developed it. There&#x27;a an extensive launch checklist that includes monitoring, having a runbook, etc. Here&#x27;s the most important part: you&#x27;re paid when you&#x27;re oncall. The amount varies depending on how important the service is and the expected response time but can easily be 5 figures a year. Oncall period varies but a week at a time varies with hopefully 8-12 people in rotation.<p>Too few people and people get burnt out. Even if nothing happens on an oncall shift, it&#x27;s an annoyance and a restriction on what you can do. Too many and people tend to forget what to do. So with a sufficiently large team you may end up with some people in the rotation and some people not. That&#x27;s why the compensation is importatnt.<p>Particularly large, important and mature services may enjoy SRE support. You can&#x27;t throw a service over the fence and have SRE deal with it. It doesn&#x27;t work that way. It typically needs to have been running for at least 6 months and SRE needs to be satisified it&#x27;s sufficiently reliable, stable and monitored with a good runbook. SRE support is globally distributed and <i>typically</i> means 8 hour shifts during normal hours.<p>The owning team will often still be secondary support.<p>Also code has to be owned by somebody. This may be a team but when I was there (some years ago now so it may have changed) this also meant 2 actual people (not just team aliases) had to be owners. This is to avoid abandonware. This very much is a support and oncall issue.<p>Facebook OTOH is a dumpster fire when it comes to oncall.<p>Not getting paid to be oncall is (IMHO) one of the biggest mistakes. The mantra is &quot;it&#x27;s part of the job&quot; but that responsibility is not shared equally. That&#x27;s the point of compensation.<p>My experience at Google was that issues were relatively infrequent. What I saw at FB however was that oncall could often be the only thing you did for the week. Noisy alerts, alerts caused by issues in downstream systems that you could do nothing about or would get ignored by their oncall, a bunch of issues raised that some would just ignore until they expired (or closed just prior to going out of SLA as &quot;could not reproduce&quot;), etc. You may also be dealing with code that nobody owns (or, rather, nobody takes responsibility for) for features that are live.<p>Plus the incentive structure, at least on the product side, was to ship new features. Oncall was often treated as just extra work you have to do on top of whatever else you&#x27;re doing.<p>Obviously I didn&#x27;t see how every team did it so none of this is absolute but I did see a reasonably high number of samples.<p>It&#x27;s also worth noting that not everything at FB is like this (eg the Web Foundation people were and I believe still are outstanding). Also, in high-visibility outage situations you have highly knowledge individuals who can and do get involved and know the right people to push.<p>The FB equivalent of SREs is Production Engineers (&quot;PEs&quot;). There are less of these and more services at FB are supported by the SWEs than at Google (IME).<p>I got the impression that FB processes and culture were forged when the company had less than 500 employees and they never really adjusted to the greater scale. There are a lot of things that work very well. Oncall just isn&#x27;t one of them. Nor is code ownership.