A Second Conversation with Werner Vogels
&gt; <i>A few things around S3 were unique. We launched with a set of ten distributed systems tenets in the press release.  The ten tenets were separate from S3, stating that this is how you would want to build distributed systems at scale. We just demonstrated that S3 is a really good example of applying those skills.</i><p>In my own naive way I believe that AWS, today, probably embodies James Hamilton&#x27;s vision [0] far more than most of those original 10 tenets.<p>-<p>&gt; <i>One of the keys to the success of S3 was that, at launch, it was as simple as it possibly could be, offering little more than GetObject and PutObject. At the time that was quite controversial, as the offering seemed almost too bare bones... most technology companies at the time were delivering everything and the kitchen sink, and it would come with a very thick book and 10 different partners that would tell you how to use the technology. We went down a path, one that Jeff [Bezos] described years before, as building tools instead of platforms.</i><p>Most folks don&#x27;t realise but starting somewhere around 2002 &quot;Web Services&quot; (SOA, UDDI, WSDL etc) was really hot and every company including Yahoo!, Google, and Microsoft were betting big on it. Amazon simply turned around and took a radically different approach to it all. This describes why Azure&#x27;s initial push in 2008 [1] and Google&#x27;s attempts with AppEngine [2] fell relatively short, even post AWS&#x27; IaaS play.<p>-<p>&gt; <i>We needed to build the right tools to support that rate of radical change in how you build software... you have to work with your customers, to wait to see how they are using your tools and see what they do. So, we sat down and asked, &quot;What is the minimum set?&quot;</i><p>I think, this might one of the key differentiators in how AWS thinks about Cloud versus the rest. AWS v1 launches are sometimes laughably inadequate, bare, buggy; and in some cases continue to remain so, but often is the case that the products take-off in all sorts of directions as the feedback trickles in and the roadmap is reshaped in response to it. James Hamilton talks about this as he contrasts AWS&#x27; approach with &quot;other IT companies&quot; (which, I like to think, is him talking about his former employer, Microsoft). [3]<p>-<p>&gt; <i>You have to be really consciously careful about API design. APIs are forever. Once you put the API out there, maybe you can version it, but you can&#x27;t take it away from your customers once you&#x27;ve built it like this.</i><p>GCP&#x27;s triggered? On a serious note, I was part of a team that &quot;deprecated&quot; a public-facing AWS API (an entire service, really) and the world didn&#x27;t even notice because the team live-migrated all customers to a newer API (service), grand-fathered their (cheaper) pricing model in, left all their historical artifacts intact, ran both older and newer deployments side-by-side just in case, and so on... A painfully slow process for engineers, but I guess, customers must love it?<p>-<p>&gt; <i>Being conservative and minimalistic in your API design helps you build fundamental tools on which you may be able to add more functionality, or which partners can build layers on top of, or where you can start putting different building blocks together.</i><p>Here&#x27;s a fun presentation from re:Invent 2014 on the distributed systems primitives (aka &quot;building blocks&quot;) that AWS engs built early-on for S3, EC2, DDB etc: &quot;Under the Covers of AWS&quot;, Al Vermeulen and Swami Sivasubramanian, <a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=QVvFVwyElLY" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=QVvFVwyElLY</a><p>-<p>&gt; <i>...in December 2004, we decided to take a deep look at how we were using storage, and it turned out that 70 percent of our usage of storage was key-value. Some of those values were large, and some were really small. One of those drove in the direction of Dynamo, in terms of small keys, a table interface, things like that, and the other one became S3, with S3 more as a blob and bigger value store, with some different attributes.</i><p>Hm, storing really small objects in S3 is an anti-pattern to this day (look at how slow the <i>List</i> operation becomes once you do this to your buckets) [4], we walk into this landmine all the time.<p>-<p>&gt; <i>I think that much of this conversation is going to be about evolvability... &quot;A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system.&quot;... It doesn&#x27;t hurt that much if you make long-term decisions on very simple interfaces, because you can build on top of them. Complex systems are much harder to evolve.</i><p>Demonstrated by AWS designing a simpler system from scratch because they couldn&#x27;t evolve existing complex systems: SDB and DDB; CF and CDK; SWF and Step Functions to name a few. One of the few times Joel&#x27;s Law [5] is justifiably worth breaking.<p>-<p>&gt; <i>...mostly a back-and-forth with our customer base about what would be the best interface—really listening to people&#x27;s requirements. And to be honest, immutability was a much bigger requirement than having a distributed lock manager, which is notoriously hard to build and operate. It requires a lot of coordination among different partners, and failure modes are not always well understood. So, we decided to go for a simpler solution: object versioning.</i><p>This is again, in my eyes, one of the unique aspects of Amazon&#x27;s approach to acting on customer feedback. The management tends to always favour inventing simple solutions that can be accomplished with fewest possible resources [6][7].<p>-<p>&gt; <i>When we designed S3, one of its powerful concepts was separating compute and storage.</i><p>Separating compute and storage is how Amazon Aurora came about as well. I wonder if Lambda then is about separating compute and the CPU (server utilization, to be precise) [8]?<p>-<p>&gt; <i>...the team &quot;is completely responsible for the service—from scoping out the functionality, to architecting it, to building it, and operating it. You build it, you run it. This brings developers into contact with the day-to-day operation of their software.&quot;</i><p>Interesting contrast with SRE orgs. Besides, a lot of new joinees I knew at Amazon weren&#x27;t really happy in eng teams where operations load was heavy because they&#x27;d feel they didn&#x27;t really build those services. That&#x27;s the only downside I could think of. Werner&#x27;s right though that &quot;you build it, you own it&quot; does inculcate in everyone a deep empathy for their customer&#x27;s pain-points. [9]<p>----<p>[0] <a href="https:&#x2F;&#x2F;mvdirona.com&#x2F;jrh&#x2F;talksAndPapers&#x2F;JamesRH_Lisa.pdf" rel="nofollow">https:&#x2F;&#x2F;mvdirona.com&#x2F;jrh&#x2F;talksAndPapers&#x2F;JamesRH_Lisa.pdf</a><p>[1] <a href="https:&#x2F;&#x2F;www.zdnet.com&#x2F;article&#x2F;microsofts-azure-cloud-platform-a-guide-for-the-perplexed&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.zdnet.com&#x2F;article&#x2F;microsofts-azure-cloud-platfor...</a><p>[2] <a href="http:&#x2F;&#x2F;highscalability.com&#x2F;google-appengine-first-look" rel="nofollow">http:&#x2F;&#x2F;highscalability.com&#x2F;google-appengine-first-look</a><p>[3] <a href="https:&#x2F;&#x2F;perspectives.mvdirona.com&#x2F;2016&#x2F;03&#x2F;a-decade-of-innovation&#x2F;" rel="nofollow">https:&#x2F;&#x2F;perspectives.mvdirona.com&#x2F;2016&#x2F;03&#x2F;a-decade-of-innova...</a><p>[4] <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=19475726" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=19475726</a><p>[5] <a href="https:&#x2F;&#x2F;www.joelonsoftware.com&#x2F;2000&#x2F;04&#x2F;06&#x2F;things-you-should-never-do-part-i&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.joelonsoftware.com&#x2F;2000&#x2F;04&#x2F;06&#x2F;things-you-should-...</a><p>[6] <a href="https:&#x2F;&#x2F;www.google.com&#x2F;books&#x2F;edition&#x2F;Working_Backwards&#x2F;jgn5DwAAQBAJ" rel="nofollow">https:&#x2F;&#x2F;www.google.com&#x2F;books&#x2F;edition&#x2F;Working_Backwards&#x2F;jgn5D...</a><p>[7] <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=26180074" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=26180074</a><p>[8] <a href="https:&#x2F;&#x2F;youtu.be&#x2F;dInADzgCI-s?t=541" rel="nofollow">https:&#x2F;&#x2F;youtu.be&#x2F;dInADzgCI-s?t=541</a><p>[9] <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=26252902" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=26252902</a>