Building a Database System in Academia
Yes, we need this - we need more people exploring and building databases. The more people we have tackling interesting problems the better solutions we will develop.<p>No, building a database is not hard. We need to stop propagating this message, because it deters newcomers from trying. And the less people that try, the more of an echo chamber we get (like everybody building on top of Postgres again and again), with no new ideas. This comment on another current HN homepage article sums it up the best: <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=13933572" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=13933572</a> .<p>I&#x27;ll share my story here, since I seem to be an outlier:<p>- In less than 2 years from building a database from scratch (no forking other database), we had a prototype that could save 100M records for $10&#x2F;day: <a href="https:&#x2F;&#x2F;youtu.be&#x2F;x_WqBuEA7s8" rel="nofollow">https:&#x2F;&#x2F;youtu.be&#x2F;x_WqBuEA7s8</a> .<p>- In 2 years we had a system that can outperform Redis on the same test on the same hardware: <a href="https:&#x2F;&#x2F;github.com&#x2F;amark&#x2F;gun&#x2F;wiki&#x2F;100000-ops-sec-in-IE6-on-2GB-Atom-CPU" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;amark&#x2F;gun&#x2F;wiki&#x2F;100000-ops-sec-in-IE6-on-2...</a><p>- Within 2.5 years we&#x27;ve built a system that can do on low end hardware across a federated system 1750 table inserts a second (we hope to get this up to 10K inserts&#x2F;sec across a distributed system), a slower demo here: <a href="https:&#x2F;&#x2F;www.dropbox.com&#x2F;s&#x2F;amjlr5gqk23et51&#x2F;load.gif?dl=0" rel="nofollow">https:&#x2F;&#x2F;www.dropbox.com&#x2F;s&#x2F;amjlr5gqk23et51&#x2F;load.gif?dl=0</a><p>- To do this, we even had to build a fully featured distributed &quot;coordination testing&quot; framework so others can build tests that verify things work even in the harshest of conditions (it is for doing Jepsen like testing), check it out here: <a href="https:&#x2F;&#x2F;github.com&#x2F;gundb&#x2F;panic-server" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;gundb&#x2F;panic-server</a><p>- And we&#x27;ve managed this without locking the data to any particular model. Graphs let you do key&#x2F;value, relational, table, document, or graph type data.<p>This story is important, because it shows what can happen when people just try. Referencing the quote from the other thread: Ivan Sutherland replied, &quot;Well, I didn&#x27;t know it was hard.&quot;