While at NDC Oslo, Jon talked to Robert Friberg about in-memory databases in general. They discussed OrigoDB, Robert’s open source in-memory database, as well as Redis and SQL Server Hekaton.
(00:18) Jon asks Robert what his presentation was about. Robert describes his talk covering in-memory databases, comparing OrigoDB, Redis and Heketon.
(00:40) OrigoDB is Robert’s open source project. Jon asks why Robert decided to write his own in-memory database when there others available. Robert says that OrigoDB is unique, and that they’ve been working on it for a long time and there was nothing available when they started on it.
(01:18) Jon asks how it compares to relational databases. Robert says that when you move to in-memory, you’re no longer constrained by the need to structure your data in a way that can be easily stored on disk. You can take advantage of the random access nature of memory, and thing more graph oriented than stream oriented.
(02:07) Jon asks if the data is persisted and follows ACID principles. Robert says that the data is in memory in any form you like, and you log transactions when you make changes – writing the events to disk or a database.
(02:48) Jon asks how data is loaded when an application starts up. Robert says it loads the most recent snapshot and then replays all events that occurred after the snapshot. You can choose the serialization format – JSON, binary and Protocol Buffers (protobuf) are supported. Protocol Buffers fast, compact and interoperable.
(03:45) Jon asks what kinds of applications work best with OrigoDB. Robert describes the problem it solves: data access and databases are too slow, so we need to use caching to compensate for that. Traditional relational databases were a good fit when memory was scarce, but now your entire application’s data can fit in memory. Also, historically, relational databases reflected the entity model and allowing business users to run queries; now we’re mapping back and forth between models which don’t match. If you keep all the data in memory, everything’s in one place and the data access is incredibly fast. That allows you to do everything single-threaded, really quickly – on Robert’s laptop he can do 50,000 ACID transactions per second.
(07:35) Jon asks what the difference is between using OrigoDB and just using his own in-memory structures. Robert explains that’s how OrigoDB works – you use your own in-memory structures and do LINQ queries against them. OrigoDB adds in snapshotting, persistence, etc.
(08:22 ) Robert says that in addition to the embedded engine, they also have an out-of-process server product that supports clustering with replication, load balancing, and off-site backup.
(09:12) Jon asks how the clusters communicate. Robert says that it’s via TCP, inspired by SQL Server clustering.
(09:50) Jon asks what happens when the master goes offline. Robert says that there’s no automatic failover, but you can do manual failover using the web-based UI.
(12:35) You work with OrigoDB using commands and queries. You can also send in LINQ commands as text and they’ll be compiled, parameterized and cached.
(13:17) Jon asks how much memory it will take. Robert says over the past twenty years, the transactional workloads for the projects he’s worked on have all been under 200GB. You can offload your reporting data to a relational database if needed.
(14:18) Jon asks if the server product is commercial software. Robert said they’ve tried the revenue model but haven’t had any sales. In the next release, they’re pivoting to everything free and open source and trying to build a support business.
(15:15) Jon asks what kinds of projects he’s built with OrigoDB. Robert talks about a consulting job for a large healthcare company in Sweden. The customer was having really bad performance problems – each service would create business components, which would then create data components. Due to the business requirements, the data transactions were complex, and many were written using cursors. Robert said he traced some database use and found that a single transaction could make thousands of database round-trips. Robert did a proof of concept using simple C# collections in-memory and found they could do tens of thousands of transactions per second. Robert says that transactions in SQL Server require logging pages of data to disk, whereas logging an OrigoDB transaction is often just a few bytes since it’s just logging the command.
(18:18) Jon asks how many snapshots are maintained. Robert says he tries to avoid snapshots since they require a read lock. You can also use an immutable model (using multi-version cursor control).
(19:21) You can truncate events when you snapshot, but then you’re losing information. Robert and Jon discuss how this relates to event sourcing.
(20:02) Jon asks what Robert showed off in his talk. Robert says he normally does workshops that are a few days long, so squeezing everything into an hour is difficult. He does demonstrations showing OrigoDB, Redis and Hekaton, but his main message is that your application’s data probably fits in memory and memory is cheap.
(21:03) Jon asks about Hekaton. Robert explains how Hekaton works, pointing out that it supports a hybrid model in which only certain tables are in-memory. The advantage is that you can use your existing SQL Server tools, ecosystem and code.
(23:20) Robert mentions VoltDB, a startup that offers an in-memory, distributed relational database engine.
Redis is a key-value store. Most people use it as a cache, but the values in themselves are structures, so a value can be a hashtable, list, queue, sorted set, etc. There are predefined commands that kind of look like assembler.
(24:45) Jon says he remembers running into some objects that were difficult to serialize. Robert says that the default formatter for OrigoDB is the binary formatter, and you have to mark your objects as serializable. If you use Protocol Buffers require you to define a mapping.
While at NDC Oslo, K Scott and Jon talked to Chris Klug about pragmatically applying SOLID principles and on moving from Silverlight to HTML5.
(00:18) Jon says hi to Chris Klug and mispronounces his name and feels bad about it but Chris is understanding and lets it go.
(00:27) Chris has been doing SOLID talks for a while, but this time he spoke about when to use SOLID principles and when not to. However, open-closed and Liskov substitution principle are both examples that might not fit into all scenarios. Chris says if you go full-SOLID you might never ship anything… and shipping is a good feature, too.
(01:28) Jon asks Chris if he gets some pushback. Chris says he does, from both sides.
(02:20) K Scott asks Chris about how he applies the Single Responsibility Principle.
(05:44) K Scott asks Chris if he’d always use TypeScript. Chris says maybe not for simple projects, since TypeScript makes things a little more complicated due to the transpiler step. But the type safety’s really nice for any larger project.
(06:28) Jon says he thinks that Silverlight was a good bridge to HTML5 since it let you get started with things like video on the web long before HTML5 supported them. Chris kind of agrees, but points out that HTML5 video still doesn’t support DRM, and he still prefers XAML to HTML and CSS.
(08:22) Jon asks what Chris would recommend to developers who are starting on the transition from XAML to HTML5. Chris says just jump in and get started – probably using Angular and TypeScript. It’s really not that hard once you get started.
(09:00) Jon asks Chris what he does for fun in his free time. Chris says he’s a kite boarder, which is a little tricky because he lives in Stockholm.
(09:42) Jon asks Chris about his picture from the NDC speakers page, and Chris tells the story.
(10:24) K Scott asks Chris about the best spots he’s been kite boarding, and Chris mentions a few in South Africa.