How we achieved write speeds of 1.4M rows per second
I used to work with a system that sent a good amount of real time updates like these and with a lot of jitter. What I don’t get from this blog post (which is very well written, by the way. Thank you!), is if you have all these constraints on your data why do you need to create a system this complicated? Your incoming data array lives purely in memory and you create a local B-tree (or any other index type you like) also in memory. You don’t need to move anything because once you need to read the data, you just read the index. Every 10 seconds a process or a thread reads the ordered data with a FLUSH bit set which tells this merge queue to empty everything out.The process that calls the FLUSH simply appends more data to a file, relying on tunable mechanics the OS provides (i.e. the commit interval).The reader of the data that is responsible for querying the data based on user requests would then have to use some kind of an index to locate the correct range of data from the appended-to files. A radix seems to be an excellent choice here.Any data that arrives more than 10 seconds late is discarded. This would of course have or be a tunable parameter.Isn’t that all you need from a system like this?