Show HN: AvionDB: A Decentralised Database with MongoDB-like Developer Interface
Neat to see this made it to HN! Congrats.<p>So, my major concern with OrbitDB is the use of in-memory indexes to hashes. It&#x27;s a very naive approach for multiple reasons. It just doesn&#x27;t scale well. Node.js handles these particularly poorly, and develops memory leaks in my own experiments with AOILs (append-only immutable logs) inspired by Secure Scuttlebutt.<p>What I mean is, take a look at this code:
<a href="https:&#x2F;&#x2F;github.com&#x2F;orbitdb&#x2F;orbit-db-kvstore&#x2F;blob&#x2F;master&#x2F;src&#x2F;KeyValueIndex.js#L5" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;orbitdb&#x2F;orbit-db-kvstore&#x2F;blob&#x2F;master&#x2F;src&#x2F;...</a><p>And this:
<a href="https:&#x2F;&#x2F;github.com&#x2F;orbitdb&#x2F;orbit-db-docstore&#x2F;blob&#x2F;master&#x2F;src&#x2F;DocumentIndex.js#L5" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;orbitdb&#x2F;orbit-db-docstore&#x2F;blob&#x2F;master&#x2F;src...</a><p>I&#x27;d suggest perhaps building that in SnapDB:<p><a href="https:&#x2F;&#x2F;github.com&#x2F;ClickSimply&#x2F;snap-db" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;ClickSimply&#x2F;snap-db</a><p>Though, there are some really hard limits to file locks &#x2F; mutexes that are needed for a decent persistent K&#x2F;V store. Since, pretty much any database is going to need an index, and an index is just a K&#x2F;V store. So, in essence, if your platform can&#x27;t be relied on to implement a good embedded K&#x2F;V store, it can&#x27;t be relied on to create anything other than toy databases.<p>I considered adding SnapDB to OrbitDB, but it would&#x27;ve required a lot of refactoring to support asynchronous lookup. I eventually realized I could do a lot of what I wanted to accomplish without the elaborate CRDTs like OrbitDB employs to keep an ordered log of information.<p>I would also add that, although it should be obvious, just in case it isn&#x27;t, AvionDB now builds on OrbitDB which itself builds on IPFS. So, that&#x27;s a big consideration for those looking to work with this.<p>(The below is mostly from personal experience and not terribly relevant to AvionDB and OrbitDB)<p>I&#x27;ve been meaning to do a more formal post-mortem on the project, but haven&#x27;t gotten anything fully organized yet. I discovered all this after creating a project a while back for what was essentially a P2P image gallery built with IPFS and Electron. I wrote it in TypeScript, React (with Hooks), Redux Toolkit (which makes Redux a lot less insufferable than it used to be, my god), and SnapDB and OrbitDB (but not together). I also had go-ipfs running in a process managed by the Electron client, and included it with the build.<p>As a result, the app image binary would generally be in excess of 500MB. And this was just our 0.1.0, we hadn&#x27;t gotten to the really good features yet. The performance was decent in the UI, but the app itself couldn&#x27;t scale very well past 1000 images. go-ipfs was the real limiting factor at the time; it would just fail in really bizarre ways, and I&#x27;d have to not only write code for restarting the go-ipfs process in case it died, but also retry requests that didn&#x27;t go through. I suspected it might&#x27;ve been due to me having pubsub enabled, but my ability to debug that was hamstrung by the fact that OrbitDB hard fails without pubsub enabled. I haven&#x27;t tried it with go-ipfs 0.5.0, I think pubsub is enabled by default with that now, and I&#x27;ve already moved on from that implementation.<p>My biggest concern in terms of performance aside from go-ipfs (which was disappointing and unexpected), was the fact that due to the design, there would be multiple copies of the same data in-memory everywhere: the in-memory indexes used by OrbitDB, the data in Redux, the data in the DOM, etc. I&#x27;m able to effectively integrate all of those using a Rust desktop binary. I&#x27;m using Iced for the UI, and I&#x27;ve moved on from bundling IPFS entirely. It allows me to build desktop and WASM binaries from the same codebase.<p>It&#x27;s possible I could probably make the Electron app work better at this point, but there are still the fundamental limitations I pointed out earlier.<p>For me, content-addressable hashes and IPNS signatures are perfectly fine. I value performance (CPU, RAM, and disk) over anything else, primarily because I&#x27;m developing P2P apps, and without a server, the nodes have to do all the work itself, including the UI. That&#x27;s why I moved off of TypeScript and into Rust. PubSub would be nice, and OrbitDB has a decent implementation of Lamport Clocks and CRDTs for decentralized ordering of events, but it&#x27;s just often overkill for most things that could just be content-addressable.<p>---<p>Just to be clear, I&#x27;ve made a lot of claims here, and I&#x27;m hardly an expert on any one of these things, so if I&#x27;ve gotten any of this wrong, I&#x27;d really like to know more. Also, the code I wrote earlier isn&#x27;t available to the public. If I have any say in the matter, it will be locked away within an enormous warehouse, Indiana Jones style, hopefully never to see the light of day again.