File systems unfit as distributed storage back ends: lessons from Ceph evolution
I don&#x27;t think the conventional wisdom of building on top of filesystems exists. In distributed systems you always naturally gravitate towards using raw storage devices instead of filesystems, it becomes obvious very early on that filesystems suck too much and only create problems. And it&#x27;s the same with all the embedded database libraries, you really want to write your own, because none of the existing ones were made to address performance and operational problems that arise even in small distributed systems. But at the same time early on you don&#x27;t yet know most of the problems and don&#x27;t want to invest time implementing something you don&#x27;t yet understand well enough, so you end up building on top of filesystems and embedded databases and making plenty of poor choices and learning on your mistakes.