Why Is Storage on Kubernetes So Hard?
I don't really think this is a Kubernetes-specific problem.  If you have a million machines, want your database to run on one of them that is selected by some upstream orchestrator, and want the physical SSDs with that data on it to be in the same machine, you're going to have to do some work.  But at the same time, you have to realize that you are doing this to get that tiny last bit of performance (most likely that 99.9%-ile latency) and that that last 0.1% is always the most expensive.  Sometimes you need it, and I get that, but it's not a problem that everyone has.Most applications are not so IOPS limited that they depend on the difference between PCIE request latency and going out over the network to get stuff that's actually stored on a nearby rack.  And in that case what Kubernetes offers (with the help of cloud providers) is fine.  You make a storage class.  You make a persistent volume claim.  Your pod mounts that.  Not all that hard.  If the performance isn't good enough, though, then you have to build something yourself.I am used to a completely different model that we had at Google.  You could not get durable storage in your job allocation.  Everything went through some other controller that did not give you a block device or even POSIX semantics, and you designed your app around that.  If you needed more IOPS you talked to more backends and duplicated more data.Meanwhile in the public cloud world, you get to have a physical block device with an ext4 filesystem that can magically appear in any of your 5 availability zones, provisioned on the type of disk you specify with a guaranteed number of IOPS.  It's honestly pretty good for 90% or even 99% of the things people are using disks for.  (In my last production environment I actually ran stuff like InfluxDB against EFS, the fully-managed POSIX filesystem that Amazon provides.  It did fine.)