Cassandra at Apple: 1000s of Clusters, 300k Nodes, 100 PB
The operational complexity of managing thousands of clusters must be mind boggling. I've been on two projects managing dozens of storage clusters, the second with more data than this in some individual clusters and adding up almost this many total nodes. There were technical problems with scaling up to 10K nodes per cluster, but the operational issues mostly scaled according to number of clusters. For example, how many alerts per hour/day can you stand? Too many and you're overwhelmed; too few and you miss stuff. Walking that fine line became successively more difficult as clusters were added. Same thing with graphs and dashboards. Also, when your storage and IOPS are siloed this much you have no elasticity, so you're going to be chasing capacity or load problems much more often. On the plus side, this probably means each tenant has their own cluster, so you don't have so many worries about them affecting each other.The big question I'd have is: how many people (including on client teams) does it take to manage this much sprawl?