Ask HN: When is federation a better solution than decentralization?
As someone who has <i>worked exactly with these issues</i> for the past 7 years, let me share what I have learned.<p>Just a bit of background. My first startup, <a href="https:&#x2F;&#x2F;qbix.com" rel="nofollow">https:&#x2F;&#x2F;qbix.com</a> focused on letting each community host its own social network, the same way Wordpress powers blogs. Wordpress powers 30% of the Web so it seemed to be a good business model (and it is). Here is a video explaining that: <a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=pZ1O_gmPneI" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=pZ1O_gmPneI</a><p>Last year we started a new company at <a href="https:&#x2F;&#x2F;intercoin.org" rel="nofollow">https:&#x2F;&#x2F;intercoin.org</a> focused on letting each community manage its own <i>currency</i>. We saw that crypto wasn&#x27;t solving payments as well as WeChat and all the other centralized social networks, so we figured we could fix that. Here is a video explaining that: <a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=HIbAz29L-FA" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=HIbAz29L-FA</a><p>OK, so what have I learned?<p>First of all, federated is much easier. You simply have one source of truth for each stream of data. It&#x27;s why it&#x27;s more prevalent. It also creates &quot;landlords&quot; that you have to trust to host your data, and often their agendas clash with your own. Two months ago I asked why have &quot;communities&quot; at all: <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=16741913" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=16741913</a><p>We are familiar with &quot;landlords&quot; all the time, and their inordinate power within their ecosystems. Consider Amazon&#x27;s bookstore, Apple&#x27;s app store, Google&#x27;s search engine, Facebook&#x27;s social network, and so on. Federating this would be a very good way of introducing <i>choice</i>, and competition among landlords. You still have to choose a landlord, but now the market can improve everyone&#x27;s experience through competition.<p>But it turns out that you can do even better than the market, if you build technology that facilitates <i>collaboration</i> instead of <i>competition</i>. Collaboration has entirely different economics, but ultimately wins in the end. Think about Wikipedia vs Britannica, Linux vs Windows, WebKit vs old IE, and so on.<p>With Collaboration, the infrastructure gets paid for, but not by extracting rents through closed source software. Instead, you distribute the work across many different actors and <i>none</i> of them own their own little fiefdom but they are all intermixed in a grand tapestry. So why would any actor do anything? One answer is that, if they don&#x27;t do it, someone else will, so they get a little bit of control &#x2F; money in the beginning, which fades with time. That part is similar to capitalism. But another part is simply because &quot;the main software is designed to facilitate that mode of collaboration by default&quot;.<p>Also the landlord can be a single point of failure for your data. Consider, for instance, how we store files. Imagine you have no backups in the cloud, and all files are stored in your apartment. Now there&#x27;s a fire. What happened to all your files? They are gone.<p>But if your&#x27;e going to have backups in the cloud, you want to make sure they are encrypted, and only you can access them. Even better would be to blur the distinction between servers run by &quot;the landlord&quot; (even if it&#x27;s your own apartment) and &quot;the cloud&quot; and just make protocols that find the best server, have it be in a consensus group, encrypt everything end-to-end, and allow reading <i>and</i> writing by multiple parties.<p>This is <i>very hard</i> to get right. But any such platform would allow people to build communities <i>on top of</i> a reliable foundation, where only members in the communities would be able to access information, and they could have validators who vouch for what actions (e.g. in a game) are valid. Clients that don&#x27;t agree with a validator can ignore it, and thus the consensus may sometimes split (e.g. a hard fork), with each client choosing which validator to listen to.<p>This is a strict improvement in terms of resilience and control, removing arbitrary landlord relations, and capable of reducing back to them as a special case. It&#x27;s just harder to implement.<p>Consider, for example, why we have domains and DNS. I have files, why do I have to &quot;choose Google Drive&quot; to store my data, and then trust that they will store it, encrypt it, and so on? Why do we have to choose a domain at all? There is a technique called Kademlia, invented in 2002, which was used to power DHTs including BitTorrent and now used in SAFENetwork. SAFENetwork is probably the ultimate example of a benign &quot;landlord&quot; - a network which encrypts and stores your files redundantly, and prevents them being taken down. (Qbix is moving towards the same thing (<a href="https:&#x2F;&#x2F;qbix.com&#x2F;blog" rel="nofollow">https:&#x2F;&#x2F;qbix.com&#x2F;blog</a>) except with dynamic data instead of static files.<p><a href="https:&#x2F;&#x2F;maidsafe.net&#x2F;docs&#x2F;Safe Network Primer.pdf" rel="nofollow">https:&#x2F;&#x2F;maidsafe.net&#x2F;docs&#x2F;Safe Network Primer.pdf</a><p>I would say the future is:<p><pre><code>  End-to-end encryption
  Kademlia (or similar) for routing
  Consensus groups for each data stream
  Churn (backend groups constantly change to prevent collusion)
  Proof of resource (every so often, within groups)
</code></pre>
On top of this, people can build apps that can spread phone-to-phone, taking advantage of a dynamic data layer that lets anyone start their own &quot;stream&quot; and not worry about the arbitrary decision of &quot;which domain hosts it&quot;. The participants get determine the rules of each stream (democracy? dictatorship?) and the landlord is just someone they give access to as well. A search engine would just be a guest in many streams.<p>Do I think Federated systems and market mechanisms are great? Yes. Do I think Distributed systems acting as totally benign landlords are better? Oh yes!<p>They are more resilient, too. For a crypto-currency, that is a must, or you can have this: <a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=E1d5VvCa8Fo" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=E1d5VvCa8Fo</a> (they who know this movie know what happened there.)<p>But they are way harder to do. It took us over 1 year at Intercoin just to come up with the right design. If you want to know more about this topic, you can read about our design decisions (and economics etc) here:<p><a href="https:&#x2F;&#x2F;forum.intercoin.org&#x2F;" rel="nofollow">https:&#x2F;&#x2F;forum.intercoin.org&#x2F;</a><p>Also feel free to email me (greg is the username, intercoin.org is the domain) if you have more questions or want to discuss this topic, I geek out on it. Here is a video I made about decentralizing more than just software:<p><a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=WzMm7-j7yIY" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=WzMm7-j7yIY</a>