Ask HN: What is the ops architecture like for AAA multiplayer game servers?
This question sounds like it&#x27;s pointed directly at me.<p>However, I can only speak for one AAA gaming company, and my team operate a bit differently than most in the company.<p>My team operates the infrastructure for &quot;Tom Clancy&#x27;s The Division&quot; video game series (1&amp;2).<p>Most of the programming approach is spent on doing the cheapest (in terms of CPU) possible thing, everything is C++<p>Things like: matchmaking will happen ideally on a single machine with no shared state, everything will happen in memory, which is much faster and can be more reliable than any distributed state or failover.<p>(it&#x27;s less reliable if you&#x27;re in matchmaking and the server or service dies; But then everyone&#x27;s client will reconnect to the newly scheduled matchmaking instance and populate in memory state.)<p>We use a lot of windows, nearly every machine that doesn&#x27;t handle state is a windows server. This has pros and cons, from my ops perspective I try to treat windows like cattle, but windows doesn&#x27;t like that. they have their own way of operating fleets of machines which include using SCCM and packaging things. There&#x27;s nice GUI&#x27;s, but we use saltstack and we removed AD, because it was a huge burden to: create a machine, link it to AD, reboot it, finally get a machine worth using.<p>From a dev perspective, Windows is good, IO Completion Ports is superior to the linux epoll in terms of interface and performance, so we can have machines that take 200,000+ connections.<p>How you decide which dedicated server you connect to is up to your client, it does a naive check as it&#x27;s logging in where it will do a TLS handshake with a random gameserver in a region, for each region. (during the login phase we send your client a list of all currently active gameservers and an int to represent the region).<p>This works fine until there&#x27;s packet loss on a particular path because your single ping might be fine but overall your experience could be better elsewhere; if you&#x27;re not able to ping anything then we fall back on geoip.<p>That said, if you have friends on another datacenter, we try to put you on the same server. So that if you join groups or whatever then it&#x27;s just a phase transition rather than a full server sync.<p>Everything is orchestrated with a &quot;core&quot; set of servers which handle player authentication and matchmaking centrally, then each of the gameserver datacenters (geographically distributed to be closer to players) is attached via ipsec VPN.<p>In Division 2 we spread out into GCP as well as having physical machines, so we developed a custom auto-scaler. The autoscaler checks the capacity of a region and how many players are currently in a region, keeps a record over 10 minutes and makes a prediction. If the prediction goes over the current capacity in 20 minutes or less, it will create a new instance (since making windows servers on GCP takes longer than linux servers)<p>If the prediction goes lower than the capacity of a server, it will send a decomission request to the machine, which takes up to 3hrs to complete (to give people time to leave the server naturally).<p>Idk, I&#x27;ve been doing this for 5 years now so I can talk at length about how we do it, but ultimately out biggest challenges are the fact that we can&#x27;t use cool shit or new trends because latency matters a lot and we use windows everywhere.<p>--<p>As an aside; the overwhelming majority of other ubisoft games (excption: For Honor) use something very similar to what we released open source in collaboration with google to do matchmaking: <a href="https:&#x2F;&#x2F;agones.dev&#x2F;site&#x2F;" rel="nofollow">https:&#x2F;&#x2F;agones.dev&#x2F;site&#x2F;</a>