Question: I am building web applications for my customer's company. At the server side, there will be 2 kinds of server to server network communication.

1. Separated REST API servers making requests among each other.
2. Communication from application load balancers (AWS ALB specifically) to their auto-scaling EC2 instances.

Currently all of these communications use HTTP protocol. Only the user-facing nodes (such as the load balancer or the web server reverse proxy) will serve HTTPS with valid certificates.

The customer ask us to change them all to HTTPS as thet believe that it is the modern best practice to always use HTTPS instead of HTTP anywhere.

I would like to dispute with the customer but I am no security expert. Please help review my understanding below and correct me if I am wrong.

---

In my view, I think the purpose of HTTPS protocol is for being a trusted channel in an untrusted environment (such as the Internet). So I cannot see any benefit of changing the already trusted channel to HTTPS. Further more, having to install certificates to all servers make it difficult to maintain, chances are, the customer will find their application servers broken someday in the future because some server has certificate expired and no one know.

Another problem, if we have to config all the application server, apache for example, behind the load balance to serve HTTPS, then what is the `ServerName` to put inside the `VirtualHost`? Currently we have no problem using the domain name such as `my-website.example.com` for HTTP `VirtualHost`. But if it were to be HTTPS we have to install certificate of `my-website.example.com` to all instances behind the load-balancer? I think it's weird because then we have many servers claiming to be `my-website.example.com`.

Answer: 
The answer to your question comes down to threat modeling. Using cryptographic protocols like HTTPS is a security mechanism to protect against certain threats. If those threats are relevant for you, must be analyzed:

* **Are there potential threat actors in your internal network?** Based on your question you seem to assume that the internal network can be fully trusted. This is often a misconception, because there are several ways your internal network can be compromised (e.g. valid users with access to this network are turning malicious, a systems in this network gets compromised, a misconfiguration opens up the network segment, etc.).
* **Will the architecture be subject to change?** It is likely that the system will change over time and prior security assumptions (e.g. my internal network is trusted) no longer hold. If that's a reasonable scenario, it might be a good idea to build the necessary security mechanism in in advance. That's what security best-practices are for. Providing security in an area of uncertainty.
* **Is there a regulatory, legal or compliance requirement that must be fulfilled?** You said that your customer considers HTTPS to be state-of-the-art / modern best-practice. The source of this friendly worded statement might actually be an externally driven requirement, that must be fulfilled. Non-compliance is a threat that should also be covered in a threat analysis.

Those are important topics worth analyzing. When I design system architectures and I am in doubt, I prefer to err on the side of security. In this case the best-practice approach is indeed using HTTPS for communication, no matter the circumstances, as long as there are isn't a considerable impact on the application (e.g. performance impact).

Difficulty to maintain server certificates shouldn't be a problem nowadays, as this is common practice. This should be part of normal scheduled operations activity.

Having said all this, there is of course additional effort required to use HTTPS instead of HTTP and it is your right to charge the customer for this additional effort. I suggest you calculate what this will cost during development and over time during operation and let the customer decide if the cost is worth the benefit.
"https" not only secures the communications as it passes over the network, but also verifies the *certificate* that is presented by the server. This enables you to know that you really are talking to the correct site. And this, in my opinion, is the real advantage of "https."

"letsencrypt.org," which issues signed certificates for free, has a lot of material on their web-site which discusses these benefits. They argue ... quite rightly, I think ... that "everything should be https, whether the material is actually sensitive or not."

(`mod_ssl` *et al* are also capable of enforcing certificate possession on the *client* side, too, although this is rarely done. For a secure internal application, however, you might want to do such a thing. The server is then able to restrict which computers are capable of connecting to it at all, restricting them by the credentials *they* must possess.)