Question: I am building web applications for my customer's company. At the server side, there will be 2 kinds of server to server network communication.

1. Separated REST API servers making requests among each other.
2. Communication from application load balancers (AWS ALB specifically) to their auto-scaling EC2 instances.

Currently all of these communications use HTTP protocol. Only the user-facing nodes (such as the load balancer or the web server reverse proxy) will serve HTTPS with valid certificates.

The customer ask us to change them all to HTTPS as thet believe that it is the modern best practice to always use HTTPS instead of HTTP anywhere.

I would like to dispute with the customer but I am no security expert. Please help review my understanding below and correct me if I am wrong.

---

In my view, I think the purpose of HTTPS protocol is for being a trusted channel in an untrusted environment (such as the Internet). So I cannot see any benefit of changing the already trusted channel to HTTPS. Further more, having to install certificates to all servers make it difficult to maintain, chances are, the customer will find their application servers broken someday in the future because some server has certificate expired and no one know.

Another problem, if we have to config all the application server, apache for example, behind the load balance to serve HTTPS, then what is the `ServerName` to put inside the `VirtualHost`? Currently we have no problem using the domain name such as `my-website.example.com` for HTTP `VirtualHost`. But if it were to be HTTPS we have to install certificate of `my-website.example.com` to all instances behind the load-balancer? I think it's weird because then we have many servers claiming to be `my-website.example.com`.

Answer: 
As a professional, you owe advises to your client, but should not take the decision yourself.

The arguments to present to your client are:

* what is the gain in using HTTPS inside the server network? If this network is isolated from any other system and only sysadmins can access it, you may argue the the gain can be neglected because it is just protecting a system against someone which already has admin priviledges. If other staff members with no admin priviledges can access it, the gain is not null, neither it is is systems for other clients can.
* what is the risk to do it? The disgression on certificates is mainly... a disgression. But the fact is that HTTPS is a more complex protocol that HTTP is, and any added complexity adds risks for implementation errors. If the previous step concluded that the gain is neglectable, that is enough to advise the client not to do.
* what would be the added cost? Here you have to considere direct and indirect costs. Direct costs could include the price of additional certificates if you use external ones, or the time for creation of private certificates if you use a private PKI. They would also include the time for the configuration of the system. And they should include the maintenance times as recurrent costs, including a programmed renewal of certificates - this part **is** in you responsability domain, but you can charge your client for the time. Indirect costs are harder to establish but you should use your own experience to evaluate the risk of errors due the the added complexity and their possible consequences. And IMHO you may charge your client for that if they insist in not following your advices.

But when you have said that all, the client is responsable for the decision.
Encryption is cheap. Data leak or data loss are not.

Use encryption between servers (and it is even better to use TLS auth between servers).

And when I say cheap, it is cheap even considering the management of the keys and the certificates. It may be reasonable to issue a self-signed and long-lived certs to both servers.

There are a few exceptions to the rule:

1. Either the client or the server are legacy having known SSL/TLS vulnerabilities. It is always better to update the vulnerable code, but we all know it is not always possible. It is sometimes better (still not good, but better) to disable the vulnerable code entirely, run plaintext and mitigate the risks in one way or another.
2. You are exchanging an insane amount of data and/or need an insanely low latency. The encryption may become a bottleneck and/or a resource hog. You may opt to no encryption and also do something else to secure the thing.