Question: I am building web applications for my customer's company. At the server side, there will be 2 kinds of server to server network communication.

1. Separated REST API servers making requests among each other.
2. Communication from application load balancers (AWS ALB specifically) to their auto-scaling EC2 instances.

Currently all of these communications use HTTP protocol. Only the user-facing nodes (such as the load balancer or the web server reverse proxy) will serve HTTPS with valid certificates.

The customer ask us to change them all to HTTPS as thet believe that it is the modern best practice to always use HTTPS instead of HTTP anywhere.

I would like to dispute with the customer but I am no security expert. Please help review my understanding below and correct me if I am wrong.

---

In my view, I think the purpose of HTTPS protocol is for being a trusted channel in an untrusted environment (such as the Internet). So I cannot see any benefit of changing the already trusted channel to HTTPS. Further more, having to install certificates to all servers make it difficult to maintain, chances are, the customer will find their application servers broken someday in the future because some server has certificate expired and no one know.

Another problem, if we have to config all the application server, apache for example, behind the load balance to serve HTTPS, then what is the `ServerName` to put inside the `VirtualHost`? Currently we have no problem using the domain name such as `my-website.example.com` for HTTP `VirtualHost`. But if it were to be HTTPS we have to install certificate of `my-website.example.com` to all instances behind the load-balancer? I think it's weird because then we have many servers claiming to be `my-website.example.com`.

Answer: 
Mixing and matching HTTP and HTTPS is not a good idea - you will constantly be juggling configurations.

Usually adding a component into a system should only be done if there is a very specific reason for it - just because someone thought it a good idea is not a specific reason.

I'm not saying that HTTPS is a bad idea - quite the opposite - but you have a lot of learning to do. The model you propose undermines the trust relationship that is the primary reason for using TLS in the first place. You also don't seem to have thought about how to plan your PKI. 

> 
> servers broken someday in the future because some server has certificate expired and no one know
> 
> 
> 

If you are providing the service, then you should be configuring monitoring for the service, including certificate expiry.

It sounds like you are looking for reasons to argue with the approach of rolling out certificates. Reading between the lines here, it seems you are currently lacking the skills and planning you need to implement this.

Yes, it's a lot of work, but that's the business model - you assess the amount of work, the skills you need to acquire and those you can buy in and you charge the customer for that. (Serge highlights the cost of the certificates - but that is the smallest cost in this whole exercise).
Encryption is cheap. Data leak or data loss are not.

Use encryption between servers (and it is even better to use TLS auth between servers).

And when I say cheap, it is cheap even considering the management of the keys and the certificates. It may be reasonable to issue a self-signed and long-lived certs to both servers.

There are a few exceptions to the rule:

1. Either the client or the server are legacy having known SSL/TLS vulnerabilities. It is always better to update the vulnerable code, but we all know it is not always possible. It is sometimes better (still not good, but better) to disable the vulnerable code entirely, run plaintext and mitigate the risks in one way or another.
2. You are exchanging an insane amount of data and/or need an insanely low latency. The encryption may become a bottleneck and/or a resource hog. You may opt to no encryption and also do something else to secure the thing.