Answers: <p>You can do this a couple of ways, but I have never seen anyone do this with a CNAME. </p>
  
  <p>One option, if you have control over your IP addressing is to use <a href="https://www.techrepublic.com/article/how-to-use-bgp-to-achieve-internet-redundancy/" rel="nofollow noreferrer">BGP to fail-over</a> (or load balance) an IP to your cloud hosting providers.</p>
  
  <p>A second option is to invest in a DNS-based geographic load balancing solution, such as the <a href="https://support.f5.com/kb/en-us/products/big-ip_gtm/manuals/product/gtm-concepts-11-3-0/1.html" rel="nofollow noreferrer">GTM from F5</a>, the <a href="https://www.a10networks.com/resources/glossary/global-server-load-balancing-gslb" rel="nofollow noreferrer">GSLB solution from A10</a>, or the <a href="https://kemptechnologies.com/global-server-load-balancing-gslb/" rel="nofollow noreferrer">GSLB solution from Kemp</a> (F5 is the most commonly used in the industry). With these solutions you will set your name server (NS1 and NS2) for your domain to be the GSLB solution of your choice. You will set a very low TTL for your DNS records and configure health monitors to your ELK stack from the GSLB solution. These can be as complex as logging into ELK and scraping for data or as simple as making sure you can establish a TCP connection and/or ping your server. When this health monitor fails, the GSLB solution will failover traffic to the alternate data center automagically. You will be able to do this as an HA pair with one GSLB solution at once DC as NS1 and the other at your secondary site as NS2. The GSLB will then forward lookup to your previous nameservers which will publish IP addresses for both sites. These will be rewritten to eliminate the IP that the GSLB does not want to direct traffic to at the time (again, this can be used for active-active sites - you can even use geolocation to send users in Europe to a Eurpoean cloud provider  and US based users to a US based provider, or do a simple active/standby)</p>
  
  <p>A final way to do this might be with local load balancing solution, such as <a href="https://f5.com/products/big-ip/local-traffic-manager-ltm" rel="nofollow noreferrer">F5's LTM</a>, <a href="https://www.a10networks.com/products/thunder-series/thunder-application_delivery_controller" rel="nofollow noreferrer">A10's SLB</a>, or <a href="https://kemptechnologies.com/" rel="nofollow noreferrer">Kemp's solution</a>. You may also be able to use <a href="https://www.nginx.com/resources/admin-guide/reverse-proxy/" rel="nofollow noreferrer">nginx</a> for this for a cheap solution. In this scenario, you might have this solution onsite in an office with the URL to the load balancer served up by your local DNS server to a local IP (a Virtual IP) on your load balancer. You would then configure the load balance to direct users to the desired cloud provider by configuring a health monitor (of whatever complexity). You would then need to configure this load balancer to bridge your SSL connection. You would add your certificate and private key to the load balancing solution. Users would establish an encrypted connection to the load balancer. This encryption would be stripped off by the load balancer and a new encrypted connection established to the cloud provider by the load balancer, and the data forwarded along over the new SSL tunnel.</p>
  
Answers: CC BY-SA 3.0
Answers: 2018-02-26T09:29:13.697Z
Answers: i have considered many of the options you suggested , but terminating SSL at the LB and sending it on unencrypted is against company policy so not an option 
  
  
  essentially im looking for something that can transparently proxy TLS connections assuming i have access to certificates of both endpoints
Answers: Saar Grin
Answers: CC BY-SA 3.0
Answers: 2018-02-26T12:28:59.303Z
Answers: Load balancing at tcp level works, you don't need access to endpoint certificates unless you're setting up a man in the middle terminating the SSL flow. Moreover, you can be terminating TLs at the load balancer and sending to backend in TLS also.
Answers: Tensibai
Answers: CC BY-SA 3.0
Answers: 2018-02-26T16:20:58.597Z
Answers: @SaarGrin - Yes, you misunderstand. [This solution *bridges*](https://f5.com/glossary/ssl-bridging) the SSL conn and the data is encrypted on both sides of the reverse proxy. It does not terminate (offload) SSL encryption. On an F5 at least this can also be done [*without* re-encapsulating](https://support.f5.com/csp/article/K13385) the traffic in a SSL using SSL. The client establishes an SSL session with the back-end server and the ADC (Application Delivery Controller or load balancer) has the private key and uses it to decrypt and inspect the traffic for the purpose of session persistence.
Answers: James Shewey
Answers: CC BY-SA 3.0
Answers: 2018-02-26T16:24:28.753Z
Answers: @Tensibai - Load balancing at a TCP level won't work if he needs session persistence. Otherwise, you may be logged in to server A, and then your next request goes to server B, which makes you log in again - and may even destroy the login cookie you had for Server A. While you can maintain persistence using the SSL session ID, it is unreliable as session ID are rotated periodically.
Answers: James Shewey
Answers: CC BY-SA 3.0
Answers: 2018-02-26T16:43:23.930Z
Answers: @JamesShewey round robin is not the only load balancing method, ip source hash, ip source/port hash are also valid solution and allow to keep sessions. In case of elasticsearch, each request is a standalone one, so there's nothing problematic to push datas to Elasticsearch with a RR load balancing.
Answers: Tensibai
Answers: CC BY-SA 3.0
Answers: 2018-02-26T17:27:24.660Z
Answers: IP persistence is possible, but has drawbacks - eg if users are coming from behind a firewall, they won't properly distribute. But you are right - I had forgotten that Elasticsearch doesn't need to be able to maintain persistence. That should make nginx a definite contender for this situation.
Answers: James Shewey
Answers: CC BY-SA 3.0
Answers: 2018-02-27T08:45:25.080Z
Answers: @JamesShewey are you aware of any opensource solution that supports this?  
  elastic puts are not session persistent so that really is not an issue
Answers: Saar Grin
Answers: CC BY-SA 3.0
Answers: 2018-02-22T19:36:02.560Z
Answers: 2018-02-23T14:11:28.440Z
Answers: 2018-02-23T14:11:28.440Z
Answers: James Shewey
Body: <p>if i have filebeat -> logstash -> elasticsearch pipeline with TLS or HTTPS encryption , is it possible to load balance between 2 destination elasticsearch clusters in active-standby ?<br>
  what i mean is, in case one of the clusters goes down i would manually or automatically switch all the data streams to the other cluster.  </p>
  
  <p>im looking at various proxy setups but im not sure i understand the terms correctly, and TLS would not solve the problem.</p>
  
Comments: CC BY-SA 3.0
Comments: 2018-02-22T05:34:23.630Z
Comments: That's unclear what you're after here.. Can you explain more what is in each cluster?
Comments: Tensibai
Comments: CC BY-SA 3.0
Comments: 2018-02-22T10:01:37.427Z
Comments: the situation is that currently we ship log data with logstash to a cloud based ELK . so we have the certificates of the destination installed on the client side .  what i want is to be able to switch to another ELK cluster located in another cloud by changing the DNS cname .
Comments: Saar Grin
Comments: CC BY-SA 3.0
Comments: 2018-02-22T10:07:21.217Z
Comments: And you don't own the certificate to set  it up on the other cluster ?
Comments: Tensibai
Comments: CC BY-SA 3.0
Comments: 2018-02-26T09:32:56.683Z
Comments: so basically we now ship data to a commercial ELK SaaS from windows/linux machines with logstash 
  i need to be able to switch over to another ELK SaaS provider without having to reconfigure the logstash or OS on which logstash is installed  
  the idea is to set up a permanent cname on our domain that will currently forward to our current vendor but may be switched over to another vendor  
  the problem is the system is using TLS transport so if the destination hostname doesnt match it wont work, therefore im looking for proxy that is able to transparently load-balance the TLS connection
Comments: Saar Grin
Comments: CC BY-SA 3.0
Comments: 2018-02-26T12:27:08.850Z
Comments: Or just ask your SaaS provider to install a certificate you'll be giving matching your cname...
Comments: Tensibai
Comments: CC BY-SA 3.0
Comments: 2018-02-26T12:31:23.813Z
Comments: We did ask. They don't have that feature available right now and cr is expensive
Comments: Saar Grin
Tags: <load-balance><logstash><ssl><elastic>
Title: Is it possible to load balance TLS/SSL based logstash communication to elasticsearch?
UserDisplayName: Saar Grin
ThreadText: # Is it possible to load balance TLS/SSL based logstash communication to elasticsearch?
  
  (Asked by: username_0 on 2018-02-21)
  
  if i have filebeat -> logstash -> elasticsearch pipeline with TLS or HTTPS encryption , is it possible to load balance between 2 destination elasticsearch clusters in active-standby ?  
  what i mean is, in case one of the clusters goes down i would manually or automatically switch all the data streams to the other cluster.
  
  im looking at various proxy setups but im not sure i understand the terms correctly, and TLS would not solve the problem.
  
  **Tags:** load-balance, logstash, ssl, elastic
  
  > **Comments:**
  > **username_1**: That's unclear what you're after here.. Can you explain more what is in each cluster?  
  > **username_0**: the situation is that currently we ship log data with logstash to a cloud based ELK . so we have the certificates of the destination installed on the client side .  what i want is to be able to switch to another ELK cluster located in another cloud by changing the DNS cname .  
  > **username_1**: And you don't own the certificate to set  it up on the other cluster ?  
  > **username_0**: so basically we now ship data to a commercial ELK SaaS from windows/linux machines with logstash  i need to be able to switch over to another ELK SaaS provider without having to reconfigure the logstash or OS on which logstash is installed   the idea is to set up a permanent cname on our domain that will currently forward to our current vendor but may be switched over to another vendor   the problem is the system is using TLS transport so if the destination hostname doesnt match it wont work, therefore im looking for proxy that is able to transparently load-balance the TLS connection  
  > **username_1**: Or just ask your SaaS provider to install a certificate you'll be giving matching your cname...  
  > **username_0**: We did ask. They don't have that feature available right now and cr is expensive  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2018-02-22. Score: 1
  
  You can do this a couple of ways, but I have never seen anyone do this with a CNAME.
  
  One option, if you have control over your IP addressing is to use [BGP to fail-over](https://www.techrepublic.com/article/how-to-use-bgp-to-achieve-internet-redundancy/) (or load balance) an IP to your cloud hosting providers.
  
  A second option is to invest in a DNS-based geographic load balancing solution, such as the [GTM from F5](https://support.f5.com/kb/en-us/products/big-ip_gtm/manuals/product/gtm-concepts-11-3-0/1.html), the [GSLB solution from A10](https://www.a10networks.com/resources/glossary/global-server-load-balancing-gslb), or the [GSLB solution from Kemp](https://kemptechnologies.com/global-server-load-balancing-gslb/) (F5 is the most commonly used in the industry). With these solutions you will set your name server (NS1 and NS2) for your domain to be the GSLB solution of your choice. You will set a very low TTL for your DNS records and configure health monitors to your ELK stack from the GSLB solution. These can be as complex as logging into ELK and scraping for data or as simple as making sure you can establish a TCP connection and/or ping your server. When this health monitor fails, the GSLB solution will failover traffic to the alternate data center automagically. You will be able to do this as an HA pair with one GSLB solution at once DC as NS1 and the other at your secondary site as NS2. The GSLB will then forward lookup to your previous nameservers which will publish IP addresses for both sites. These will be rewritten to eliminate the IP that the GSLB does not want to direct traffic to at the time (again, this can be used for active-active sites - you can even use geolocation to send users in Europe to a Eurpoean cloud provider and US based users to a US based provider, or do a simple active/standby)
  
  A final way to do this might be with local load balancing solution, such as [F5's LTM](https://f5.com/products/big-ip/local-traffic-manager-ltm), [A10's SLB](https://www.a10networks.com/products/thunder-series/thunder-application_delivery_controller), or [Kemp's solution](https://kemptechnologies.com/). You may also be able to use [nginx](https://www.nginx.com/resources/admin-guide/reverse-proxy/) for this for a cheap solution. In this scenario, you might have this solution onsite in an office with the URL to the load balancer served up by your local DNS server to a local IP (a Virtual IP) on your load balancer. You would then configure the load balance to direct users to the desired cloud provider by configuring a health monitor (of whatever complexity). You would then need to configure this load balancer to bridge your SSL connection. You would add your certificate and private key to the load balancing solution. Users would establish an encrypted connection to the load balancer. This encryption would be stripped off by the load balancer and a new encrypted connection established to the cloud provider by the load balancer, and the data forwarded along over the new SSL tunnel.
  
  > **Comments:**
  > **username_0**: i have considered many of the options you suggested , but terminating SSL at the LB and sending it on unencrypted is against company policy so not an option    essentially im looking for something that can transparently proxy TLS connections assuming i have access to certificates of both endpoints  
  > **username_1**: Load balancing at tcp level works, you don't need access to endpoint certificates unless you're setting up a man in the middle terminating the SSL flow. Moreover, you can be terminating TLs at the load balancer and sending to backend in TLS also.  
  > **username_2**: @username_0 - Yes, you misunderstand. [This solution *bridges*](https://f5.com/glossary/ssl-bridging) the SSL conn and the data is encrypted on both sides of the reverse proxy. It does not terminate (offload) SSL encryption. On an F5 at least this can also be done [*without* re-encapsulating](https://support.f5.com/csp/article/K13385) the traffic in a SSL using SSL. The client establishes an SSL session with the back-end server and the ADC (Application Delivery Controller or load balancer) has the private key and uses it to decrypt and inspect the traffic for the purpose of session persistence.  
  > **username_2**: @username_1 - Load balancing at a TCP level won't work if he needs session persistence. Otherwise, you may be logged in to server A, and then your next request goes to server B, which makes you log in again - and may even destroy the login cookie you had for Server A. While you can maintain persistence using the SSL session ID, it is unreliable as session ID are rotated periodically.  
  > **username_1**: @username_2 round robin is not the only load balancing method, ip source hash, ip source/port hash are also valid solution and allow to keep sessions. In case of elasticsearch, each request is a standalone one, so there's nothing problematic to push datas to Elasticsearch with a RR load balancing.  
  > **username_2**: IP persistence is possible, but has drawbacks - eg if users are coming from behind a firewall, they won't properly distribute. But you are right - I had forgotten that Elasticsearch doesn't need to be able to maintain persistence. That should make nginx a definite contender for this situation.  
  > **username_0**: @username_2 are you aware of any opensource solution that supports this?   elastic puts are not session persistent so that really is not an issue  
  
  ---
  
  