Body: <p>Problem statement: Make a deployment with minimum disruption to clients connected via websockets.</p>
  <p>Stack: GKE (Regional cluster - 1.22.8-gke.200), helm, Node.js/websockets, <a href="https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/" rel="nofollow noreferrer">HPA</a>, <a href="https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#rolling-update-deployment" rel="nofollow noreferrer">Rolling Update</a>.</p>
  <p>Our Kubernetes Cluster is composed by micro-services separated to processing and connectivity. Our connectivity micro-service, let's call it <code>ws-gateway</code> is responsible for maintaining websocket connections with our clients. We have the requirement to keep the connections alive on our end as long as possible. We can afford to close a connection ourselves when we release a new version of the ws-gateway. The clients can take up to 5' to establish a new connection and we do not have any influence on that back-off connectivity behavior.</p>
  <p>The motivation of this post is to optimize our release strategy so clients can disconnect only once.</p>
  <h2>Context</h2>
  <p>Currently we are deploying with a rolling update strategy.</p>
  <p><strong>Important</strong>: we cannot afford all connections to be dropped at once, the load of re-establishing connections is expensive and out of scope of this post. During the rolling update the clients are still able to connect to old versions of the pods. According to rolling update documentation I didn't find anything that says otherwise.</p>
  <blockquote>
  <p>if a Deployment is exposed publicly, the Service will load-balance the traffic only to available Pods during the update.</p>
  </blockquote>
  <p>My take from this is that both versions receive traffic as long as they are available.
  Pods with old version will be terminated by k8s and nothing stops clients connecting to an old pod again.</p>
  <p>Clients are connecting to our pods via a service (protocol: TCP, type: LoadBalancer).</p>
  <h2>Desired Outcome</h2>
  <p><strong>During the release, clients connected to the old version are still connected until the pod is terminated, but new or disconnected clients can only connect to the new version of the pod.</strong></p>
  <p>Bonus: Passing-off a websocket would be quite interesting for us (Node.js).</p>
  <h2>Options</h2>
  <h3>1. Using selectors in the service.</h3>
  <p>Label the version on pods and match the selector of the service to the new version. This sounds exactly what I need in the pipeline. The service is now routing traffic only on v2.</p>
  <h4>Unknowns</h4>
  <ol>
  <li>What happens when we update the service?</li>
  </ol>
  <blockquote>
  <p>When you create a Service of type LoadBalancer, a Google Cloud controller wakes up and configures a network load balancer. Wait a minute for the controller to configure the network load balancer and generate a stable IP address.</p>
  </blockquote>
  <p><a href="https://cloud.google.com/kubernetes-engine/docs/how-to/exposing-apps#creating_a_service_of_type_loadbalancer" rel="nofollow noreferrer">https://cloud.google.com/kubernetes-engine/docs/how-to/exposing-apps#creating_a_service_of_type_loadbalancer</a></p>
  <p>The LB is pass-through.</p>
  <blockquote>
  <p>Network load balancers are not proxies.
  Load-balanced packets are received by backend VMs with the packet's source and destination IP addresses, protocol, and, if the protocol is port-based, the source and destination ports unchanged.
  Load-balanced connections are terminated by the backend VMs.
  Responses from the backend VMs go directly to the clients, not back through the load balancer. The industry term for this is direct server return.</p>
  </blockquote>
  <p><a href="https://cloud.google.com/load-balancing/docs/network" rel="nofollow noreferrer">https://cloud.google.com/load-balancing/docs/network</a></p>
  <p>My understanding here is that the LB, in our case, is only used once, when the connection has been upgraded to websockets, then there is a direct connection with the pod and the client. So, changes to LB should not affect the established connections. But. I ran a test and updated the selector to not match v1 pods. I lost 70% of the connections. 30% was left intact, which was confusing. Why were those connections lost and why remained intact?</p>
  <p>Setup test env: 3 Nodes running on different zones and only one pod on the cluster.</p>
  <h3>2. Ingress and weighted canary</h3>
  <p>Currently, we don't have an ingress for those pods, it's all done in the Service. Nginx ingress offers <code>nginx.ingress.kubernetes.io/canary-weight</code> which might be helpful.</p>
  <h3>Unknowns</h3>
  <ol>
  <li>How can the switch be gradual? Can we switch the traffic in a matter of minutes?</li>
  </ol>
  <h3>3. Blue/Green Deployment</h3>
  <p>Similar to 2, it seems that all traffic is switched immediately.</p>
  <h3>4. Anthos (GCP Service Mesh)</h3>
  <p>From a quick read, it seems a bit overkill as those deployments can be achieved with nginx ingress.</p>
  <h3>5. Fail readinessProbe</h3>
  <p>Intentionally make the readinessProbe fail on old pods so they are removed from the service. Seems a bit hacky and not what the readinessProbe is intended for.</p>
  <p>How does the HPA respond to failing readinessProbes?</p>
  <p>We should also take into account the case of a rollback.</p>
  <hr />
  <p>There is some reading I would like to do regarding option 2,3,4 but maybe you can enlighten me.</p>
  <p>Other options?</p>
  
Comments: CC BY-SA 4.0
Comments: 2023-02-10T12:10:06.987Z
Comments: it depends on cluster type, but for some clusters even when pod becomes unready, the ws connection is still opened and working
Comments: Ivan Aracki
Comments: CC BY-SA 4.0
Comments: 2023-11-28T04:10:24.883Z
Comments: @thanosam did you end up making any progress on refining your approach?
Comments: Josh Lyon
Comments: CC BY-SA 4.0
Comments: 2024-02-29T10:04:34.467Z
Comments: We have switched to Blue/Green deployments with argo rollouts and it is seamless — significantly fewer disconnects/reconnects.
Comments: thanosam
Tags: <kubernetes><continuous-deployment><load-balance><google-kubernetes-engine>
Title: Kubernetes deployement with minimum disruption to websocket connections
UserDisplayName: thanosam
ThreadText: # Kubernetes deployement with minimum disruption to websocket connections
  
  (Asked by: username_0 on 2022-04-30)
  
  Problem statement: Make a deployment with minimum disruption to clients connected via websockets.
  
  Stack: GKE (Regional cluster - 1.22.8-gke.200), helm, Node.js/websockets, [HPA](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/), [Rolling Update](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#rolling-update-deployment).
  
  Our Kubernetes Cluster is composed by micro-services separated to processing and connectivity. Our connectivity micro-service, let's call it `ws-gateway` is responsible for maintaining websocket connections with our clients. We have the requirement to keep the connections alive on our end as long as possible. We can afford to close a connection ourselves when we release a new version of the ws-gateway. The clients can take up to 5' to establish a new connection and we do not have any influence on that back-off connectivity behavior.
  
  The motivation of this post is to optimize our release strategy so clients can disconnect only once.
  
  ## Context
  
  Currently we are deploying with a rolling update strategy.
  
  **Important**: we cannot afford all connections to be dropped at once, the load of re-establishing connections is expensive and out of scope of this post. During the rolling update the clients are still able to connect to old versions of the pods. According to rolling update documentation I didn't find anything that says otherwise.
  
  > if a Deployment is exposed publicly, the Service will load-balance the traffic only to available Pods during the update.
  
  My take from this is that both versions receive traffic as long as they are available.
  Pods with old version will be terminated by k8s and nothing stops clients connecting to an old pod again.
  
  Clients are connecting to our pods via a service (protocol: TCP, type: LoadBalancer).
  
  ## Desired Outcome
  
  **During the release, clients connected to the old version are still connected until the pod is terminated, but new or disconnected clients can only connect to the new version of the pod.**
  
  Bonus: Passing-off a websocket would be quite interesting for us (Node.js).
  
  ## Options
  
  ### 1. Using selectors in the service.
  
  Label the version on pods and match the selector of the service to the new version. This sounds exactly what I need in the pipeline. The service is now routing traffic only on v2.
  
  #### Unknowns
  
  1. What happens when we update the service?
  
  > When you create a Service of type LoadBalancer, a Google Cloud controller wakes up and configures a network load balancer. Wait a minute for the controller to configure the network load balancer and generate a stable IP address.
  
  <https://cloud.google.com/kubernetes-engine/docs/how-to/exposing-apps#creating_a_service_of_type_loadbalancer>
  
  The LB is pass-through.
  
  > Network load balancers are not proxies.
  > Load-balanced packets are received by backend VMs with the packet's source and destination IP addresses, protocol, and, if the protocol is port-based, the source and destination ports unchanged.
  > Load-balanced connections are terminated by the backend VMs.
  > Responses from the backend VMs go directly to the clients, not back through the load balancer. The industry term for this is direct server return.
  
  <https://cloud.google.com/load-balancing/docs/network>
  
  My understanding here is that the LB, in our case, is only used once, when the connection has been upgraded to websockets, then there is a direct connection with the pod and the client. So, changes to LB should not affect the established connections. But. I ran a test and updated the selector to not match v1 pods. I lost 70% of the connections. 30% was left intact, which was confusing. Why were those connections lost and why remained intact?
  
  Setup test env: 3 Nodes running on different zones and only one pod on the cluster.
  
  ### 2. Ingress and weighted canary
  
  Currently, we don't have an ingress for those pods, it's all done in the Service. Nginx ingress offers `nginx.ingress.kubernetes.io/canary-weight` which might be helpful.
  
  ### Unknowns
  
  1. How can the switch be gradual? Can we switch the traffic in a matter of minutes?
  
  ### 3. Blue/Green Deployment
  
  Similar to 2, it seems that all traffic is switched immediately.
  
  ### 4. Anthos (GCP Service Mesh)
  
  From a quick read, it seems a bit overkill as those deployments can be achieved with nginx ingress.
  
  ### 5. Fail readinessProbe
  
  Intentionally make the readinessProbe fail on old pods so they are removed from the service. Seems a bit hacky and not what the readinessProbe is intended for.
  
  How does the HPA respond to failing readinessProbes?
  
  We should also take into account the case of a rollback.
  
  ---
  
  There is some reading I would like to do regarding option 2,3,4 but maybe you can enlighten me.
  
  Other options?
  
  **Tags:** kubernetes, continuous-deployment, load-balance, google-kubernetes-engine
  
  > **Comments:**
  > **username_1**: it depends on cluster type, but for some clusters even when pod becomes unready, the ws connection is still opened and working  
  > **username_2**: @username_0 did you end up making any progress on refining your approach?  
  > **username_0**: We have switched to Blue/Green deployments with argo rollouts and it is seamless — significantly fewer disconnects/reconnects.  
  
  ---
  
  ## 0 answers
  
  