Answers: <p>AWS have a specific service to deal with this type of use case, <a href="https://aws.amazon.com/systems-manager/" rel="nofollow noreferrer">EC2 Systems Manager</a></p>
  
  <p>This allows making base patch sets and then checking they are applied to target systems. </p>
  
  <blockquote>
    <p>"Patch Manager is a fully-managed service (provided at no additional
    cost) that helps you simplify your operating system patching process,
    including defining the patches you want to approve for deployment, the
    method of patch deployment, the timing for patch roll-outs, and
    determining patch compliance status across your entire fleet of
    instances. It’s extremely configurable with some sensible defaults and
    helps you easily deal with patching heterogeneous clusters."</p>
  </blockquote>
  
  <p>-- <a href="https://aws.amazon.com/blogs/aws/amazon-ec2-systems-manager-patch-manager-now-supports-linux/" rel="nofollow noreferrer">"Amazon EC2 Systems Manager Patch Manager now supports Linux"</a></p>
  
  <p>You'll need to install an SSM agent on each system and add a role but other than that it should be unobtrusive </p>
  
Answers: CC BY-SA 3.0
Answers: 2018-04-08T04:33:07.953Z
Answers: looks good but this solution is not including ELB/ALB disconnect/connect orchestration while applying patches
Answers: rombob
Answers: CC BY-SA 3.0
Answers: 2018-04-08T18:05:03.707Z
Answers: The AWS solution to your problem is to use an immutable solution: build AMI with the new kernel and then use ASG to cycle them out of ALB but I know you don't want this!!
Answers: Vorsprung
Answers: CC BY-SA 3.0
Answers: 2018-04-08T19:09:42.273Z
Answers: If I had to solve this problem I'd be less concerned about rotating them out before the update.  Apart from increased IO a kernel update will have no effect until the reboot.  You could have a init.d (or upstart or systemd or whatever) script to make a controlled exit from the ALB group on a controlled rebooted
Answers: Vorsprung
Answers: CC BY-SA 3.0
Answers: 2018-04-06T12:39:28.930Z
Answers: 2018-04-06T12:39:28.930Z
Answers: Vorsprung
Answers: <p>As people in the comments are saying, you can do this with aws-cli in bash or in the python boto3 library. </p>
  
  <p>What about ansible? It has the ability to see your ec2 instances and it could handle the automation for maintenance tasks. If you are not using ansible I highly recommend you add that tool to your DevOps toolbox. </p>
  
  <p>Here is how you might structure your <a href="https://stackoverflow.com/questions/27315469/ansible-how-to-sequentially-execute-playbook-for-each-host">playbook</a> to do what you are wanting.</p>
  
Answers: CC BY-SA 3.0
Answers: 2018-04-04T17:32:08.120Z
Answers: 2018-04-04T17:32:08.120Z
Answers: Levi
Body: <p>Would like to ask for recommendations on ready automated solutions or templates for rolling ec2 instance upgrade on AWS, requiring a reboot with ELB/ALB.
  Specifically required to perform the maintenance upgrade for the kernel on hundreds of Linux machines.</p>
  
  <p>For each ELB/ALB steps are:</p>
  
  <ol>
  <li>Disconnect ec2 instance from ELB/ALB wait for connection draining</li>
  <li>Upgrade kernel and reboot machine (not part of automation, e.g. running external ./upgrade.sh)</li>
  <li>Wait for instance to be online, connect to ELB/ALB, wait for health check status </li>
  <li>Continue to next instance in the load balancer</li>
  </ol>
  
  <p>Note: not looking for migration to the solution similar to Netflix Spinnaker or immutable deployment strategy implementation, but for general and simple on-demand maintenance tool.</p>
  
Comments: CC BY-SA 3.0
Comments: 2018-03-27T15:21:47.327Z
Comments: Is doing a +1 deploy possible for you? So instead of removing an instance from the ALB, you *add* an already upgraded instance? This would save a bunch of time from the two _waiting_ states you listed.
Comments: MrDuk
Comments: CC BY-SA 3.0
Comments: 2018-03-27T18:01:35.953Z
Comments: Thank you @MrDuk for the suggestion, this is what was mentioned as "immutable deployment" which will require currently more resources and coordination for deploying tens of different applications on hundreds of servers with different configurations, ideally this would be possible with better deployment automation, but it's not the case currently
Comments: rombob
Comments: CC BY-SA 3.0
Comments: 2018-03-27T18:13:25.863Z
Comments: If each node is distinct, with custom changes to each, I don't know that a tool exists that can help you with this. However, you could script this fairly easily using the aws-cli, the most complex part being the upgrade/reboot/detecting
Comments: MrDuk
Comments: CC BY-SA 3.0
Comments: 2018-03-27T18:16:51.780Z
Comments: @rombob I have to agree with MrDuk. If your instances have the capability to run different versions in parallel you should be able to use an immutable deployment strategy with no more effort than the steps you proposed.
Comments: Preston Martin
Comments: CC BY-SA 3.0
Comments: 2018-03-27T18:22:30.283Z
Comments: @MrDuk agreed, aws-cli or aws-sdk is what make sense to use, wondering if there ready solutions, other than custom upgrade, sequential/parallel traversing of balanced instances seems like very common task before we jump in writing our own, worth checking with community
Comments: rombob
Comments: CC BY-SA 3.0
Comments: 2018-03-27T18:28:50.237Z
Comments: @PrestonM I agree with with you both :) but it's not an option currently to re-engineer deployment strategy especially for legacy apps, and yet, we need production kernels to have latest patches
Comments: rombob
Comments: CC BY-SA 3.0
Comments: 2018-03-27T18:48:30.883Z
Comments: Is step 2 (including the application starting when it reboots) already automated?
Comments: MrDuk
Comments: CC BY-SA 3.0
Comments: 2018-03-28T11:09:28.360Z
Comments: clarified step 2, yes, it's already automated
Comments: rombob
Comments: CC BY-SA 4.0
Comments: 2018-05-24T07:13:23.637Z
Comments: This could be (somewhat) easily achieved by 1) updating the AMI to get kernel updates, 2) update the ASG's launch config to point to the new AMI and 3) terminate an instance and let the ASG replace it with the new AMI. This is what we're doing for stateless services. For stateful services, we add a volume snapshot/restore or remount the data volumes to the new instances.
Comments: SaxDaddy
Tags: <amazon-web-services><amazon-ec2><automation><linux><aws-elb>
Title: How to automate rolling upgrades for ec2 running linux behind elastic and/or application load balancer?
UserDisplayName: rombob
ThreadText: # How to automate rolling upgrades for ec2 running linux behind elastic and/or application load balancer?
  
  (Asked by: username_0 on 2018-03-27)
  
  Would like to ask for recommendations on ready automated solutions or templates for rolling ec2 instance upgrade on AWS, requiring a reboot with ELB/ALB.
  Specifically required to perform the maintenance upgrade for the kernel on hundreds of Linux machines.
  
  For each ELB/ALB steps are:
  
  1. Disconnect ec2 instance from ELB/ALB wait for connection draining
  2. Upgrade kernel and reboot machine (not part of automation, e.g. running external ./upgrade.sh)
  3. Wait for instance to be online, connect to ELB/ALB, wait for health check status
  4. Continue to next instance in the load balancer
  
  Note: not looking for migration to the solution similar to Netflix Spinnaker or immutable deployment strategy implementation, but for general and simple on-demand maintenance tool.
  
  **Tags:** amazon-web-services, amazon-ec2, automation, linux, aws-elb
  
  > **Comments:**
  > **username_1**: Is doing a +1 deploy possible for you? So instead of removing an instance from the ALB, you *add* an already upgraded instance? This would save a bunch of time from the two _waiting_ states you listed.  
  > **username_0**: Thank you @username_1 for the suggestion, this is what was mentioned as "immutable deployment" which will require currently more resources and coordination for deploying tens of different applications on hundreds of servers with different configurations, ideally this would be possible with better deployment automation, but it's not the case currently  
  > **username_1**: If each node is distinct, with custom changes to each, I don't know that a tool exists that can help you with this. However, you could script this fairly easily using the aws-cli, the most complex part being the upgrade/reboot/detecting  
  > **username_2**: @username_0 I have to agree with username_1. If your instances have the capability to run different versions in parallel you should be able to use an immutable deployment strategy with no more effort than the steps you proposed.  
  > **username_0**: @username_1 agreed, aws-cli or aws-sdk is what make sense to use, wondering if there ready solutions, other than custom upgrade, sequential/parallel traversing of balanced instances seems like very common task before we jump in writing our own, worth checking with community  
  > **username_0**: @PrestonM I agree with with you both :) but it's not an option currently to re-engineer deployment strategy especially for legacy apps, and yet, we need production kernels to have latest patches  
  > **username_1**: Is step 2 (including the application starting when it reboots) already automated?  
  > **username_0**: clarified step 2, yes, it's already automated  
  > **username_3**: This could be (somewhat) easily achieved by 1) updating the AMI to get kernel updates, 2) update the ASG's launch config to point to the new AMI and 3) terminate an instance and let the ASG replace it with the new AMI. This is what we're doing for stateless services. For stateful services, we add a volume snapshot/restore or remount the data volumes to the new instances.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_4 on 2018-04-06. Score: 2 | Accepted answer
  
  AWS have a specific service to deal with this type of use case, [EC2 Systems Manager](https://aws.amazon.com/systems-manager/)
  
  This allows making base patch sets and then checking they are applied to target systems.
  
  > "Patch Manager is a fully-managed service (provided at no additional
  > cost) that helps you simplify your operating system patching process,
  > including defining the patches you want to approve for deployment, the
  > method of patch deployment, the timing for patch roll-outs, and
  > determining patch compliance status across your entire fleet of
  > instances. It’s extremely configurable with some sensible defaults and
  > helps you easily deal with patching heterogeneous clusters."
  
  -- ["Amazon EC2 Systems Manager Patch Manager now supports Linux"](https://aws.amazon.com/blogs/aws/amazon-ec2-systems-manager-patch-manager-now-supports-linux/)
  
  You'll need to install an SSM agent on each system and add a role but other than that it should be unobtrusive
  
  > **Comments:**
  > **username_0**: looks good but this solution is not including ELB/ALB disconnect/connect orchestration while applying patches  
  > **username_4**: The AWS solution to your problem is to use an immutable solution: build AMI with the new kernel and then use ASG to cycle them out of ALB but I know you don't want this!!  
  > **username_4**: If I had to solve this problem I'd be less concerned about rotating them out before the update.  Apart from increased IO a kernel update will have no effect until the reboot.  You could have a init.d (or upstart or systemd or whatever) script to make a controlled exit from the ALB group on a controlled rebooted  
  
  ---
  
  ### Answer by username_5 on 2018-04-04. Score: 1
  
  As people in the comments are saying, you can do this with aws-cli in bash or in the python boto3 library.
  
  What about ansible? It has the ability to see your ec2 instances and it could handle the automation for maintenance tasks. If you are not using ansible I highly recommend you add that tool to your DevOps toolbox.
  
  Here is how you might structure your [playbook](https://stackoverflow.com/questions/27315469/ansible-how-to-sequentially-execute-playbook-for-each-host) to do what you are wanting.
  
  
  ---
  
  