Answers: <p>So I found some answers over time, posting them here in case it helps anyone.</p>
  <ul>
  <li><p>The Userdata never ran, I can't find see any of the above echo statements in the logs /var/log/cloud-init.log. When I SSH into the instance I can't find any of these files. How do I debug this better?</p>
  <p>ANS: There was an error in my Userdata, I SSHed into the instance and ran the script manually to debug and fix all issues.</p>
  </li>
  <li><p>On code update, Cloudformation stack is updated, this does not run Userdata(?) (I know it only runs when an instance is first created, but I would like some confirmation that Cloudformation update does not trigger this</p>
  <p>ANS: Cloudformation terminates the old instance and spins up a new one on each deploy, this causes the Userdata to be run each time.</p>
  </li>
  <li><p>Is this an ideal flow? Are there any improvements I can make here, considering I'm not an expert, but willing to spend some time learning where required.</p>
  <p>ANS: It was not. The primary reason I was doing this was to fire async requests from one lambda to another, without waiting for the results. This is not possible, as the lambda will not wait for any requests that you do not <code>await</code> meaning you still have to wait for those operations. I tried asynchronous invocation but it still did not work (For those interested I posted a question about it <a href="https://stackoverflow.com/questions/72493614/serverless-express-lambda-async-invoke-not-working">here</a>). So I switched to using SNS to publish messages from one lambda to trigger another, achieving my requirement for fire and forget requests.</p>
  </li>
  </ul>
  
Answers: CC BY-SA 4.0
Answers: 2022-07-26T17:20:41.897Z
Answers: 2022-07-26T17:20:41.897Z
Answers: kvnam
Body: <p>I'm relatively new to Cloudformation, and had a few questions about my template and best practices. I'm facing a few hurdles, and there is a lot of information out there, it's a bit overwhelming, so any feedback would be highly appreciated. I'm not looking for detailed code etc. just some good insights on how I can improve my steps.</p>
  <p>I'm trying to set up a basic Node/Express API:</p>
  <ul>
  <li>On push to Git repo
  <ul>
  <li>Build Docker image and push to private AWS ECR repo</li>
  <li>After successful push, deploy Cloudformation template that provisions
  <ul>
  <li>An EC2 + security group with Elastic IP assigned</li>
  <li>Run docker compose in Userdata of EC2 to get app up and running</li>
  </ul>
  </li>
  </ul>
  </li>
  </ul>
  <p>This is my UserData (I do need some specific help here!)</p>
  <pre><code>UserData: !Base64 |
          #!/bin/bash -ex
          yum update -y
          yum install docker -y
          service docker start
          usermod -a -G docker ec2-user
          echo &quot;Start Docker service&quot;
          apt-get update
          apt-get install docker-compose-plugin
          apt install amazon-ecr-credential-helper
          echo &quot;APT-GET update complete&quot;
          echo &quot;{ \&quot;credHelpers\&quot;: { \&quot;&lt;my_acc_id&gt;.dkr.ecr.&lt;my_region&gt;.amazonaws.com\&quot;: \&quot;ecr-login\&quot; } }&quot; &gt; ~/.docker/config.json
          systemctl restart docker
          echo &quot;
          version: &quot;3.9&quot;
          services:
            my-app:
              image: &lt;acc_id&gt;.dkr.ecr.&lt;my_region&gt;.amazonaws.com/my-repo
              environment:
                STAGE: staging
                VIRTUAL_HOST: my-customdomain.com
                VIRTUAL_PORT: 3000
              ports:
                - &quot;3000:3000&quot;
              restart: always
              networks:
                - my-network
  
            https-portal:
              image: steveltn/https-portal:1
              ports:
                - '80:80'
                - '443:443'
              links:
                - my-app
              environment:
                STAGE: production
              volumes:
                - /var/run/docker.sock:/var/run/docker.sock:ro
              networks:
                - my-network
  
          volumes:
            https-portal-data:
          networks:
            my-network:
              driver: bridge
          &quot; &gt; docker-compose.yaml
          docker compose up -d
  </code></pre>
  <p>Status:
  Cloudformation template deploys successfully, all resources set up. But the Userdata doesn't run, so my EC2 never sets up my app.</p>
  <p>Issues / Questions:</p>
  <ul>
  <li>The Userdata never ran, I can't find see any of the above echo statements in the logs <code>/var/log/cloud-init.log</code>. When I SSH into the instance I can't find any of these files. How do I debug this better?</li>
  <li>Is there a better way to get the docker-compose data in there? writing the whole file in the UserData script seems inefficient. Is there a better way to do this?</li>
  <li>On code update, Cloudformation stack is updated, this does not run Userdata(?) (I know it only runs when an instance is first created, but I would like some confirmation that Cloudformation update does not trigger this.</li>
  <li>What is the best practice here if I want to re-run docker compose in my Ec2 after every Cloudformation deploy? If it does trigger Userdata, what could be wrong here?</li>
  <li>Is this an ideal flow? Are there any improvements I can make here, considering I'm not an expert, but willing to spend some time learning where required.</li>
  </ul>
  <p>I appreciate anyone taking the time to answer these questions. Thanks!</p>
  
Tags: <docker><docker-compose><amazon-ec2><aws-cloudformation><aws-ecr>
Title: Cloudformation template with EC2 using docker compose
UserDisplayName: kvnam
ThreadText: # Cloudformation template with EC2 using docker compose
  
  (Asked by: username_0 on 2022-07-21)
  
  I'm relatively new to Cloudformation, and had a few questions about my template and best practices. I'm facing a few hurdles, and there is a lot of information out there, it's a bit overwhelming, so any feedback would be highly appreciated. I'm not looking for detailed code etc. just some good insights on how I can improve my steps.
  
  I'm trying to set up a basic Node/Express API:
  
  * On push to Git repo
    + Build Docker image and push to private AWS ECR repo
    + After successful push, deploy Cloudformation template that provisions
      - An EC2 + security group with Elastic IP assigned
      - Run docker compose in Userdata of EC2 to get app up and running
  
  This is my UserData (I do need some specific help here!)
  
  ```
  UserData: !Base64 |
          #!/bin/bash -ex
          yum update -y
          yum install docker -y
          service docker start
          usermod -a -G docker ec2-user
          echo "Start Docker service"
          apt-get update
          apt-get install docker-compose-plugin
          apt install amazon-ecr-credential-helper
          echo "APT-GET update complete"
          echo "{ \"credHelpers\": { \"<my_acc_id>.dkr.ecr.<my_region>.amazonaws.com\": \"ecr-login\" } }" > ~/.docker/config.json
          systemctl restart docker
          echo "
          version: "3.9"
          services:
            my-app:
              image: <acc_id>.dkr.ecr.<my_region>.amazonaws.com/my-repo
              environment:
                STAGE: staging
                VIRTUAL_HOST: my-customdomain.com
                VIRTUAL_PORT: 3000
              ports:
                - "3000:3000"
              restart: always
              networks:
                - my-network
  
            https-portal:
              image: steveltn/https-portal:1
              ports:
                - '80:80'
                - '443:443'
              links:
                - my-app
              environment:
                STAGE: production
              volumes:
                - /var/run/docker.sock:/var/run/docker.sock:ro
              networks:
                - my-network
  
          volumes:
            https-portal-data:
          networks:
            my-network:
              driver: bridge
          " > docker-compose.yaml
          docker compose up -d
  
  ```
  
  Status:
  Cloudformation template deploys successfully, all resources set up. But the Userdata doesn't run, so my EC2 never sets up my app.
  
  Issues / Questions:
  
  * The Userdata never ran, I can't find see any of the above echo statements in the logs `/var/log/cloud-init.log`. When I SSH into the instance I can't find any of these files. How do I debug this better?
  * Is there a better way to get the docker-compose data in there? writing the whole file in the UserData script seems inefficient. Is there a better way to do this?
  * On code update, Cloudformation stack is updated, this does not run Userdata(?) (I know it only runs when an instance is first created, but I would like some confirmation that Cloudformation update does not trigger this.
  * What is the best practice here if I want to re-run docker compose in my Ec2 after every Cloudformation deploy? If it does trigger Userdata, what could be wrong here?
  * Is this an ideal flow? Are there any improvements I can make here, considering I'm not an expert, but willing to spend some time learning where required.
  
  I appreciate anyone taking the time to answer these questions. Thanks!
  
  **Tags:** docker, docker-compose, amazon-ec2, aws-cloudformation, aws-ecr
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2022-07-26. Score: 1
  
  So I found some answers over time, posting them here in case it helps anyone.
  
  * The Userdata never ran, I can't find see any of the above echo statements in the logs /var/log/cloud-init.log. When I SSH into the instance I can't find any of these files. How do I debug this better?
  
    ANS: There was an error in my Userdata, I SSHed into the instance and ran the script manually to debug and fix all issues.
  * On code update, Cloudformation stack is updated, this does not run Userdata(?) (I know it only runs when an instance is first created, but I would like some confirmation that Cloudformation update does not trigger this
  
    ANS: Cloudformation terminates the old instance and spins up a new one on each deploy, this causes the Userdata to be run each time.
  * Is this an ideal flow? Are there any improvements I can make here, considering I'm not an expert, but willing to spend some time learning where required.
  
    ANS: It was not. The primary reason I was doing this was to fire async requests from one lambda to another, without waiting for the results. This is not possible, as the lambda will not wait for any requests that you do not `await` meaning you still have to wait for those operations. I tried asynchronous invocation but it still did not work (For those interested I posted a question about it [here](https://stackoverflow.com/questions/72493614/serverless-express-lambda-async-invoke-not-working)). So I switched to using SNS to publish messages from one lambda to trigger another, achieving my requirement for fire and forget requests.
  
  
  ---
  
  