Answers: <p>Bash functions (even when exported) are not preserved across <code>commands</code> steps. This is in contrast to exported env variables, so it seems AWS is doing some shenanigans behind the scenes to prevent it. I see two basic options to work around that:</p>
  <ol>
  <li><p>Combine steps which need the function in a single step. E.g.</p>
  <pre class="lang-yaml prettyprint-override"><code>commands:
    - |
      replaceConfig() { sed -i 's|{'$1'}|'${!1}'|' config.json; }
      replaceConfig DB_USERNAME &amp;&amp;
        replaceConfig DB_PASSWORD &amp;&amp;
        replaceConfig DB_HOST
  </code></pre>
  </li>
  <li><p>Source the file with the functions you need in every step. This may sound laborious, but can be automated via the <code>BASH_ENV</code> environment variable. E.g. let's say you put all the functions you need in a <code>codbuild-helper.bash</code> file in your project root, then you can do:</p>
  <pre class="lang-yaml prettyprint-override"><code>env:
    shell: bash
    variables:
      AWS_REGION: us-east-2
      BASH_ENV: &quot;$CODEBUILD_SRC_DIR/codebuild-helper.bash&quot;
    secrets-manager:
      DB_USERNAME: db-credentials:username
      DB_PASSWORD: db-credentials:password
      DB_HOST: db-credentials:host
  ....
  phases:
    ....
    post_build:
      commands:
        - replaceConfig DB_USERNAME
        - replaceConfig DB_PASSWORD
        - replaceConfig DB_HOST
  ....
  </code></pre>
  <p>But since this file will be sourced in every step (regardless if it's needed or not) and even in bash scripts called along the way, you should take care to indeed only put function definitions in there.</p>
  </li>
  </ol>
  
Answers: CC BY-SA 4.0
Answers: 2022-08-26T02:07:11.107Z
Answers: 2022-08-26T02:07:11.107Z
Answers: David Ongaro
Answers: <p>In cases where you have a complex bash script with functions for AWS CodeBuild that may not translate well into YAML syntax, encoding the script with base64 provides a robust solution. It preserves the script's integrity, particularly when it contains special characters or constructs that are problematic to escape directly in YAML.</p>
  <p>Here's a streamlined approach</p>
  <pre><code>import base64
  
  # Read the script content from a file
  with open('path/to/your/script.sh', 'r') as file:
      script_content = file.read()
  
  # Base64 encode the script to avoid issues with special characters
  encoded_script = base64.b64encode(script_content.encode()).decode()
  
  # Generate the commands to decode and execute the script within the build environment
  commands = [
      &quot;echo 'Starting the pre-build process'&quot;,
      f&quot;echo {encoded_script} | base64 --decode &gt; setup_env.sh&quot;,
      &quot;chmod +x setup_env.sh&quot;,
      &quot;./setup_env.sh&quot;
  ]
  
  # Integrate these commands into your buildspec configuration
  build_spec_yaml = {
      &quot;version&quot;: &quot;0.2&quot;,
      &quot;env&quot;: {
          &quot;parameter-store&quot;: {
              &quot;ModelPackageName&quot;: config['SsmModelPackageNameParameter'],
          }
      },
      &quot;phases&quot;: {
          &quot;pre_build&quot;: {
              &quot;commands&quot;: commands
          }
          # Other phases as needed...
      }
      # Additional buildspec configurations...
  }
  
  # If using AWS CDK, you can then convert this dictionary to YAML format
  build_spec = BuildSpec.from_object_to_yaml(build_spec_yaml)
  </code></pre>
  <p>This method is advantageous for those utilizing AWS CDK for their infrastructure as code, allowing the dynamic generation and modification of buildspec configurations directly from Python scripts. It leverages the BuildSpec.from_object_to_yaml method for seamless translation to YAML, which can then be used by CodeBuild for provisioning through AWS CloudFormation.</p>
  <p>Hope this solution aids anyone facing similar challenges and enhances the flexibility of managing build specifications in complex environments.</p>
  <p>Feel free to share your insights or improvements to this approach!</p>
  
Answers: CC BY-SA 4.0
Answers: 2024-04-13T17:22:13.250Z
Answers: 2024-04-13T17:22:13.250Z
Answers: cjoun16
Body: <p>I thought I could slightly improve readability of repetitive <code>sed</code> calls when inserting secrets into a file with AWS CodeBuild. I defined a bash function:</p>
  <pre><code>replaceConfig() { sed -i 's|{'$1'}|'${!1}'|' config.json; }
  </code></pre>
  <p>This can be called like so:</p>
  <pre><code>replaceConfig DB_PASSWORD
  </code></pre>
  <p>And this will replace all instances of the text <code>{DB_PASSWORD}</code> in config.json with the value held in secret environment variable <code>DB_PASSWORD</code>. Unfotunately, this does not work in CodeBuild. Inspecting the log file, I see:</p>
  <pre><code>[Container] 2020*07*31 15:44:40 Running command replaceConfig DB_USERNAME
  *codebuild*output*tmp*script.sh: line 4: replaceConfig: command not found
  </code></pre>
  <p>I get the same error defining the bash function with the alternative syntax:</p>
  <pre><code>function replaceConfig { sed -i 's|{'$1'}|'${!1}'|' config.json; }
  </code></pre>
  <p>I have read that buildspec version 0.1 used a separate shell process for each command, which would have caused this problem, but I am using buildspec version 0.2, so these commands should be sharing the same shell.</p>
  <p>Of course I can just stick with the repetitive sed calls, but am I correct in understanding that bash functions cannot be used in buildspec commands at all? Is there a better way to get these secrets into a configuration file?</p>
  <pre class="lang-yaml prettyprint-override"><code>version: 0.2
  
  run-as: root
  
  env:
    shell: bash
    variables:
      AWS_REGION: us-east-2
    secrets-manager:
      DB_USERNAME: db-credentials:username
      DB_PASSWORD: db-credentials:password
      DB_HOST: db-credentials:host
  ....
  phases:
    ....
    post_build:
      commands:
        - replaceConfig() { sed -i 's|{'$1'}|'${!1}'|' config.json; }
        - replaceConfig DB_USERNAME
        - replaceConfig DB_PASSWORD
        - replaceConfig DB_HOST
  ....
  
  </code></pre>
  
Comments: CC BY-SA 4.0
Comments: 2020-08-03T22:14:28.783Z
Comments: Welcome to DevOps.SE, Kyle.  This seems like a good idea to me.  I'm not sure why it doesn't work.  Have you tried contacting AWS support?
Comments: chicks
Comments: CC BY-SA 4.0
Comments: 2020-08-15T15:35:13.670Z
Comments: Thank you for commenting. I use free tier, which as far as I know does not allow me to contact AWS support for this kind of issue. I will try posting in the AWS Developer Forums though.
Comments: Kyle
Comments: CC BY-SA 4.0
Comments: 2021-08-04T18:21:14.477Z
Comments: Sidenote: using envsubst is easier than sed almost every time. `envsubst < config.json.tpl > config.json`. It substitutes env vars into a template file, simple one liner. We can also tell envsubst to only replace a set of specific var names if desired.
  `envsubst` is installed by default since codebuild standard 2.0 iirc
Comments: OnethingSimple
Comments: CC BY-SA 4.0
Comments: 2022-01-17T14:57:01.053Z
Comments: @OnethingSimple Thanks for this. Although it's a sidenote to the "XY Problem" I asked about, it does solve my original problem. :-)
Comments: Kyle
Tags: <amazon-web-services><bash><aws-codepipeline>
Title: Defining and using bash functions in AWS CodeBuild
UserDisplayName: Kyle
ThreadText: # Defining and using bash functions in AWS CodeBuild
  
  (Asked by: username_0 on 2020-07-31)
  
  I thought I could slightly improve readability of repetitive `sed` calls when inserting secrets into a file with AWS CodeBuild. I defined a bash function:
  
  ```
  replaceConfig() { sed -i 's|{'$1'}|'${!1}'|' config.json; }
  
  ```
  
  This can be called like so:
  
  ```
  replaceConfig DB_PASSWORD
  
  ```
  
  And this will replace all instances of the text `{DB_PASSWORD}` in config.json with the value held in secret environment variable `DB_PASSWORD`. Unfotunately, this does not work in CodeBuild. Inspecting the log file, I see:
  
  ```
  [Container] 2020*07*31 15:44:40 Running command replaceConfig DB_USERNAME
  *codebuild*output*tmp*script.sh: line 4: replaceConfig: command not found
  
  ```
  
  I get the same error defining the bash function with the alternative syntax:
  
  ```
  function replaceConfig { sed -i 's|{'$1'}|'${!1}'|' config.json; }
  
  ```
  
  I have read that buildspec version 0.1 used a separate shell process for each command, which would have caused this problem, but I am using buildspec version 0.2, so these commands should be sharing the same shell.
  
  Of course I can just stick with the repetitive sed calls, but am I correct in understanding that bash functions cannot be used in buildspec commands at all? Is there a better way to get these secrets into a configuration file?
  
  ```yaml
  version: 0.2
  
  run-as: root
  
  env:
    shell: bash
    variables:
      AWS_REGION: us-east-2
    secrets-manager:
      DB_USERNAME: db-credentials:username
      DB_PASSWORD: db-credentials:password
      DB_HOST: db-credentials:host
  ....
  phases:
    ....
    post_build:
      commands:
        - replaceConfig() { sed -i 's|{'$1'}|'${!1}'|' config.json; }
        - replaceConfig DB_USERNAME
        - replaceConfig DB_PASSWORD
        - replaceConfig DB_HOST
  ....
  
  ```
  
  **Tags:** amazon-web-services, bash, aws-codepipeline
  
  > **Comments:**
  > **username_1**: Welcome to DevOps.SE, username_0.  This seems like a good idea to me.  I'm not sure why it doesn't work.  Have you tried contacting AWS support?  
  > **username_0**: Thank you for commenting. I use free tier, which as far as I know does not allow me to contact AWS support for this kind of issue. I will try posting in the AWS Developer Forums though.  
  > **username_2**: Sidenote: using envsubst is easier than sed almost every time. `envsubst < config.json.tpl > config.json`. It substitutes env vars into a template file, simple one liner. We can also tell envsubst to only replace a set of specific var names if desired. `envsubst` is installed by default since codebuild standard 2.0 iirc  
  > **username_0**: @username_2 Thanks for this. Although it's a sidenote to the "XY Problem" I asked about, it does solve my original problem. :-)  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2022-08-26. Score: 1
  
  Bash functions (even when exported) are not preserved across `commands` steps. This is in contrast to exported env variables, so it seems AWS is doing some shenanigans behind the scenes to prevent it. I see two basic options to work around that:
  
  1. Combine steps which need the function in a single step. E.g.
  
     ```yaml
     commands:
       - |
         replaceConfig() { sed -i 's|{'$1'}|'${!1}'|' config.json; }
         replaceConfig DB_USERNAME &&
           replaceConfig DB_PASSWORD &&
           replaceConfig DB_HOST
  
     ```
  2. Source the file with the functions you need in every step. This may sound laborious, but can be automated via the `BASH_ENV` environment variable. E.g. let's say you put all the functions you need in a `codbuild-helper.bash` file in your project root, then you can do:
  
     ```yaml
     env:
       shell: bash
       variables:
         AWS_REGION: us-east-2
         BASH_ENV: "$CODEBUILD_SRC_DIR/codebuild-helper.bash"
       secrets-manager:
         DB_USERNAME: db-credentials:username
         DB_PASSWORD: db-credentials:password
         DB_HOST: db-credentials:host
     ....
     phases:
       ....
       post_build:
         commands:
           - replaceConfig DB_USERNAME
           - replaceConfig DB_PASSWORD
           - replaceConfig DB_HOST
     ....
  
     ```
  
     But since this file will be sourced in every step (regardless if it's needed or not) and even in bash scripts called along the way, you should take care to indeed only put function definitions in there.
  
  
  ---
  
  ### Answer by username_4 on 2024-04-13. Score: 0
  
  In cases where you have a complex bash script with functions for AWS CodeBuild that may not translate well into YAML syntax, encoding the script with base64 provides a robust solution. It preserves the script's integrity, particularly when it contains special characters or constructs that are problematic to escape directly in YAML.
  
  Here's a streamlined approach
  
  ```
  import base64
  
  # Read the script content from a file
  with open('path/to/your/script.sh', 'r') as file:
      script_content = file.read()
  
  # Base64 encode the script to avoid issues with special characters
  encoded_script = base64.b64encode(script_content.encode()).decode()
  
  # Generate the commands to decode and execute the script within the build environment
  commands = [
      "echo 'Starting the pre-build process'",
      f"echo {encoded_script} | base64 --decode > setup_env.sh",
      "chmod +x setup_env.sh",
      "./setup_env.sh"
  ]
  
  # Integrate these commands into your buildspec configuration
  build_spec_yaml = {
      "version": "0.2",
      "env": {
          "parameter-store": {
              "ModelPackageName": config['SsmModelPackageNameParameter'],
          }
      },
      "phases": {
          "pre_build": {
              "commands": commands
          }
          # Other phases as needed...
      }
      # Additional buildspec configurations...
  }
  
  # If using AWS CDK, you can then convert this dictionary to YAML format
  build_spec = BuildSpec.from_object_to_yaml(build_spec_yaml)
  
  ```
  
  This method is advantageous for those utilizing AWS CDK for their infrastructure as code, allowing the dynamic generation and modification of buildspec configurations directly from Python scripts. It leverages the BuildSpec.from\_object\_to\_yaml method for seamless translation to YAML, which can then be used by CodeBuild for provisioning through AWS CloudFormation.
  
  Hope this solution aids anyone facing similar challenges and enhances the flexibility of managing build specifications in complex environments.
  
  Feel free to share your insights or improvements to this approach!
  
  
  ---
  
  