Answers: <p>Yes, an input host file for ansible looks like <a href="https://pastebin.com/Vtfw5FtV" rel="nofollow noreferrer">https://pastebin.com/Vtfw5FtV</a>
  The upper part node stuff is generated using terraform, we can dynamically up/down scale the number of nodes with terraform and terraform outputs the yaml used for the hosts part of the file. The tenant-specifiek yml vars (about 150 additional lines of tenant specific config yml) are kept in a separate git location and a jenkins step/python concatenates them together and pushes them to the host directory location for each tenant and runs individual roles (hotfix) or all of 'em (full deploy) or perhaps just stop/start. The Jenkins job is pretty much a control screen with a lot of dropdowns, radio buttons and selectors. We can select how many nodes we want for a single service, which build to grab from nexus, do a full deploy, a single service hotfix or just a configuration hotfix. All this is selectable/executable per environment, per tenant and per individual module/service. The Jenkins screen is dynamically build from the host (targets) &amp; roles (actions) directory.</p>
  <p>Again, we're most likely abusing Ansible &quot;best practices&quot; a fair bit but for us this approach has been working remarkably well for last &gt;3 years in terms of overall manageability and gives us maximum flexibility of the whole CI/CD pipeline.</p>
  
Answers: CC BY-SA 4.0
Answers: 2021-04-25T22:32:47.813Z
Answers: This is awesome.  I just started using terraform today (similar to Vagrant but for cloud stuff). I'm still confused about the beginning of the file you linked, and how terraform outputs that; can you point me to any documentation?  I do understand the ansible inventory (yaml style) section.  It seems like if you fed this file to ansible, it would throw an error, since it wouldn't be a valid inventory file?  Or wait, is this file a pseudo code / tree representation of what your project's directory structure looks like?  Also, what is this hostfile consumed by?
Answers: Life5ign
Answers: CC BY-SA 4.0
Answers: 2021-04-26T08:16:37.587Z
Answers: https://pastebin.com/rGaydTh5 is a hostile.yml, the rest of the paste was just example. To deploy, for instance, the service component we would run ```ansible-role deploy/service -I hosts/tenant/hostfile.yml --hosts=SERVICE_NODES``` You do need to set the roles variable for the toolbox etc.
Answers: wowbagger
Answers: CC BY-SA 4.0
Answers: 2021-04-26T08:26:02.830Z
Answers: I can't really remember perfectly how we went from terraform output to ansible but looking at the code we apparently parse the terraform *.tfstate file using jq cmdline. It starts from an "empty host file skeleton" as input, for each "node group" in that skeleton file a function gets called that parses the terraform state file and grabs the hostnames for that type of node.
Answers: wowbagger
Answers: CC BY-SA 4.0
Answers: 2021-04-26T19:23:41.437Z
Answers: ah, I see it's a static inventory file.  My current idea is to use terraform to run an ansible-playook command as a local provisioner.  Maybe it would work to have terraform spin up a rig of instances, and provision them with my playbook(s), or individual roles with import_role, or include_tasks in ad hoc mode.  At least then running test deployment wouldn't be like setting a sixteen legged monster loose in our vpc, only able to be destroyed manually in the console, with awscli, or by writing another teardown playbook.. then I could combine that with hostvars per host
Answers: Life5ign
Answers: CC BY-SA 4.0
Answers: 2021-04-26T19:26:31.803Z
Answers: I guess this would effectively glue the hostvars to the terraform rig hosts; and thus, in my original question, it would be the ansible memory cache used during playbook execution that build the variable space; instead, it would be something like what you're describing: a way of writing a file of variables that are generated dynamically somewhere in the pipeline outside of ansible, that are associated with only certain hosts in the rig
Answers: Life5ign
Answers: CC BY-SA 4.0
Answers: 2021-04-25T11:34:53.293Z
Answers: 2021-04-25T11:34:53.293Z
Answers: wowbagger
Body: <p>I am using an ansible playbook to deploy a php web application in aws.  The deployment is repeated many times, with a different environment, for each customer.  The different scenarios range from single server deployments to full deployments using elastic load balancers, and use varying code repositories based on product type.</p>
  <p>To allow a human to deploy this app repetitively and adjust the deployment type and environment, I am running <code>ansible-playbook</code> with <code>--extra-vars &quot;@deployment_vars_file.yml&quot;</code> and copying and filling this vars file on a per-deployment basis, instead of putting the variables in <code>group_vars/all.yml</code>, so that I can keep a copy of the variable space I set for each deployment, and avoid overwriting (or .template suffixing) <code>group_vars/all.yml</code> and multiple roles' <code>vars/main.yml</code> files, for each deployment.  I rarely rely on variable precedence, since the role and play variables inherit (are constructed out of) my deployment vars file variables.</p>
  <p>My roles are already run conditionally at the play level, and I have a handle on how to run tasks/blocks/roles/plays conditionally, depending on the environment type that is defined in the <code>--extra-vars</code> file.</p>
  <p>I need to define many <strong>other</strong> variables, dynamically (during the playbook run), conditionally (based on the values of variables in my <code>--extra-vars</code> file), with as little work as possible (i.e., in one place), at the beginning of the playbook.  My roles' <code>vars/main.yml</code> files, as well as <code>vars:</code> play keywords, will inherit these other variables, (in addition to directly inheriting some of the <code>--extra-vars</code> variables).  <em>The values of these other variables need to persist at the playbook level, not just within each play.</em> I don't want to have to do this in multiple locations with the same lines of code.</p>
  <p>How can I do this, and what is the most simple way?</p>
  
Comments: CC BY-SA 4.0
Comments: 2021-04-24T13:23:42.113Z
Comments: I'm going to follow this thread. About 4 years ago I had the same problem with a big application spanning >30 "items to deploy", some terraform, some containers, some hard package installs and a lot of custom configuration/templates for each individual tenant and also split up in several OTAP environments. Headache time. I eventually settled using https://github.com/larsks/ansible-toolbox. That allows us to do ansible-role style deploys instead of full playbook style. That approach has been working really good, for each new tenant we copy a host file & update the variables for that tenant.
Comments: wowbagger
Comments: CC BY-SA 4.0
Comments: 2021-04-24T13:25:54.910Z
Comments: Alot of the host file creation is done with python and manually edited once, we generate tenant-specific online documentation out of those host files. The "core-hardly-ever-changing" variables are read using a common include role. We deploy all our tenants in parallel from Jenkins as one big pipeline with all the required steps calling ansible-role with the required hostfile.  We created "remove" roles (as opposed to deploy roles) so we can completely wipe out an environment and rebuild it again in a few minutes without needing to wait on terraform to do low-level stuff.
Comments: wowbagger
Comments: CC BY-SA 4.0
Comments: 2021-04-24T13:28:05.530Z
Comments: We created individual stop/start/check roles for each individual component. It's perhaps/most definitive not at all following the best-practice way from a strict Ansible point-of-view but from our application & infrastructure CI/CD point of view it's pretty solid, it allows us to do full deploys, rapid single-component hot fixes, partial or full roll backs, application start/stop/check etc per tenant and per component whilst keeping the variables clearly seperated and easily manageable.
Comments: wowbagger
Comments: CC BY-SA 4.0
Comments: 2021-04-25T05:16:07.453Z
Comments: @wowbagger I just installed ansible-toolbox; I'd been using the include_role module in ad hoc mod, and it seems like toolbox is a more compact way of doing that.  The insane amount of "per deployment" parameterization of my whole project is part of what makes it difficult to adapt components of it to things like updates/hotfixes, which is why I'm increasingly isolating specific sets of tasks into roles. When you say host file, do you mean an inventory file with variables in it? I'm also making a python script to help populate the vars.yml file.
Comments: Life5ign
Comments: CC BY-SA 4.0
Comments: 2021-04-25T05:17:38.933Z
Comments: And do you keep reusing your hostfile for all the updates/rollbacks/hotfixes, after the initial deployment, so as to load all the same variables?
Comments: Life5ign
Tags: <amazon-web-services><ansible><amazon-ec2><deployment><php>
Title: ansible-playbook: how to dynamically, conditionally set a large deployment environment (variable space) in a playbook?
UserDisplayName: Life5ign
ThreadText: # ansible-playbook: how to dynamically, conditionally set a large deployment environment (variable space) in a playbook?
  
  (Asked by: username_0 on 2021-04-21)
  
  I am using an ansible playbook to deploy a php web application in aws. The deployment is repeated many times, with a different environment, for each customer. The different scenarios range from single server deployments to full deployments using elastic load balancers, and use varying code repositories based on product type.
  
  To allow a human to deploy this app repetitively and adjust the deployment type and environment, I am running `ansible-playbook` with `--extra-vars "@deployment_vars_file.yml"` and copying and filling this vars file on a per-deployment basis, instead of putting the variables in `group_vars/all.yml`, so that I can keep a copy of the variable space I set for each deployment, and avoid overwriting (or .template suffixing) `group_vars/all.yml` and multiple roles' `vars/main.yml` files, for each deployment. I rarely rely on variable precedence, since the role and play variables inherit (are constructed out of) my deployment vars file variables.
  
  My roles are already run conditionally at the play level, and I have a handle on how to run tasks/blocks/roles/plays conditionally, depending on the environment type that is defined in the `--extra-vars` file.
  
  I need to define many **other** variables, dynamically (during the playbook run), conditionally (based on the values of variables in my `--extra-vars` file), with as little work as possible (i.e., in one place), at the beginning of the playbook. My roles' `vars/main.yml` files, as well as `vars:` play keywords, will inherit these other variables, (in addition to directly inheriting some of the `--extra-vars` variables). *The values of these other variables need to persist at the playbook level, not just within each play.* I don't want to have to do this in multiple locations with the same lines of code.
  
  How can I do this, and what is the most simple way?
  
  **Tags:** amazon-web-services, ansible, amazon-ec2, deployment, php
  
  > **Comments:**
  > **username_1**: I'm going to follow this thread. About 4 years ago I had the same problem with a big application spanning >30 "items to deploy", some terraform, some containers, some hard package installs and a lot of custom configuration/templates for each individual tenant and also split up in several OTAP environments. Headache time. I eventually settled using https://github.com/larsks/ansible-toolbox. That allows us to do ansible-role style deploys instead of full playbook style. That approach has been working really good, for each new tenant we copy a host file & update the variables for that tenant.  
  > **username_1**: Alot of the host file creation is done with python and manually edited once, we generate tenant-specific online documentation out of those host files. The "core-hardly-ever-changing" variables are read using a common include role. We deploy all our tenants in parallel from Jenkins as one big pipeline with all the required steps calling ansible-role with the required hostfile.  We created "remove" roles (as opposed to deploy roles) so we can completely wipe out an environment and rebuild it again in a few minutes without needing to wait on terraform to do low-level stuff.  
  > **username_1**: We created individual stop/start/check roles for each individual component. It's perhaps/most definitive not at all following the best-practice way from a strict Ansible point-of-view but from our application & infrastructure CI/CD point of view it's pretty solid, it allows us to do full deploys, rapid single-component hot fixes, partial or full roll backs, application start/stop/check etc per tenant and per component whilst keeping the variables clearly seperated and easily manageable.  
  > **username_0**: @username_1 I just installed ansible-toolbox; I'd been using the include_role module in ad hoc mod, and it seems like toolbox is a more compact way of doing that.  The insane amount of "per deployment" parameterization of my whole project is part of what makes it difficult to adapt components of it to things like updates/hotfixes, which is why I'm increasingly isolating specific sets of tasks into roles. When you say host file, do you mean an inventory file with variables in it? I'm also making a python script to help populate the vars.yml file.  
  > **username_0**: And do you keep reusing your hostfile for all the updates/rollbacks/hotfixes, after the initial deployment, so as to load all the same variables?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-04-25. Score: 1
  
  Yes, an input host file for ansible looks like <https://pastebin.com/Vtfw5FtV>
  The upper part node stuff is generated using terraform, we can dynamically up/down scale the number of nodes with terraform and terraform outputs the yaml used for the hosts part of the file. The tenant-specifiek yml vars (about 150 additional lines of tenant specific config yml) are kept in a separate git location and a jenkins step/python concatenates them together and pushes them to the host directory location for each tenant and runs individual roles (hotfix) or all of 'em (full deploy) or perhaps just stop/start. The Jenkins job is pretty much a control screen with a lot of dropdowns, radio buttons and selectors. We can select how many nodes we want for a single service, which build to grab from nexus, do a full deploy, a single service hotfix or just a configuration hotfix. All this is selectable/executable per environment, per tenant and per individual module/service. The Jenkins screen is dynamically build from the host (targets) & roles (actions) directory.
  
  Again, we're most likely abusing Ansible "best practices" a fair bit but for us this approach has been working remarkably well for last >3 years in terms of overall manageability and gives us maximum flexibility of the whole CI/CD pipeline.
  
  > **Comments:**
  > **username_0**: This is awesome.  I just started using terraform today (similar to Vagrant but for cloud stuff). I'm still confused about the beginning of the file you linked, and how terraform outputs that; can you point me to any documentation?  I do understand the ansible inventory (yaml style) section.  It seems like if you fed this file to ansible, it would throw an error, since it wouldn't be a valid inventory file?  Or wait, is this file a pseudo code / tree representation of what your project's directory structure looks like?  Also, what is this hostfile consumed by?  
  > **username_1**: https://pastebin.com/rGaydTh5 is a hostile.yml, the rest of the paste was just example. To deploy, for instance, the service component we would run ```ansible-role deploy/service -I hosts/tenant/hostfile.yml --hosts=SERVICE_NODES``` You do need to set the roles variable for the toolbox etc.  
  > **username_1**: I can't really remember perfectly how we went from terraform output to ansible but looking at the code we apparently parse the terraform *.tfstate file using jq cmdline. It starts from an "empty host file skeleton" as input, for each "node group" in that skeleton file a function gets called that parses the terraform state file and grabs the hostnames for that type of node.  
  > **username_0**: ah, I see it's a static inventory file.  My current idea is to use terraform to run an ansible-playook command as a local provisioner.  Maybe it would work to have terraform spin up a rig of instances, and provision them with my playbook(s), or individual roles with import_role, or include_tasks in ad hoc mode.  At least then running test deployment wouldn't be like setting a sixteen legged monster loose in our vpc, only able to be destroyed manually in the console, with awscli, or by writing another teardown playbook.. then I could combine that with hostvars per host  
  > **username_0**: I guess this would effectively glue the hostvars to the terraform rig hosts; and thus, in my original question, it would be the ansible memory cache used during playbook execution that build the variable space; instead, it would be something like what you're describing: a way of writing a file of variables that are generated dynamically somewhere in the pipeline outside of ansible, that are associated with only certain hosts in the rig  
  
  ---
  
  