Answers: <p>First of all, I propose a change in terminology. What you're calling a &quot;project&quot; should be referred to as a &quot;module&quot;.</p>
  <p>An environment is built out of a composition of modules.</p>
  <p>This is a design decision. A key part of architecting a cloud environment is to design its components in a such a way that you can maintain them independently. In terraform, these come in the form of <a href="https://developer.hashicorp.com/terraform/language/modules/" rel="nofollow noreferrer">modules</a>:</p>
  <blockquote>
  <p>You can use modules to create lightweight abstractions, so that you can describe your infrastructure in terms of its architecture, rather than directly in terms of physical objects.</p>
  </blockquote>
  <p>Instead of having a huge lump of resources, you have a set of building blocks which express higher levels of abstraction. An example might be &quot;compute cluster&quot;. This will be comprised of the instances, the networks, block storage, object storage buckets, access permissions, etc, all packaged in a module. If this concept (compute cluster) is a useful one for you, then you should design a Terraform module which encapsulates them, so that you can reuse the concept.</p>
  <p>There is a tradeoff to be made between tiny modules (ie single resource), and mega-modules (ie, all the resources).</p>
  <p>Consider your architecture, try to identify which components are re-usable and build modules out of those. To make an analogy with construction: a window and a door are both re-usable components. A wall is a useful concept, not a brick.</p>
  
Answers: CC BY-SA 4.0
Answers: 2023-05-11T05:21:05.567Z
Answers: Thanks for the very detailled answer :) So instead of creating multiple GIT projects, I will use one GIT project with different terraform modules.
Answers: Marvin Kern
Answers: CC BY-SA 4.0
Answers: 2023-05-11T06:32:24.370Z
Answers: This is a matter of choice too (sorry!). I typically advocate for developing modules independently of architectures. So, n git repos , one for each module, independently developed and maintained. Then one repo per architecture, perhaps with different environments. However -- do what works for you!
Answers: Bruce Becker
Answers: CC BY-SA 4.0
Answers: 2023-05-16T22:16:23.300Z
Answers: It is pretty common to start off with one git repo and many terraform modules.  Once that setup starts to be a problem it is straight forward to split it into multiple git repos.
Answers: chicks
Answers: CC BY-SA 4.0
Answers: 2023-05-10T13:55:17.067Z
Answers: 2023-05-10T13:55:17.067Z
Answers: Bruce Becker
Answers: <p>[I agree with everything in the two other answers, but I'd like to add something else to the discussion.]</p>
  <h1>Terraform challenges</h1>
  <p>I've had a number of cases with terraform where the number of resources in a terraform module/directory become so huge that terraform takes an unreasonable amount of time to converge on a plan.  This often seems to happen with <code>for_each</code> loops or other data-driven approaches where a few lines in a data file lead to lots of resources and work for terraform.  With these huge directories it can takes hours for terraform to think about it.  This is worse on Intel MacBook Pros than ARM/M2-based ones, but I've seen a plan that took hours on an Intel CPU take 15-30 minutes on an ARM processor.</p>
  <p>Another issue is provider upgrades.  These can lead to needing to do state transformations and often require debugging and iteration to get right.  Do you want this debugging to be happening on all of your infrastructure?  Probably not.</p>
  <h1>Solutions</h1>
  <ul>
  <li>As others have said, using modules to keep your code organized is a very good habit to get into with terraform.</li>
  <li>Use environments - you can have different values for variables per environment and adding environments doesn't entail writing more code.</li>
  <li>Keep logical things together in the same directory.</li>
  <li>Beyond common providers that you need everywhere (like vault), minimize the number of different providers in each directory.  I wouldn't develop something for AWS in the same directory where I'm managing datadog or another cloud service.</li>
  <li>Use remote state references to share data internally.  Terraform modules that are just <code>local</code>s that get turned into <code>output</code>s are good for centralizing information like domain names, IP lists, and other things so you're not copying and pasting this information all over the place.</li>
  </ul>
  
Answers: CC BY-SA 4.0
Answers: 2023-08-01T18:51:32.563Z
Answers: 2023-08-01T18:51:32.563Z
Answers: chicks
Answers: <p>All three ways will work (modules mentioned in answers as third answer), in some cases better, in some - well.</p>
  <p>But the exercise you have to do is you have to think, how your infrastructure will look in the nearest future. This is, as it was mentioned, a design and architecture decision. For sure you have some goals to achieve and this should be a base for your design.</p>
  <p>You have to find the answers to couple of questions:</p>
  <ul>
  <li>how your infrastructure will be multiplied?</li>
  <li>do you allow to re-use of specific elements of the infrastructure (for example VPC)?</li>
  <li>are you going to serve these templates through Platform Engineering teams - self-service solutions?</li>
  <li>how do you deal with multiple environments?</li>
  <li>how do you manage the state?</li>
  <li>how do you orchestrate different modules/templates/environments?</li>
  <li>how do you want to manage the whole SDLC of your infrastructure?</li>
  </ul>
  <p>These questions are the starting point for your design. I know, it sounds like I complicate things, but I try to think about big picture. And believe me, it will be very important later.</p>
  <p>For example: if my solution is simple and I will run it generally only once, I will go with the simplest solution. If I know I will re-use some components, I will go with modules. If I will re-use my modules in wider way (many projects, env, etc), I will go with modules stored in registry. If I go even further, it is time to think about dedicated tools, like <a href="https://spacelift.io/" rel="nofollow noreferrer">Spacelift</a> (disclosure: I am Developer Advocate there). But if you want to orchestrate very complicated setup, with multiple teams, projects, accounts (or subscriptions), modules, dependencies, you should, really, look at tools like this.</p>
  
Answers: CC BY-SA 4.0
Answers: 2023-08-01T07:43:46.700Z
Answers: 2023-08-01T11:48:18.977Z
Answers: 2023-08-01T11:48:18.977Z
Answers: Pawel Piwosz
Body: <p>I am starting to build a Terraform GIT project to build resources for azure.
  Currently my idea is to have a project that is taking care of identity and access topics &amp; to have a project that is taking care of infrastructure topics.</p>
  <p>Is it a good idea to have multiple terraform projects for different topics, or would it be better to just have one terraform project that is creating all resources for my organisation?</p>
  
Tags: <terraform>
Title: Should I use multiple Terraform projects or only one?
UserDisplayName: Marvin Kern
ThreadText: # Should I use multiple Terraform projects or only one?
  
  (Asked by: username_0 on 2023-05-10)
  
  I am starting to build a Terraform GIT project to build resources for azure.
  Currently my idea is to have a project that is taking care of identity and access topics & to have a project that is taking care of infrastructure topics.
  
  Is it a good idea to have multiple terraform projects for different topics, or would it be better to just have one terraform project that is creating all resources for my organisation?
  
  **Tags:** terraform
  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_1 on 2023-05-10. Score: 1 | Accepted answer
  
  First of all, I propose a change in terminology. What you're calling a "project" should be referred to as a "module".
  
  An environment is built out of a composition of modules.
  
  This is a design decision. A key part of architecting a cloud environment is to design its components in a such a way that you can maintain them independently. In terraform, these come in the form of [modules](https://developer.hashicorp.com/terraform/language/modules/):
  
  > You can use modules to create lightweight abstractions, so that you can describe your infrastructure in terms of its architecture, rather than directly in terms of physical objects.
  
  Instead of having a huge lump of resources, you have a set of building blocks which express higher levels of abstraction. An example might be "compute cluster". This will be comprised of the instances, the networks, block storage, object storage buckets, access permissions, etc, all packaged in a module. If this concept (compute cluster) is a useful one for you, then you should design a Terraform module which encapsulates them, so that you can reuse the concept.
  
  There is a tradeoff to be made between tiny modules (ie single resource), and mega-modules (ie, all the resources).
  
  Consider your architecture, try to identify which components are re-usable and build modules out of those. To make an analogy with construction: a window and a door are both re-usable components. A wall is a useful concept, not a brick.
  
  > **Comments:**
  > **username_0**: Thanks for the very detailled answer :) So instead of creating multiple GIT projects, I will use one GIT project with different terraform modules.  
  > **username_1**: This is a matter of choice too (sorry!). I typically advocate for developing modules independently of architectures. So, n git repos , one for each module, independently developed and maintained. Then one repo per architecture, perhaps with different environments. However -- do what works for you!  
  > **username_2**: It is pretty common to start off with one git repo and many terraform modules.  Once that setup starts to be a problem it is straight forward to split it into multiple git repos.  
  
  ---
  
  ### Answer by username_2 on 2023-08-01. Score: 0
  
  [I agree with everything in the two other answers, but I'd like to add something else to the discussion.]
  
  # Terraform challenges
  
  I've had a number of cases with terraform where the number of resources in a terraform module/directory become so huge that terraform takes an unreasonable amount of time to converge on a plan. This often seems to happen with `for_each` loops or other data-driven approaches where a few lines in a data file lead to lots of resources and work for terraform. With these huge directories it can takes hours for terraform to think about it. This is worse on Intel MacBook Pros than ARM/M2-based ones, but I've seen a plan that took hours on an Intel CPU take 15-30 minutes on an ARM processor.
  
  Another issue is provider upgrades. These can lead to needing to do state transformations and often require debugging and iteration to get right. Do you want this debugging to be happening on all of your infrastructure? Probably not.
  
  # Solutions
  
  * As others have said, using modules to keep your code organized is a very good habit to get into with terraform.
  * Use environments - you can have different values for variables per environment and adding environments doesn't entail writing more code.
  * Keep logical things together in the same directory.
  * Beyond common providers that you need everywhere (like vault), minimize the number of different providers in each directory. I wouldn't develop something for AWS in the same directory where I'm managing datadog or another cloud service.
  * Use remote state references to share data internally. Terraform modules that are just `local`s that get turned into `output`s are good for centralizing information like domain names, IP lists, and other things so you're not copying and pasting this information all over the place.
  
  
  ---
  
  ### Answer by username_3 on 2023-08-01. Score: 0
  
  All three ways will work (modules mentioned in answers as third answer), in some cases better, in some - well.
  
  But the exercise you have to do is you have to think, how your infrastructure will look in the nearest future. This is, as it was mentioned, a design and architecture decision. For sure you have some goals to achieve and this should be a base for your design.
  
  You have to find the answers to couple of questions:
  
  * how your infrastructure will be multiplied?
  * do you allow to re-use of specific elements of the infrastructure (for example VPC)?
  * are you going to serve these templates through Platform Engineering teams - self-service solutions?
  * how do you deal with multiple environments?
  * how do you manage the state?
  * how do you orchestrate different modules/templates/environments?
  * how do you want to manage the whole SDLC of your infrastructure?
  
  These questions are the starting point for your design. I know, it sounds like I complicate things, but I try to think about big picture. And believe me, it will be very important later.
  
  For example: if my solution is simple and I will run it generally only once, I will go with the simplest solution. If I know I will re-use some components, I will go with modules. If I will re-use my modules in wider way (many projects, env, etc), I will go with modules stored in registry. If I go even further, it is time to think about dedicated tools, like [Spacelift](https://spacelift.io/) (disclosure: I am Developer Advocate there). But if you want to orchestrate very complicated setup, with multiple teams, projects, accounts (or subscriptions), modules, dependencies, you should, really, look at tools like this.
  
  
  ---
  
  