Answers: <p>Checkout flux(<a href="https://github.com/weaveworks/flux/" rel="nofollow noreferrer">https://github.com/weaveworks/flux/</a>), it introduces a new kubernetes crd called HelmRelease, in which you can include your helm release info(like the chart repo, name, version, namespace, and values. You can even pull your charts from a git repo).</p>
  
  <p>The flux controller monitors a git repo of your choice(you can monitor a path inside the repo as well, which I think might best suit your requirements). And applies all the yaml/json files(including HelmRelease definitions) found in that path periodically.</p>
  
  <p>When a new HelmRelease definition is created, this other component of flux called helm-operator takes over and try to create/update your helm release to the desired state. </p>
  
  <p>The nice thing about this is you can define the desired cluster state using various yaml/json files and flux will try to keep it in sync. If you want to crate a new cluster, just copy the files to a new folder, deploy your cluster, configure flux on it with the new folder path and it'll do the rest of the work for you. 
  Mind you flux is not a full ci/cd solution, so you can't run your shell scripts in it, you will have to find a different way to encode that information into yaml files. </p>
  
  <p>Take a look here(<a href="https://github.com/stefanprodan/gitops-helm" rel="nofollow noreferrer">https://github.com/stefanprodan/gitops-helm</a>) for a minimal git repo. </p>
  
Answers: CC BY-SA 4.0
Answers: 2019-03-19T02:41:02.983Z
Answers: 2019-03-19T02:41:02.983Z
Answers: Madushan
Answers: <p>My 2 cents.  We have similar requirement like you do.  We end up having multiple values files in the form of <code>values-&lt;namespace&gt;.yaml</code> and it should come with the defaults set to <code>values.yaml</code></p>
  
  <p>We also have some shell scripts to deploy to the namespace.  Something like this deploy.sh --namespace app.</p>
  
  <p>It is not without its problem, the number of files are steadily growing.  But so far we are ok with it.</p>
  
Answers: CC BY-SA 4.0
Answers: 2019-03-17T22:52:35.797Z
Answers: I’m using the same solution plus using kms to decrypt the secrets with helm secrets plugin. So each team can’t decrypt the other teams secrets.
Answers: RuBiCK
Answers: CC BY-SA 4.0
Answers: 2019-03-17T22:49:26.790Z
Answers: 2019-03-17T22:49:26.790Z
Answers: Bal Chua
Answers: <p>I have made the scripts and tools that we use to solve these problems opensource on GitHub calling it <a href="https://github.com/ocd-scm/ocd-meta/blob/master/README.md" rel="nofollow noreferrer">OCD</a>. You mention that you have thought the following feels wrong:</p>
  <blockquote>
  <ul>
  <li><p>Store all the environment configurations in separate &quot;configurations&quot; git repo</p>
  </li>
  <li><p>Use ansible or something similar to store the scripts and configurations for each environment.</p>
  </li>
  </ul>
  </blockquote>
  <p>With OCD we use <a href="https://github.com/roboll/helmfile" rel="nofollow noreferrer">Helmfile</a> which makes it easy to manage many helm release in either one of a small number of simple yaml file. If you are familiar with Ansible then it gives all the benefits of that tool but it only manages helm releases. It makes total sense to put the <code>helmfile.yml</code> configuration into Git. These act as the playbooks to setup your environment. They shouldn't live in the application repos it should be in one or more &quot;environments repo(s)&quot;.</p>
  <p>We have created a generic chart to install secrets and another to install configmaps. Your folder structure might look like:</p>
  <pre><code>|____config
  | |____nexmo.yaml.secret // &lt;- gpg encrypted secret
  | |____reepay.yaml.secret
  | |____backoffice-db.yaml.secret
  | |____helmfile.yaml // &lt;- hemlfile manages 3 secret releases and inline configmaps
  |____backoffice-app
  | |____helmfile.yaml // &lt;- hemlfile manages 1 app
  |____frontoffice-app
  | |____helmfile.yaml // &lt;- hemlfile manages 1 app
  </code></pre>
  <p>In the above example, we have three helmfiles. There is one that manages shared configuration and one each for two apps that use the shared config. Since shared config is going to be multipe secret and configmaps it just uses the two generic charts. It uses the <a href="https://github.com/ocd-scm/ocd-secret" rel="nofollow noreferrer">generic secret chart</a> to stamp out many separately <a href="https://github.com/ocd-scm/ocd-meta/wiki/Encrypting-Secrets#introduction" rel="nofollow noreferrer">encrypted secrets</a> using different values files. Likewise, one <a href="https://github.com/ocd-scm/ocd-configmap" rel="nofollow noreferrer">generic configmap chart</a> is used to stamp out many configmaps by feeding it different values.</p>
  <p>With OCD you setup webhooks on your config repo to point at an <a href="https://github.com/adnanh/webhook" rel="nofollow noreferrer">adnanh/webhook</a>. When you push a change to the config repo the webhook fires. The webhook runs a script to checkout the config repo, decrypt the secrets, then does <code>helmfile apply</code> in each folder that has a <code>helmfile.yaml</code>. That does a helm diff on every release in each helmfile and so will upgrade any secrets, configmaps or application deployments where the yaml has changed.</p>
  <p>We have a wizard script to install the webhook into a new environment. This means that a team can spin up a new environment using the config in git quickly. Since it is loading the config from git it is an accurate process. Here is a video of <a href="https://vimeo.com/manage/318887046/general" rel="nofollow noreferrer">setting up a new environment in six minutes</a> which would be quicker if I was not making a video.</p>
  <p>Helmfile reduces the burden of having to go through all the files to try to find all the settings to change for a new team environment. The <code>helmfile.yaml</code> captures whatever you are overriding over the chart defaults. All the configmaps and secrets are organised to be easily found in one place. Helmfile has quite a lot of capabilities that make managing config and managing many helm releases easier.</p>
  <p>We happen to be using openshift kubernetes so our application deployment chart for microservices uses openshift specific yaml. Yet the secrets and configmaps charts are vanilla k8s. So the OCD approach and scripts to manage config should work with any k8s distro. We plan to have a 1.0 release of OCD in the next couple of weeks. I was considering doing a demo of running OCD on vanilla k8s to manage configuration. So I would be really interested in your thoughts and perhaps we can collaborate. Just raise an issue on the ocd-meta github asking about running it on the version of k8s you are using.</p>
  
Answers: CC BY-SA 4.0
Answers: 2019-03-23T22:51:17.280Z
Answers: 2019-03-24T07:59:11.237Z
Answers: 2020-06-18T08:28:10.803Z
Answers: simbo1905
Answers: <p>I would recommend using something like <a href="https://argoproj.github.io/cd/" rel="nofollow noreferrer">ArgoCD</a> - a declarative, continuous  delivery for Kubernetes manifests.</p>
  <p>With ArgoCD you can:</p>
  <ul>
  <li>structure the Kubernetes objects into separate repositories instead of  one huge repo and therefore assign permissions as needed,</li>
  <li>set up RBAC permissions around ArgoCD projects, so each team can control only &quot;their&quot; objects,</li>
  <li>orchestrate deployments into multiple clusters from a central place if desired.</li>
  </ul>
  <p>ArgoCD works with plain Kubernetes manifests, Helm charts, Kustomize dynamic manifests ... just to scratch the surface.</p>
  
Answers: CC BY-SA 4.0
Answers: 2022-03-28T14:13:53.803Z
Answers: 2022-03-28T14:13:53.803Z
Answers: David Lukac
Body: <p>I'm looking for feedback on how we can improve our team environment and staging deployment process.
  We currently have 10 agile teams, and set up our Kubernetes deployments through Helm as follows:</p>
  
  <p>All our helm charts are in a single git repo. At the top layer of the repo we have a collection of bash scripts (<code>deploy.sh</code>, <code>buildSecrets.sh</code>, and <code>buildConfigMaps.sh</code>) and some json configuration files. (<code>namespaceCharts.json</code>, <code>secrets.json</code>, <code>configmaps.json</code>)</p>
  
  <p>The two main configuration files are a long list of secrets, which are encrypted, and a long list of config map environment variables, 80% of which are shared between helm charts. The <code>namespaceCharts.json</code> defines which charts need to be deployed into each namespace grouping.</p>
  
  <p>For each environment that we deploy into, we clone the repo, change the values of the  <code>secrets.json</code> and <code>configmaps.json</code> and then run the scripts passing in our base namespace name. (For example, the team name)</p>
  
  <p>This results in us basically creating a series of namespaces such as <code>teamA-frontend</code> and <code>TeamA-backend</code> each of which contains the right microservices with the connection strings to the correct DB etc.</p>
  
  <p>We are currently hitting a few main roadblocks.<br>
   1. It's hard to deploy to multiple clusters or new clusters.<br>
   2. It's hard to get a single source of truth for the configurations.<br>
   3. It's hard to set up a brand new Team environment because we have to go through the entire configuration file to see what needs to be changed or kept.  </p>
  
  <p>I have come up with three ways of moving forward, all of which "feel wrong" to me.<br>
  1.  Store all the environment configurations in separate "configurations" git repo.<br>
  2. Create a configuration service with a DB table defining each environment, and create a pub/sub feature to dynamically grab configuration at runtime, as well as dynamically grab configurations in the .sh scripts.<br>
  3. Use ansible or something similar to store the scripts and configurations for each environment.</p>
  
  <p>Are we doing this all wrong? Are we missing some obvious best practice to do this better? Are one of those three ideas actually the right approach?</p>
  
Tags: <kubernetes><configuration-management><helm>
Title: How to manage 10+ team environments with Helm and Kubernetes?
UserDisplayName: avi
ThreadText: # How to manage 10+ team environments with Helm and Kubernetes?
  
  (Asked by: username_0 on 2019-03-17)
  
  I'm looking for feedback on how we can improve our team environment and staging deployment process.
  We currently have 10 agile teams, and set up our Kubernetes deployments through Helm as follows:
  
  All our helm charts are in a single git repo. At the top layer of the repo we have a collection of bash scripts (`deploy.sh`, `buildSecrets.sh`, and `buildConfigMaps.sh`) and some json configuration files. (`namespaceCharts.json`, `secrets.json`, `configmaps.json`)
  
  The two main configuration files are a long list of secrets, which are encrypted, and a long list of config map environment variables, 80% of which are shared between helm charts. The `namespaceCharts.json` defines which charts need to be deployed into each namespace grouping.
  
  For each environment that we deploy into, we clone the repo, change the values of the `secrets.json` and `configmaps.json` and then run the scripts passing in our base namespace name. (For example, the team name)
  
  This results in us basically creating a series of namespaces such as `teamA-frontend` and `TeamA-backend` each of which contains the right microservices with the connection strings to the correct DB etc.
  
  We are currently hitting a few main roadblocks.  
  1. It's hard to deploy to multiple clusters or new clusters.  
  2. It's hard to get a single source of truth for the configurations.  
  3. It's hard to set up a brand new Team environment because we have to go through the entire configuration file to see what needs to be changed or kept.
  
  I have come up with three ways of moving forward, all of which "feel wrong" to me.  
  1. Store all the environment configurations in separate "configurations" git repo.  
  2. Create a configuration service with a DB table defining each environment, and create a pub/sub feature to dynamically grab configuration at runtime, as well as dynamically grab configurations in the .sh scripts.  
  3. Use ansible or something similar to store the scripts and configurations for each environment.
  
  Are we doing this all wrong? Are we missing some obvious best practice to do this better? Are one of those three ideas actually the right approach?
  
  **Tags:** kubernetes, configuration-management, helm
  
  
  ---
  
  ## 4 answers
  
  ### Answer by username_1 on 2019-03-19. Score: 1
  
  Checkout flux(<https://github.com/weaveworks/flux/>), it introduces a new kubernetes crd called HelmRelease, in which you can include your helm release info(like the chart repo, name, version, namespace, and values. You can even pull your charts from a git repo).
  
  The flux controller monitors a git repo of your choice(you can monitor a path inside the repo as well, which I think might best suit your requirements). And applies all the yaml/json files(including HelmRelease definitions) found in that path periodically.
  
  When a new HelmRelease definition is created, this other component of flux called helm-operator takes over and try to create/update your helm release to the desired state.
  
  The nice thing about this is you can define the desired cluster state using various yaml/json files and flux will try to keep it in sync. If you want to crate a new cluster, just copy the files to a new folder, deploy your cluster, configure flux on it with the new folder path and it'll do the rest of the work for you.
  Mind you flux is not a full ci/cd solution, so you can't run your shell scripts in it, you will have to find a different way to encode that information into yaml files.
  
  Take a look here(<https://github.com/stefanprodan/gitops-helm>) for a minimal git repo.
  
  
  ---
  
  ### Answer by username_2 on 2019-03-17. Score: 1
  
  My 2 cents. We have similar requirement like you do. We end up having multiple values files in the form of `values-<namespace>.yaml` and it should come with the defaults set to `values.yaml`
  
  We also have some shell scripts to deploy to the namespace. Something like this deploy.sh --namespace app.
  
  It is not without its problem, the number of files are steadily growing. But so far we are ok with it.
  
  > **Comments:**
  > **username_3**: I’m using the same solution plus using kms to decrypt the secrets with helm secrets plugin. So each team can’t decrypt the other teams secrets.  
  
  ---
  
  ### Answer by username_4 on 2019-03-23. Score: 1
  
  I have made the scripts and tools that we use to solve these problems opensource on GitHub calling it [OCD](https://github.com/ocd-scm/ocd-meta/blob/master/README.md). You mention that you have thought the following feels wrong:
  
  > * Store all the environment configurations in separate "configurations" git repo
  > * Use ansible or something similar to store the scripts and configurations for each environment.
  
  With OCD we use [Helmfile](https://github.com/roboll/helmfile) which makes it easy to manage many helm release in either one of a small number of simple yaml file. If you are familiar with Ansible then it gives all the benefits of that tool but it only manages helm releases. It makes total sense to put the `helmfile.yml` configuration into Git. These act as the playbooks to setup your environment. They shouldn't live in the application repos it should be in one or more "environments repo(s)".
  
  We have created a generic chart to install secrets and another to install configmaps. Your folder structure might look like:
  
  ```
  |____config
  | |____nexmo.yaml.secret // <- gpg encrypted secret
  | |____reepay.yaml.secret
  | |____backoffice-db.yaml.secret
  | |____helmfile.yaml // <- hemlfile manages 3 secret releases and inline configmaps
  |____backoffice-app
  | |____helmfile.yaml // <- hemlfile manages 1 app
  |____frontoffice-app
  | |____helmfile.yaml // <- hemlfile manages 1 app
  
  ```
  
  In the above example, we have three helmfiles. There is one that manages shared configuration and one each for two apps that use the shared config. Since shared config is going to be multipe secret and configmaps it just uses the two generic charts. It uses the [generic secret chart](https://github.com/ocd-scm/ocd-secret) to stamp out many separately [encrypted secrets](https://github.com/ocd-scm/ocd-meta/wiki/Encrypting-Secrets#introduction) using different values files. Likewise, one [generic configmap chart](https://github.com/ocd-scm/ocd-configmap) is used to stamp out many configmaps by feeding it different values.
  
  With OCD you setup webhooks on your config repo to point at an [adnanh/webhook](https://github.com/adnanh/webhook). When you push a change to the config repo the webhook fires. The webhook runs a script to checkout the config repo, decrypt the secrets, then does `helmfile apply` in each folder that has a `helmfile.yaml`. That does a helm diff on every release in each helmfile and so will upgrade any secrets, configmaps or application deployments where the yaml has changed.
  
  We have a wizard script to install the webhook into a new environment. This means that a team can spin up a new environment using the config in git quickly. Since it is loading the config from git it is an accurate process. Here is a video of [setting up a new environment in six minutes](https://vimeo.com/manage/318887046/general) which would be quicker if I was not making a video.
  
  Helmfile reduces the burden of having to go through all the files to try to find all the settings to change for a new team environment. The `helmfile.yaml` captures whatever you are overriding over the chart defaults. All the configmaps and secrets are organised to be easily found in one place. Helmfile has quite a lot of capabilities that make managing config and managing many helm releases easier.
  
  We happen to be using openshift kubernetes so our application deployment chart for microservices uses openshift specific yaml. Yet the secrets and configmaps charts are vanilla k8s. So the OCD approach and scripts to manage config should work with any k8s distro. We plan to have a 1.0 release of OCD in the next couple of weeks. I was considering doing a demo of running OCD on vanilla k8s to manage configuration. So I would be really interested in your thoughts and perhaps we can collaborate. Just raise an issue on the ocd-meta github asking about running it on the version of k8s you are using.
  
  
  ---
  
  ### Answer by username_5 on 2022-03-28. Score: 0
  
  I would recommend using something like [ArgoCD](https://argoproj.github.io/cd/) - a declarative, continuous delivery for Kubernetes manifests.
  
  With ArgoCD you can:
  
  * structure the Kubernetes objects into separate repositories instead of one huge repo and therefore assign permissions as needed,
  * set up RBAC permissions around ArgoCD projects, so each team can control only "their" objects,
  * orchestrate deployments into multiple clusters from a central place if desired.
  
  ArgoCD works with plain Kubernetes manifests, Helm charts, Kustomize dynamic manifests ... just to scratch the surface.
  
  
  ---
  
  