Answers: <blockquote>
    <p>The things that would be backed up would be the config files of the servers, the database and uploaded content as these will almost always be unique and necessary.</p>
  </blockquote>
  
  <p>But would they? Config files can go into <code>git</code>. Databases (for the "CI" part) should be 100% creatable from source as well (migrations, fixtures etc.), and may be so on each single test run anyways. I'm not sure what you mean with "uploaded content" in the context of a CI pipeline.</p>
  
  <p>Your goal would be to reduce <em>everything</em> to some kind of middle layer that is acting as a barrier between the hardcore low-level stuff, and the application (including CI/CD). This could be straight Docker - i.e., packaging up <em>everything</em> (including your CI/CD driver, Jenkins or whatever) in Dockerfiles, with everything needed included as configuration - everything in <code>git</code>.</p>
  
  <p>Or a little more advanced, a Kubernetes/OpenShift cluster which does the same; or the respective solution of the cloud providers (AWS, Azure etc.).</p>
  
  <p>Below that border, you are running with classical solutions as you are used to; there should be very little "special" you need to take care of, and it should be easy (as nothing from the application should influence this at all) to make it so you can add nodes without backup/recovery at all. I.e., if some piece of hardware or non-containerized software breaks, you simply throw it away and replace it with a new one.</p>
  
  <p>Dynamic data that needs to persist (e.g., the history of job runs stored by your actual CI driver) will be stored on dedicated volumes (which contain said data, and only it - not interspersed with OS or other files), which, in turn, you can backup with classical tools (offline backups etc.) - the biggest issue, there, is to decide whether to run the backup in "container land", or on the underlying abstraction layer; but aside from that, it would be just copying files around as usual (with whatever tool you like most), not special just because it's a "CI environment".</p>
  
  <p>TL;DR: make as much as possible run in containers, with dedicated volumes; don't backup containers; backup volumes as usual.</p>
  
Answers: CC BY-SA 4.0
Answers: 2018-12-24T17:07:45.593Z
Answers: I think the OP means the database used by the CI system itself, that is not something you can simply recreate as it has all your results in
Answers: Gaius
Answers: CC BY-SA 4.0
Answers: 2018-12-24T17:42:06.247Z
Answers: I've added a paragraph about this, @Gaius.
Answers: AnoE
Answers: CC BY-SA 4.0
Answers: 2018-12-24T22:46:15.710Z
Answers: As you said,  CI, the backups and recovery is taken care of by GIT for the containers.  CD is another beast where configs should be taken care of by Ansible/Chef but the other dynamic elements such as the database and uploaded contents have to be backed up.  Based on what Gaius posted, filesystem snapshot backup tools sound like they may be the way to go.  I'm curious about specific tools for backup and recovery from your experience that work well?  ideally, they can work with tools like Ansible and Chef even on the recovery end.
Answers: Roadkillon101
Answers: CC BY-SA 4.0
Answers: 2018-12-31T03:45:11.757Z
Answers: `Databases (for the "CI" part) should be 100% creatable from source` - this is wholly false. Database *schema* should be 100% creatable from source, database *data* however will not be.
Answers: James Shewey
Answers: CC BY-SA 4.0
Answers: 2018-12-31T09:04:32.010Z
Answers: @JamesShewey, these are CI databases - so unit test fixtures, integration test data and such. Keeping that data external to the DB, be they SQL or code in whatever language you use, seems to be pretty darn important to me, or how would you evolve that together with the code under test... whether you structure your stuff so you actually have to drop the whole DB for each test run, or trust your transactions to take care of most of it, is up to you, of course.
Answers: AnoE
Answers: CC BY-SA 4.0
Answers: 2018-12-24T15:10:33.610Z
Answers: 2018-12-24T17:41:50.527Z
Answers: 2018-12-24T17:41:50.527Z
Answers: AnoE
Answers: <p>I think from a DevOps POV that you are approaching this from slightly the wrong angle. DevOps thinking for example is that there is no need to backup a configuration file; the configuration file is written by a configuration management tool such as Chef and can be recreated at any time. </p>
  
  <p>The database does need to be backed up obviously as that is the truly stateful component of the whole system; you don't mention which you are using but head on over to DBA.SE and someone (perhaps me) can give you some specific pointers on that side of things.</p>
  
  <p>For the uploaded content, if this is outside the database, I'd recommend some filesystem-snapshot based approach, it's easy to do and low overhead. </p>
  
Answers: CC BY-SA 4.0
Answers: 2018-12-24T21:38:46.913Z
Answers: Filesystem snapshot backup does sound like potentially a better approach for both database and even dynamic or uploaded content.  Do you have any recommendations of some of the tools you use for this?  Do any of these tools have a clean recovery system?
  After all, the demons are in the details...
Answers: Roadkillon101
Answers: CC BY-SA 4.0
Answers: 2018-12-24T09:21:56.507Z
Answers: 2018-12-24T09:21:56.507Z
Answers: Gaius
Answers: <blockquote>
    <p>The things that would be backed up would be the config files of the servers, the database and uploaded content as these will almost always be unique and necessary. The containers, VMs, even the webapps would NOT be backed up as these should all be uniform and constantly updated under the CI/CD principles. Currently I use a combination of scp and tar into an archive directory sorted by date and machine. I like to find something better if possible.</p>
  </blockquote>
  
  <p>Config files, containers and VMs <em>should</em> be backed up and versioned by a tool like git or SVN if you are doing <a href="https://devops.stackexchange.com/questions/5847/how-do-configuration-management-tools-deal-with-someone-messing-up-configuration">configuration management properly.</a>. Backing up config files and versioning them is the whole point of <em>configuration</em> management tools like, salt, puppet, ansible, and chef.</p>
  
  <p>As you duly note however, dynamic content such as database data and similarly dynamic data is a more challenging issue. These however, are outside of the scope of <em>configuration</em> management, CI, and CD principles. These principles purposefully remain silent on dealing with these issues: because they have no new answers for these problems. Instead, <a href="https://devops.stackexchange.com/questions/742/what-practices-or-tools-enable-continuous-deployment-of-databases/1220#1220">you should use the many battle-tested techniques and strategies for dealing with this kind of data</a> instead of expecting CI/CD/CM tools to solve problems that they were never intended to address. </p>
  
Answers: CC BY-SA 4.0
Answers: 2019-01-02T20:35:50.423Z
Answers: Basically, if your in the position that you need to RECOVER the production backup of a database then your already in trouble as the database is SO large that it'll take too much time to bring it back online without significant downtime.  At the end of the day you rely on making the database more manageable using various refactoring strategies, rely on proxies and replication and HOPE that your data isn't corrupted (that bad) by whatever (Cloud storage hardware failure, applications failure, etc).
Answers: Roadkillon101
Answers: CC BY-SA 4.0
Answers: 2018-12-31T03:40:13.367Z
Answers: 2018-12-31T03:40:13.367Z
Answers: James Shewey
Body: <p>I have used tar, dump, <a href="http://www.amanda.org/" rel="noreferrer">Amanda</a>, <a href="https://www.emc.com/data-protection/networker.htm" rel="noreferrer">Legato (EMC) Networker</a> and other backup software for making backups of systems.  However, I don't know the best tools for making backups in a CI/CD (Continuous Integration/Continuous Deployment) environment and doing "rolling recoveries" of systems "as you go" which is the case in a DevOps oriented system.</p>
  
  <p>Many of these backup utilities are not necessarily best suited for a CI/CD environment because of the continuous changes taking place on both the development and production environment.</p>
  
  <p>I'm looking at programs like <a href="https://www.borgbackup.org/" rel="noreferrer">Borg</a> 
  and Git based backups like <a href="https://github.com/bup/bup" rel="noreferrer">Bup</a> 
  These programs would allow for many incremental backups and finer granularity of backups and work well with CD tools like Ansible.  Even Ansible has a couple of way to <a href="https://superuser.com/questions/732358/backup-a-server-using-ansible">make backups</a>.  However, there is no clean way to recover the most recent backup in an automated way. </p>
  
  <p>Relying on some form of caching or mirroring of your content on the cluster and hope it's not corrupted is not a way to go :(</p>
  
  <p>The things that would be backed up would be the config files of the servers, the database and uploaded content as these will almost always be unique and necessary.
  The containers, VMs, even the webapps would NOT be backed up as these should all be uniform and constantly updated under the CI/CD principles.
  Currently I use a combination of scp and tar into an archive directory sorted by date and machine.  I like to find something better if possible.</p>
  
  <p>I would like to know what are the best backup and recovery tools used in the CI/CD environment?  and an example of how you would configure it?</p>
  
  <p>I don't expect a "magic bullet" just some possible solutions.</p>
  
Tags: <ansible><cicd><backup>
Title: What are tools that lend themselves for backup and recovery in a CI/CD environment?
UserDisplayName: Roadkillon101
ThreadText: # What are tools that lend themselves for backup and recovery in a CI/CD environment?
  
  (Asked by: username_0 on 2018-12-20)
  
  I have used tar, dump, [Amanda](http://www.amanda.org/), [Legato (EMC) Networker](https://www.emc.com/data-protection/networker.htm) and other backup software for making backups of systems. However, I don't know the best tools for making backups in a CI/CD (Continuous Integration/Continuous Deployment) environment and doing "rolling recoveries" of systems "as you go" which is the case in a DevOps oriented system.
  
  Many of these backup utilities are not necessarily best suited for a CI/CD environment because of the continuous changes taking place on both the development and production environment.
  
  I'm looking at programs like [Borg](https://www.borgbackup.org/)
  and Git based backups like [Bup](https://github.com/bup/bup)
  These programs would allow for many incremental backups and finer granularity of backups and work well with CD tools like Ansible. Even Ansible has a couple of way to [make backups](https://superuser.com/questions/732358/backup-a-server-using-ansible). However, there is no clean way to recover the most recent backup in an automated way.
  
  Relying on some form of caching or mirroring of your content on the cluster and hope it's not corrupted is not a way to go :(
  
  The things that would be backed up would be the config files of the servers, the database and uploaded content as these will almost always be unique and necessary.
  The containers, VMs, even the webapps would NOT be backed up as these should all be uniform and constantly updated under the CI/CD principles.
  Currently I use a combination of scp and tar into an archive directory sorted by date and machine. I like to find something better if possible.
  
  I would like to know what are the best backup and recovery tools used in the CI/CD environment? and an example of how you would configure it?
  
  I don't expect a "magic bullet" just some possible solutions.
  
  **Tags:** ansible, cicd, backup
  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_3 on 2018-12-31. Score: 3 | Accepted answer
  
  > The things that would be backed up would be the config files of the servers, the database and uploaded content as these will almost always be unique and necessary. The containers, VMs, even the webapps would NOT be backed up as these should all be uniform and constantly updated under the CI/CD principles. Currently I use a combination of scp and tar into an archive directory sorted by date and machine. I like to find something better if possible.
  
  Config files, containers and VMs *should* be backed up and versioned by a tool like git or SVN if you are doing [configuration management properly.](https://devops.stackexchange.com/questions/5847/how-do-configuration-management-tools-deal-with-someone-messing-up-configuration). Backing up config files and versioning them is the whole point of *configuration* management tools like, salt, puppet, ansible, and chef.
  
  As you duly note however, dynamic content such as database data and similarly dynamic data is a more challenging issue. These however, are outside of the scope of *configuration* management, CI, and CD principles. These principles purposefully remain silent on dealing with these issues: because they have no new answers for these problems. Instead, [you should use the many battle-tested techniques and strategies for dealing with this kind of data](https://devops.stackexchange.com/questions/742/what-practices-or-tools-enable-continuous-deployment-of-databases/1220#1220) instead of expecting CI/CD/CM tools to solve problems that they were never intended to address.
  
  > **Comments:**
  > **username_0**: Basically, if your in the position that you need to RECOVER the production backup of a database then your already in trouble as the database is SO large that it'll take too much time to bring it back online without significant downtime.  At the end of the day you rely on making the database more manageable using various refactoring strategies, rely on proxies and replication and HOPE that your data isn't corrupted (that bad) by whatever (Cloud storage hardware failure, applications failure, etc).  
  
  ---
  
  ### Answer by username_1 on 2018-12-24. Score: 0
  
  > The things that would be backed up would be the config files of the servers, the database and uploaded content as these will almost always be unique and necessary.
  
  But would they? Config files can go into `git`. Databases (for the "CI" part) should be 100% creatable from source as well (migrations, fixtures etc.), and may be so on each single test run anyways. I'm not sure what you mean with "uploaded content" in the context of a CI pipeline.
  
  Your goal would be to reduce *everything* to some kind of middle layer that is acting as a barrier between the hardcore low-level stuff, and the application (including CI/CD). This could be straight Docker - i.e., packaging up *everything* (including your CI/CD driver, Jenkins or whatever) in Dockerfiles, with everything needed included as configuration - everything in `git`.
  
  Or a little more advanced, a Kubernetes/OpenShift cluster which does the same; or the respective solution of the cloud providers (AWS, Azure etc.).
  
  Below that border, you are running with classical solutions as you are used to; there should be very little "special" you need to take care of, and it should be easy (as nothing from the application should influence this at all) to make it so you can add nodes without backup/recovery at all. I.e., if some piece of hardware or non-containerized software breaks, you simply throw it away and replace it with a new one.
  
  Dynamic data that needs to persist (e.g., the history of job runs stored by your actual CI driver) will be stored on dedicated volumes (which contain said data, and only it - not interspersed with OS or other files), which, in turn, you can backup with classical tools (offline backups etc.) - the biggest issue, there, is to decide whether to run the backup in "container land", or on the underlying abstraction layer; but aside from that, it would be just copying files around as usual (with whatever tool you like most), not special just because it's a "CI environment".
  
  TL;DR: make as much as possible run in containers, with dedicated volumes; don't backup containers; backup volumes as usual.
  
  > **Comments:**
  > **username_2**: I think the OP means the database used by the CI system itself, that is not something you can simply recreate as it has all your results in  
  > **username_1**: I've added a paragraph about this, @username_2.  
  > **username_0**: As you said,  CI, the backups and recovery is taken care of by GIT for the containers.  CD is another beast where configs should be taken care of by Ansible/Chef but the other dynamic elements such as the database and uploaded contents have to be backed up.  Based on what username_2 posted, filesystem snapshot backup tools sound like they may be the way to go.  I'm curious about specific tools for backup and recovery from your experience that work well?  ideally, they can work with tools like Ansible and Chef even on the recovery end.  
  > **username_3**: `Databases (for the "CI" part) should be 100% creatable from source` - this is wholly false. Database *schema* should be 100% creatable from source, database *data* however will not be.  
  > **username_1**: @username_3, these are CI databases - so unit test fixtures, integration test data and such. Keeping that data external to the DB, be they SQL or code in whatever language you use, seems to be pretty darn important to me, or how would you evolve that together with the code under test... whether you structure your stuff so you actually have to drop the whole DB for each test run, or trust your transactions to take care of most of it, is up to you, of course.  
  
  ---
  
  ### Answer by username_2 on 2018-12-24. Score: -1
  
  I think from a DevOps POV that you are approaching this from slightly the wrong angle. DevOps thinking for example is that there is no need to backup a configuration file; the configuration file is written by a configuration management tool such as Chef and can be recreated at any time.
  
  The database does need to be backed up obviously as that is the truly stateful component of the whole system; you don't mention which you are using but head on over to DBA.SE and someone (perhaps me) can give you some specific pointers on that side of things.
  
  For the uploaded content, if this is outside the database, I'd recommend some filesystem-snapshot based approach, it's easy to do and low overhead.
  
  > **Comments:**
  > **username_0**: Filesystem snapshot backup does sound like potentially a better approach for both database and even dynamic or uploaded content.  Do you have any recommendations of some of the tools you use for this?  Do any of these tools have a clean recovery system? After all, the demons are in the details...  
  
  ---
  
  