Answers.Body: <p>I started this out as a comment but it got too long. Maybe not a complete answers, but some thoughts on your proposed solution.</p>
  <p>I think the key point to contest (when talking to your management) is this:</p>
  <blockquote>
  <p>We had presented migrations as a solution here already, but the above is the current focus of exploration.</p>
  </blockquote>
  <p>What's important to realize is that by rolling your own content merging/propagation strategy <em>you're still using migrations</em>. You're just not using any of the proven, useful tools for performing migrations. I feel like I can hear the conversation where somebody went &quot;But can't we just<sup>1</sup> …&quot; and the proposed solution was merging some part of the content from a pre-prod environment with the production environment. But where are the boundaries between those? Are they even crystal clear?</p>
  <p>It's not as simple as grabbing a couple of tables (the content and the user data, respectively) from each database and merging them together. For one, content is always split across multiple tables. Many fields store content in their own dedicated tables, so what if you're using one of those fields on both regular entries (coming from pre-prod) and user accounts (coming from production). So you have to find a way to merge those, while retaining all sorts of constraints on the database level. By sidestepping Craft's API and going directly to the database, you basically have to ensure you know all the ins and outs of how and where Craft stores data and handle all edge-cases that Craft already handles for you. Of course, you'll also have to maintain your custom solution whenever a new Craft version breaks something in your migration code.</p>
  <p>In terms of talking to management – I'd start by talking about risks and benefits. What is the risk of updating content on a production environment? Some internal inconsistencies while changes are being made? On the other hand, what's the risk of rolling your own migration script with difficult rules for merging content? Accidentally dropping important user data? High maintenance costs for constant updates? It's just not worth it.</p>
  <h2>Better solutions</h2>
  <p>I'd say what you really want is a way to export content from your pre-prod environment to production in a predictable way. Exporting is even built into Craft. Importing could be done with the <a href="https://plugins.craftcms.com/feed-me" rel="nofollow noreferrer">Feedme plugin</a>. If you want to get fancier than manual imports/exports, you could build a small plugin with a button that just pushes content from one environment to the other (target, authentication keys etc can come from environment variables). This way, you could still update content across the entire side in one atomic step, but you could use Crafts API and keep your sanity.</p>
  <p>Another approach: Craft 3 separates content from config and the deployment/migration processes are clearly geared towards that separation. So if migrating content alongside code and configuration is an important requirement – how about using another CMS that is better suited for that task? <a href="https://statamic.com/" rel="nofollow noreferrer">Statamic</a> has a git plugin that allows you to track content changes in git. And there are other platforms that allow you to separate production data from site content/copy, in particular headless platforms like <a href="https://strapi.io/" rel="nofollow noreferrer">Strapi</a>. Maybe one of those would work better for your usecase.</p>
  <p><sup>1</sup> <a href="https://signalvnoise.com/posts/439-four-letter-words" rel="nofollow noreferrer">https://signalvnoise.com/posts/439-four-letter-words</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-06T13:45:00.690Z
Answers.Text: I can't thank you enough for this thoughtful response. There's a lot to consider here and I feel much better with this validation and insight. Thanks again!!
Answers.UserDisplayName: Christopher Spence
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-06T13:55:33.287Z
Answers.Text: @ChristopherSpence I'm glad it came out helpful and not just a rant ^^ I'd be really interested in reading how this turns out! Maybe you can post an update in your question once you've decided how to solve this problem ...
Answers.UserDisplayName: MoritzLost
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-06T17:07:01.357Z
Answers.Text: Will do, thanks again!
Answers.UserDisplayName: Christopher Spence
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-06T18:22:32.423Z
Answers.Text: Great answer, @MoritzLost
Answers.UserDisplayName: Brad Bell
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-06T13:17:31.443Z
Answers.LastActivityDate: 2021-08-06T18:22:16.773Z
Answers.LastEditDate: 2021-08-06T18:22:16.773Z
Answers.UserDisplayName: MoritzLost
Body: <p>I was on a dev ops call this afternoon and am wondering about the viability of a proposed solution for pushing changes to a Craft product through a dev pipeline.</p>
  <p>For context, this is for a big company with big security/scalability concerns, and their leadership team was thinking through this.</p>
  <p>The idea is that dev and content changes would be applied to a database (Postgres in this case, if it matters) in a low environment (e.g., dev.ourproduct.com). When content or technology changes, a build pipeline would then transfer the entire database (along with whatever requisite code) to a higher environment (e.g., prod.ourproduct.com). During the migration, we would capture the user generated data rows/tables in the higher environment (e.g., users tables, user groups, any custom tables we use to capture user activity in production etc), and then &quot;reapply&quot; this live content and its corresponding data to the recently promoted/imported database.</p>
  <p>The obvious challenge I'm struggling with is that we'd need to reliably track and map the &quot;element&quot; and related objects from the outgoing database into the incoming database. I've only ever used the craftcms api for generating users and elements, is this something that could be leveraged for this case and at scale, with at least tens of thousands users, perhaps more? From what I've read over the years, manually attempting to insert elements/rows/relations into a CraftCMS is dicey business.</p>
  <p>My reaction to the proposal was &quot;ooh boy&quot;. Has anyone traveled this road before or have thoughts about this? Thanks!!</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-06T08:16:29.260Z
Comments.Text: What problem are you actually trying to solve with this approach? Craft is built to strictly seperate content from configuration. By mixing them together you go down a rabbit hole of problems ... for dev and QA, absolutely use separate dev, staging  and production environments. But why would you want to copy content from a staging environment to a live environment? Especially since that content might also have changes in the live environment. How would you tell (in an automated way) how to merge different content states from your production and staging environments?
Comments.UserDisplayName: MoritzLost
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-06T08:19:58.020Z
Comments.Text: Just to be sure ... you are using a [deployment pipeline](https://craftcms.com/knowledge-base/deployment-best-practices) that goes from dev to production (possibly with a staging environment in between), right? If not, your management is absolutely right, you should have one. But don't mix code and config changes with content changes. If you are deploying changes that require simultaneous content changes, [migrations](https://craftcms.com/docs/3.x/extend/migrations.html) are the way to go.
Comments.UserDisplayName: MoritzLost
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-06T12:40:42.710Z
Comments.Text: Thanks @MoritzLost! There will be four environments here, with a pipeline automating promotion between them. I believe the content team will be editing and validating entries on a middle stage (i.e., not dev / not prod). This published content won't be altered in production, users will register for the site, interact with it, favorite it, etc. The user elements, though, are subject to a similar scenario that you identify and it's what I'm trying to get my ahead around. We had presented migrations as a solution here already, but the above is the current focus of exploration.
Comments.UserDisplayName: Christopher Spence
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-06T12:44:50.893Z
Comments.Text: My hunch is that the management team will see the risks inherent here, but I wanted to validate. I suppose if we had to automate it, we'd need to capture extant users and related data rows (across several tables) in a temporary store and script their "re-entry" into the database first as elements, then user rows, then.. then.. then.. It feels gnarly to me and, as you suggest, not well aligned with how Craft is meant to work.
Comments.UserDisplayName: Christopher Spence
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-06T12:46:26.587Z
Comments.Text: I think the rationale here is that we can be guaranteed that the "core" product is identical across the different environments... so it's almost like we're migrating a template through the pipeline, and "dropping the users back in" at the end. Which sounds simple enough in theory, but seems risky and problematic, to me at least, in execution.
Comments.UserDisplayName: Christopher Spence
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-06T12:52:22.747Z
Comments.Text: And I'm also not sure what the impact would be (we're using redis) for session storage for active/logged in users.
Comments.UserDisplayName: Christopher Spence
Tags: <database><migration>
Title: Reapplying Users and Related Content to Database During Pipeline Promotion
ThreadText: # Reapplying Users and Related Content to Database During Pipeline Promotion
  
  (Asked by: username_0 on 2021-08-05)
  
  I was on a dev ops call this afternoon and am wondering about the viability of a proposed solution for pushing changes to a Craft product through a dev pipeline.
  
  For context, this is for a big company with big security/scalability concerns, and their leadership team was thinking through this.
  
  The idea is that dev and content changes would be applied to a database (Postgres in this case, if it matters) in a low environment (e.g., dev.ourproduct.com). When content or technology changes, a build pipeline would then transfer the entire database (along with whatever requisite code) to a higher environment (e.g., prod.ourproduct.com). During the migration, we would capture the user generated data rows/tables in the higher environment (e.g., users tables, user groups, any custom tables we use to capture user activity in production etc), and then "reapply" this live content and its corresponding data to the recently promoted/imported database.
  
  The obvious challenge I'm struggling with is that we'd need to reliably track and map the "element" and related objects from the outgoing database into the incoming database. I've only ever used the craftcms api for generating users and elements, is this something that could be leveraged for this case and at scale, with at least tens of thousands users, perhaps more? From what I've read over the years, manually attempting to insert elements/rows/relations into a CraftCMS is dicey business.
  
  My reaction to the proposal was "ooh boy". Has anyone traveled this road before or have thoughts about this? Thanks!!
  
  **Tags:** database, migration
  
  > **Comments:**
  > **username_1**: What problem are you actually trying to solve with this approach? Craft is built to strictly seperate content from configuration. By mixing them together you go down a rabbit hole of problems ... for dev and QA, absolutely use separate dev, staging  and production environments. But why would you want to copy content from a staging environment to a live environment? Especially since that content might also have changes in the live environment. How would you tell (in an automated way) how to merge different content states from your production and staging environments?  
  > **username_1**: Just to be sure ... you are using a [deployment pipeline](https://craftcms.com/knowledge-base/deployment-best-practices) that goes from dev to production (possibly with a staging environment in between), right? If not, your management is absolutely right, you should have one. But don't mix code and config changes with content changes. If you are deploying changes that require simultaneous content changes, [migrations](https://craftcms.com/docs/3.x/extend/migrations.html) are the way to go.  
  > **username_0**: Thanks @username_1! There will be four environments here, with a pipeline automating promotion between them. I believe the content team will be editing and validating entries on a middle stage (i.e., not dev / not prod). This published content won't be altered in production, users will register for the site, interact with it, favorite it, etc. The user elements, though, are subject to a similar scenario that you identify and it's what I'm trying to get my ahead around. We had presented migrations as a solution here already, but the above is the current focus of exploration.  
  > **username_0**: My hunch is that the management team will see the risks inherent here, but I wanted to validate. I suppose if we had to automate it, we'd need to capture extant users and related data rows (across several tables) in a temporary store and script their "re-entry" into the database first as elements, then user rows, then.. then.. then.. It feels gnarly to me and, as you suggest, not well aligned with how Craft is meant to work.  
  > **username_0**: I think the rationale here is that we can be guaranteed that the "core" product is identical across the different environments... so it's almost like we're migrating a template through the pipeline, and "dropping the users back in" at the end. Which sounds simple enough in theory, but seems risky and problematic, to me at least, in execution.  
  > **username_0**: And I'm also not sure what the impact would be (we're using redis) for session storage for active/logged in users.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-08-06. Score: 1 | Accepted answer
  
  I started this out as a comment but it got too long. Maybe not a complete answers, but some thoughts on your proposed solution.
  
  I think the key point to contest (when talking to your management) is this:
  
  > We had presented migrations as a solution here already, but the above is the current focus of exploration.
  
  What's important to realize is that by rolling your own content merging/propagation strategy *you're still using migrations*. You're just not using any of the proven, useful tools for performing migrations. I feel like I can hear the conversation where somebody went "But can't we just1 …" and the proposed solution was merging some part of the content from a pre-prod environment with the production environment. But where are the boundaries between those? Are they even crystal clear?
  
  It's not as simple as grabbing a couple of tables (the content and the user data, respectively) from each database and merging them together. For one, content is always split across multiple tables. Many fields store content in their own dedicated tables, so what if you're using one of those fields on both regular entries (coming from pre-prod) and user accounts (coming from production). So you have to find a way to merge those, while retaining all sorts of constraints on the database level. By sidestepping Craft's API and going directly to the database, you basically have to ensure you know all the ins and outs of how and where Craft stores data and handle all edge-cases that Craft already handles for you. Of course, you'll also have to maintain your custom solution whenever a new Craft version breaks something in your migration code.
  
  In terms of talking to management – I'd start by talking about risks and benefits. What is the risk of updating content on a production environment? Some internal inconsistencies while changes are being made? On the other hand, what's the risk of rolling your own migration script with difficult rules for merging content? Accidentally dropping important user data? High maintenance costs for constant updates? It's just not worth it.
  
  ## Better solutions
  
  I'd say what you really want is a way to export content from your pre-prod environment to production in a predictable way. Exporting is even built into Craft. Importing could be done with the [Feedme plugin](https://plugins.craftcms.com/feed-me). If you want to get fancier than manual imports/exports, you could build a small plugin with a button that just pushes content from one environment to the other (target, authentication keys etc can come from environment variables). This way, you could still update content across the entire side in one atomic step, but you could use Crafts API and keep your sanity.
  
  Another approach: Craft 3 separates content from config and the deployment/migration processes are clearly geared towards that separation. So if migrating content alongside code and configuration is an important requirement – how about using another CMS that is better suited for that task? [Statamic](https://statamic.com/) has a git plugin that allows you to track content changes in git. And there are other platforms that allow you to separate production data from site content/copy, in particular headless platforms like [Strapi](https://strapi.io/). Maybe one of those would work better for your usecase.
  
  1 <https://signalvnoise.com/posts/439-four-letter-words>
  
  > **Comments:**
  > **username_0**: I can't thank you enough for this thoughtful response. There's a lot to consider here and I feel much better with this validation and insight. Thanks again!!  
  > **username_1**: @username_0 I'm glad it came out helpful and not just a rant ^^ I'd be really interested in reading how this turns out! Maybe you can post an update in your question once you've decided how to solve this problem ...  
  > **username_0**: Will do, thanks again!  
  > **username_2**: Great answer, @username_1  
  
  ---
  
  