Body: <p>In current deployment of a web app we are using svn to deploy a specific revision from a repository. The amount of data transferred is just the diff between the currently deployed revision and the revision being deployed, which is rather small. Deployments go to hundreds of machines, hundreds times a week. </p>
  
  <p>We would like to run the app in a container for extra testing, fast reverts and service separation, but there is a problem. When creating a container, we copy the app in there as one layer, then run the container through all tests, security scanning, etc. But during deployments, the amount of data transferred is the entire layer with the app each time. There has to be some way to minimize the amount of data transferred just to the diff between revisions, that is standard and has some tools around it. It does not seem like we are the only ones trying to solve the problem.</p>
  
  <p>So before we implement some home grown solution, what would you do?</p>
  
Comments: CC BY-SA 4.0
Comments: 2018-11-26T05:06:54.817Z
Comments: wouldn't it be acceptable to run the `svn up` inside the container instead of `COPY` of the full source code? The container layers would then be a bit smaller and you can iteratively move forward with the base image.
Comments: Bruce Becker
Comments: CC BY-SA 4.0
Comments: 2018-11-26T16:45:46.430Z
Comments: The app is only a small part of the mono-repo, but yes, we considered some sort of home grown solution where we do a full COPY every week or so and multiple small COPY for revision differences from that base container with the full copy layer. Should we do copy for each update and have multiple layers each day or should we use one big copy from the weekly or daily base? There are questions. But that is something that will take a lot of work to code and maintain and I was hoping someone has already faced similar problems and has a solution somewhere ready to reuse.
Comments: Jiri Klouda
Comments: CC BY-SA 4.0
Comments: 2018-11-29T11:45:45.047Z
Comments: Have you read this https://github.com/moby/moby/issues/21950. Let us better understand why you need this; because *its slow* or because of *expensive* bandwidth or both or curiosity only? Are you using docker on target machines already?
Comments: Kyslik
Comments: CC BY-SA 4.0
Comments: 2018-11-29T17:55:40.660Z
Comments: Yes, I've read it before I asked, it was one of the pages coming in the searches and we already use overlay driver so we don't hit this issue. I am trying to see if someone implemented the multiple nested containers and the management of the updating of the base containers. If two or three nested levels would be better, how would look the orchestration of the build pipeline through jenkins for example. If someone already solved the issues, maybe has a solution template available somewhere on github, where they already solved the dozens of small issues that usually pop up.
Comments: Jiri Klouda
Tags: <docker><containers><continuous-deployment><testing>
Title: How to deploy large web app in container minimal data transfer?
UserDisplayName: Jiri Klouda
ThreadText: # How to deploy large web app in container minimal data transfer?
  
  (Asked by: username_0 on 2018-11-26)
  
  In current deployment of a web app we are using svn to deploy a specific revision from a repository. The amount of data transferred is just the diff between the currently deployed revision and the revision being deployed, which is rather small. Deployments go to hundreds of machines, hundreds times a week.
  
  We would like to run the app in a container for extra testing, fast reverts and service separation, but there is a problem. When creating a container, we copy the app in there as one layer, then run the container through all tests, security scanning, etc. But during deployments, the amount of data transferred is the entire layer with the app each time. There has to be some way to minimize the amount of data transferred just to the diff between revisions, that is standard and has some tools around it. It does not seem like we are the only ones trying to solve the problem.
  
  So before we implement some home grown solution, what would you do?
  
  **Tags:** docker, containers, continuous-deployment, testing
  
  > **Comments:**
  > **username_1**: wouldn't it be acceptable to run the `svn up` inside the container instead of `COPY` of the full source code? The container layers would then be a bit smaller and you can iteratively move forward with the base image.  
  > **username_0**: The app is only a small part of the mono-repo, but yes, we considered some sort of home grown solution where we do a full COPY every week or so and multiple small COPY for revision differences from that base container with the full copy layer. Should we do copy for each update and have multiple layers each day or should we use one big copy from the weekly or daily base? There are questions. But that is something that will take a lot of work to code and maintain and I was hoping someone has already faced similar problems and has a solution somewhere ready to reuse.  
  > **username_2**: Have you read this https://github.com/moby/moby/issues/21950. Let us better understand why you need this; because *its slow* or because of *expensive* bandwidth or both or curiosity only? Are you using docker on target machines already?  
  > **username_0**: Yes, I've read it before I asked, it was one of the pages coming in the searches and we already use overlay driver so we don't hit this issue. I am trying to see if someone implemented the multiple nested containers and the management of the updating of the base containers. If two or three nested levels would be better, how would look the orchestration of the build pipeline through jenkins for example. If someone already solved the issues, maybe has a solution template available somewhere on github, where they already solved the dozens of small issues that usually pop up.  
  
  ---
  
  ## 0 answers
  
  