Answers: <p>Solution was to put a docker pull into my launch script ... as in</p>
  
  <pre><code>docker-compose -f ${GKE_COMPOSE_YAML} pull
  </code></pre>
  
  <p>just before issuing </p>
  
  <pre><code>docker-compose -f ${GKE_COMPOSE_YAML} up -d
  </code></pre>
  
  <p>which seems strange since most of the time it works OK without issuing pull ... I am using a local docker registry</p>
  
Answers: CC BY-SA 4.0
Answers: 2018-06-05T14:29:37.937Z
Answers: 2018-06-05T14:29:37.937Z
Answers: Scott Stensland
Body: <p>I have a devops container inside of which I execute <code>docker build</code> and <code>docker-compose up</code> commands which run OK then (on same server or different server with same code) sometimes incorrectly fail to <code>Recreate</code> sibling containers after their images are rebuilt.</p>
  
  <p>My process successfully rebuilds a pair of two underlying docker images however </p>
  
  <pre><code>docker-compose up 
  </code></pre>
  
  <p>sometimes fails to pickup that change and incorrectly says up-to-date on freshly rebuilt images when it should be saying Recreating</p>
  
  <p>I am executing identical code across several servers each with varying OS ( ubuntu 16.04 or debian 8 or debian 9) and varying docker versions ( either  18.04.0-ce-rc1  or  18.03.0-ce ) ... am currently working to track down root cause however see good and bad behavior across each of above OS and docker versions</p>
  
  <p>This issue does not seem related to <a href="https://github.com/docker/compose/issues/3574" rel="nofollow noreferrer">https://github.com/docker/compose/issues/3574</a></p>
  
  <p>Yes I am tagging the images and doing a push to local docker repository</p>
  
  <pre><code>docker build --tag localhost:5000/hygge/loudweb-admin .
  </code></pre>
  
  <p>docker push to my local docker repository is </p>
  
  <pre><code>docker push localhost:5000/hygge/loudweb-admin
  </code></pre>
  
  <p>it also does similar to above image loudweb-enduser</p>
  
  <p>here my my docker-compose.yaml</p>
  
  <pre><code>version: '3'
  services:
  
  nodejs-admin:
    image: ${GKE_APP_REPO_PREFIX}/${PROJECT_ID}/loudweb-admin
    container_name: loud_admin
    restart: always
    depends_on:
      - loudmongo
      - loudmail
    volumes:
      - /cryptdata4/var/log/loudlog-admin:/loudlog-admin
      - /cryptdata5/var/log/blobs:/blobs
      - /cryptdata5/var:/cryptdata5/var
      - /cryptdata5/var/tools:/tools
      - /cryptdata6/var/log/loudlog-enduser:/loudlog-enduser
      - $SOURCE_REPO_DIR/tests:/tmp/tests
      - ${TMPDIR_GRAND_PARENT}/curr/loud-build/${PROJECT_ID}/webapp/admin/bundle:/tmp
    environment:
      - MONGO_SERVICE_HOST=loudmongo
      - MONGO_SERVICE_PORT=$GKE_MONGO_PORT 
      - MONGO_URL=mongodb://loudmongo:$GKE_MONGO_PORT/test
      - METEOR_SETTINGS=${METEOR_SETTINGS}
      - MAIL_URL=smtp://support@${GKE_DOMAIN_NAME}:blah@loudmail:587/
      - GKE_NOTIF_TASK_OVERDUE=$GKE_NOTIF_TASK_OVERDUE
      - GKE_NOTIF_TASK_PUSH=$GKE_NOTIF_TASK_PUSH
      - GKE_NOTIF_PLANS_RECUR=$GKE_NOTIF_PLANS_RECUR
    links:
      - loudmongo
      - loudmail
    ports:
     - 127.0.0.1:3001:3001
    working_dir: /tmp
    command: /usr/bin/supervisord -c /etc/supervisor/conf.d/supervisord.conf
  
  nodejs-enduser:
    image: ${GKE_APP_REPO_PREFIX}/${PROJECT_ID}/loudweb-enduser
    container_name: loud_enduser
    restart: always
    depends_on:
      - nodejs-admin
      - loudmongo
      - loudmail
    volumes:
      - /cryptdata6/var/log/loudlog-enduser:/loudlog-enduser
      - /cryptdata5/var/log/blobs:/blobs
      - ${TMPDIR_GRAND_PARENT}/curr/loud-build/${PROJECT_ID}/webapp/enduser/bundle:/tmp
      - ${TMPDIR_GRAND_PARENT}/curr/loud-build/${PROJECT_ID}/webapp/admin/bundle/programs/server/assets/app/config/apn-cert.pem:/private/config/apn-cert.pem
      - ${TMPDIR_GRAND_PARENT}/curr/loud-build/${PROJECT_ID}/webapp/admin/bundle/programs/server/assets/app/config/apn-key.pem:/private/config/apn-key.pem
    environment:
      - MONGO_SERVICE_HOST=loudmongo
      - MONGO_SERVICE_PORT=$GKE_MONGO_PORT 
      - MONGO_URL=mongodb://loudmongo:$GKE_MONGO_PORT/test
      - METEOR_SETTINGS=${METEOR_SETTINGS}
      - MAIL_URL=smtp://support@${GKE_DOMAIN_NAME}:puMDcxNmEwMDU5MmJjZjNiNzQ3M2ZlNTJjNDYyOGI1NTggIC0Ksy@loudmail:587/
    links:
      - loudmongo
      - loudmail
    ports:
     - 127.0.0.1:3000:3000
    working_dir: /tmp
    command: /usr/bin/supervisord -c /etc/supervisor/conf.d/supervisord.conf
  
  loudmongo:
    image: mongo
    container_name: loud_mongo
    restart: always
    ports:
     - 127.0.0.1:$GKE_MONGO_PORT:$GKE_MONGO_PORT
    volumes:
     - /cryptdata7/var/data/db:/data/db
  
  loudmail:
    image: ${GKE_APP_REPO_PREFIX}/${PROJECT_ID}/loudmail
    hostname: mail
    domainname: ${GKE_DOMAIN_NAME}
    container_name: loud_mail
    restart: always
    environment:
      - DMS_DEBUG=1
      - ENABLE_MANAGESIEVE=1
      - ENABLE_FAIL2BAN=1
      - SA_TAG=2.0
      - SA_TAG2=6.31
      - SA_KILL=6.31
      - SSL_TYPE=letsencrypt
    ports:
      - 127.0.0.1:25:25
      - 127.0.0.1:110:110
      - 127.0.0.1:143:143
      - 127.0.0.1:587:587
      - 127.0.0.1:993:993
      - 127.0.0.1:995:995
      - 127.0.0.1:4190:4190
  
  
  loud-devops:
    image: dind
    container_name: loud_devops
    restart: always
    ports:
      - 127.0.0.1:9000:9000
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock   
      - /usr/bin/docker:/usr/bin/docker   
      - /usr/lib/x86_64-linux-gnu/libltdl.so.7:/usr/lib/x86_64-linux-gnu/libltdl.so.7  
      - /usr/lib/x86_64-linux-gnu/libgpm.so.2:/usr/lib/x86_64-linux-gnu/libgpm.so.2
      - /home/asya/src/github.com/blahblah:/inner_home/asya/src/github.com/blahblah 
      - /home/asya/.docker:/inner_home/asya/.docker
      - /cryptdata5/var/tools/inner_home/asya:/inner_home/asya
      - $GKE_DIND_SUPERVISOR_LOG_DIR:/var/log/supervisor
      - /cryptdata5/var/tools/usr/local/go:/usr/local/go
      - /cryptdata6/var/log/tmp/asya01:/cryptdata6/var/log/tmp/asya01
      - /cryptdata6/var/log/tmp/shared:/cryptdata6/var/log/tmp/shared
      - /cryptdata5/var/tools/usr/local/bin:/usr/local/bin 
      - /cryptdata:/cryptdata
      - /cryptdata4:/cryptdata4
      - /cryptdata5:/cryptdata5
      - /cryptdata6:/cryptdata6
      - /cryptdata7:/cryptdata7
      - /etc/letsencrypt/live:/etc/letsencrypt/live
      - /usr/local/ssl:/usr/local/ssl
      - /var/lib/docker:/var/lib/docker
    command: /usr/bin/supervisord -c /etc/supervisor/supervisord.conf
  </code></pre>
  
  <p>The bottom most container in above ( loud-devops ) contains a webhook daemon which listens for a git push to trigger a rebuild cycle. While inside that container the system issues all my docker command including docker build create fresh images for two containers ( nodejs-admin and nodejs-enduser ) then while inside the container I issue the </p>
  
  <pre><code>docker-compose -f /docker-compose.yaml up -d 
  </code></pre>
  
  <p>which then sometimes incorrectly displays</p>
  
  <pre><code>loud_devops is up-to-date
  loud_mongo is up-to-date
  loud_mail is up-to-date
  loud_admin is up-to-date
  loud_enduser is up-to-date
  </code></pre>
  
  <p>other times it incorrectly displays</p>
  
  <pre><code>docker-compose -f /docker-compose.yml up -d
  Recreating loud_devops ...
  Recreating loud_mongo ...
  Recreating loud_mail ...
  </code></pre>
  
  <p>which should NEVER happen since those 3 images are never rebuilt ... to me that scenario is a blatant docker logic bug ... </p>
  
  <p>whereas other times works OK and correctly displays</p>
  
  <pre><code>loud_mongo is up-to-date
  loud_mail is up-to-date
  loud_devops is up-to-date
  Recreating loud_admin ... done
  Recreating loud_enduser ... done
  </code></pre>
  
  <p>On the host of each server I have docker installed then as you can see in above yaml for container loud-devops  I mount as volumes various docker dirs to enable the execution of docker in docker from inside container loud-devops </p>
  
  <p>A critical piece of evidence :  on a server which has finished its rebuild cycle where it successfully rebuilt both images loudweb-enduser and loudweb-admin  then incorrectly shows</p>
  
  <pre><code>docker-compose -f /docker-compose.yml up -d 
  loud_mail is up-to-date
  loud_mongo is up-to-date
  loud_devops is up-to-date
  loud_admin is up-to-date
  loud_enduser is up-to-date
  </code></pre>
  
  <p>where the <code>docker-compose up</code> command gets executed by the system while inside container  loud-devops ... IF I then manually login to that same server then manually issue the same command it works correctly :</p>
  
  <pre><code>docker-compose -f /docker-compose.yml up -d 
  loud_devops is up-to-date
  loud_mail is up-to-date
  loud_mongo is up-to-date
  Recreating loud_admin ... done
  Recreating loud_enduser ... done
  </code></pre>
  
Comments: CC BY-SA 3.0
Comments: 2018-04-04T12:45:17.273Z
Comments: Do you tag the built images? this might happen if you build & tag an image with the tag which already exists = rewrite the previous version, then when you do docker-compose up it won't recreate because the version is the same. I would like to know if --force-rebuild and/or --always-recreate-deps solves the issue and I would also like to see the docker-compose file, I might be able to advice further afterwards
Comments: Tomas Tibensky
Comments: CC BY-SA 3.0
Comments: 2018-04-04T17:11:16.043Z
Comments: @TomášTibenský  to respond to your questions - the rebuild logic is outside the docker-compose.yaml so I have never used nor believe I can use --force-rebuild or --always-recreate-deps
Comments: Scott Stensland
Comments: CC BY-SA 3.0
Comments: 2018-04-06T11:38:45.447Z
Comments: what about running docker-compose -f /docker-compose.yml pull and then docker-compose -f /docker-compose.yml up -d ?
Comments: Tomas Tibensky
Tags: <docker><docker-compose><docker-in-docker>
Title: docker-compose fails to recognize when images are rebuilt sometimes ... often same code is OK
UserDisplayName: Scott Stensland
ThreadText: # docker-compose fails to recognize when images are rebuilt sometimes ... often same code is OK
  
  (Asked by: username_0 on 2018-03-29)
  
  I have a devops container inside of which I execute `docker build` and `docker-compose up` commands which run OK then (on same server or different server with same code) sometimes incorrectly fail to `Recreate` sibling containers after their images are rebuilt.
  
  My process successfully rebuilds a pair of two underlying docker images however
  
  ```
  docker-compose up 
  
  ```
  
  sometimes fails to pickup that change and incorrectly says up-to-date on freshly rebuilt images when it should be saying Recreating
  
  I am executing identical code across several servers each with varying OS ( ubuntu 16.04 or debian 8 or debian 9) and varying docker versions ( either 18.04.0-ce-rc1 or 18.03.0-ce ) ... am currently working to track down root cause however see good and bad behavior across each of above OS and docker versions
  
  This issue does not seem related to <https://github.com/docker/compose/issues/3574>
  
  Yes I am tagging the images and doing a push to local docker repository
  
  ```
  docker build --tag localhost:5000/hygge/loudweb-admin .
  
  ```
  
  docker push to my local docker repository is
  
  ```
  docker push localhost:5000/hygge/loudweb-admin
  
  ```
  
  it also does similar to above image loudweb-enduser
  
  here my my docker-compose.yaml
  
  ```
  version: '3'
  services:
  
  nodejs-admin:
    image: ${GKE_APP_REPO_PREFIX}/${PROJECT_ID}/loudweb-admin
    container_name: loud_admin
    restart: always
    depends_on:
      - loudmongo
      - loudmail
    volumes:
      - /cryptdata4/var/log/loudlog-admin:/loudlog-admin
      - /cryptdata5/var/log/blobs:/blobs
      - /cryptdata5/var:/cryptdata5/var
      - /cryptdata5/var/tools:/tools
      - /cryptdata6/var/log/loudlog-enduser:/loudlog-enduser
      - $SOURCE_REPO_DIR/tests:/tmp/tests
      - ${TMPDIR_GRAND_PARENT}/curr/loud-build/${PROJECT_ID}/webapp/admin/bundle:/tmp
    environment:
      - MONGO_SERVICE_HOST=loudmongo
      - MONGO_SERVICE_PORT=$GKE_MONGO_PORT 
      - MONGO_URL=mongodb://loudmongo:$GKE_MONGO_PORT/test
      - METEOR_SETTINGS=${METEOR_SETTINGS}
      - MAIL_URL=smtp://support@${GKE_DOMAIN_NAME}:blah@loudmail:587/
      - GKE_NOTIF_TASK_OVERDUE=$GKE_NOTIF_TASK_OVERDUE
      - GKE_NOTIF_TASK_PUSH=$GKE_NOTIF_TASK_PUSH
      - GKE_NOTIF_PLANS_RECUR=$GKE_NOTIF_PLANS_RECUR
    links:
      - loudmongo
      - loudmail
    ports:
     - 127.0.0.1:3001:3001
    working_dir: /tmp
    command: /usr/bin/supervisord -c /etc/supervisor/conf.d/supervisord.conf
  
  nodejs-enduser:
    image: ${GKE_APP_REPO_PREFIX}/${PROJECT_ID}/loudweb-enduser
    container_name: loud_enduser
    restart: always
    depends_on:
      - nodejs-admin
      - loudmongo
      - loudmail
    volumes:
      - /cryptdata6/var/log/loudlog-enduser:/loudlog-enduser
      - /cryptdata5/var/log/blobs:/blobs
      - ${TMPDIR_GRAND_PARENT}/curr/loud-build/${PROJECT_ID}/webapp/enduser/bundle:/tmp
      - ${TMPDIR_GRAND_PARENT}/curr/loud-build/${PROJECT_ID}/webapp/admin/bundle/programs/server/assets/app/config/apn-cert.pem:/private/config/apn-cert.pem
      - ${TMPDIR_GRAND_PARENT}/curr/loud-build/${PROJECT_ID}/webapp/admin/bundle/programs/server/assets/app/config/apn-key.pem:/private/config/apn-key.pem
    environment:
      - MONGO_SERVICE_HOST=loudmongo
      - MONGO_SERVICE_PORT=$GKE_MONGO_PORT 
      - MONGO_URL=mongodb://loudmongo:$GKE_MONGO_PORT/test
      - METEOR_SETTINGS=${METEOR_SETTINGS}
      - MAIL_URL=smtp://support@${GKE_DOMAIN_NAME}:puMDcxNmEwMDU5MmJjZjNiNzQ3M2ZlNTJjNDYyOGI1NTggIC0Ksy@loudmail:587/
    links:
      - loudmongo
      - loudmail
    ports:
     - 127.0.0.1:3000:3000
    working_dir: /tmp
    command: /usr/bin/supervisord -c /etc/supervisor/conf.d/supervisord.conf
  
  loudmongo:
    image: mongo
    container_name: loud_mongo
    restart: always
    ports:
     - 127.0.0.1:$GKE_MONGO_PORT:$GKE_MONGO_PORT
    volumes:
     - /cryptdata7/var/data/db:/data/db
  
  loudmail:
    image: ${GKE_APP_REPO_PREFIX}/${PROJECT_ID}/loudmail
    hostname: mail
    domainname: ${GKE_DOMAIN_NAME}
    container_name: loud_mail
    restart: always
    environment:
      - DMS_DEBUG=1
      - ENABLE_MANAGESIEVE=1
      - ENABLE_FAIL2BAN=1
      - SA_TAG=2.0
      - SA_TAG2=6.31
      - SA_KILL=6.31
      - SSL_TYPE=letsencrypt
    ports:
      - 127.0.0.1:25:25
      - 127.0.0.1:110:110
      - 127.0.0.1:143:143
      - 127.0.0.1:587:587
      - 127.0.0.1:993:993
      - 127.0.0.1:995:995
      - 127.0.0.1:4190:4190
  
  loud-devops:
    image: dind
    container_name: loud_devops
    restart: always
    ports:
      - 127.0.0.1:9000:9000
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock   
      - /usr/bin/docker:/usr/bin/docker   
      - /usr/lib/x86_64-linux-gnu/libltdl.so.7:/usr/lib/x86_64-linux-gnu/libltdl.so.7  
      - /usr/lib/x86_64-linux-gnu/libgpm.so.2:/usr/lib/x86_64-linux-gnu/libgpm.so.2
      - /home/asya/src/github.com/blahblah:/inner_home/asya/src/github.com/blahblah 
      - /home/asya/.docker:/inner_home/asya/.docker
      - /cryptdata5/var/tools/inner_home/asya:/inner_home/asya
      - $GKE_DIND_SUPERVISOR_LOG_DIR:/var/log/supervisor
      - /cryptdata5/var/tools/usr/local/go:/usr/local/go
      - /cryptdata6/var/log/tmp/asya01:/cryptdata6/var/log/tmp/asya01
      - /cryptdata6/var/log/tmp/shared:/cryptdata6/var/log/tmp/shared
      - /cryptdata5/var/tools/usr/local/bin:/usr/local/bin 
      - /cryptdata:/cryptdata
      - /cryptdata4:/cryptdata4
      - /cryptdata5:/cryptdata5
      - /cryptdata6:/cryptdata6
      - /cryptdata7:/cryptdata7
      - /etc/letsencrypt/live:/etc/letsencrypt/live
      - /usr/local/ssl:/usr/local/ssl
      - /var/lib/docker:/var/lib/docker
    command: /usr/bin/supervisord -c /etc/supervisor/supervisord.conf
  
  ```
  
  The bottom most container in above ( loud-devops ) contains a webhook daemon which listens for a git push to trigger a rebuild cycle. While inside that container the system issues all my docker command including docker build create fresh images for two containers ( nodejs-admin and nodejs-enduser ) then while inside the container I issue the
  
  ```
  docker-compose -f /docker-compose.yaml up -d 
  
  ```
  
  which then sometimes incorrectly displays
  
  ```
  loud_devops is up-to-date
  loud_mongo is up-to-date
  loud_mail is up-to-date
  loud_admin is up-to-date
  loud_enduser is up-to-date
  
  ```
  
  other times it incorrectly displays
  
  ```
  docker-compose -f /docker-compose.yml up -d
  Recreating loud_devops ...
  Recreating loud_mongo ...
  Recreating loud_mail ...
  
  ```
  
  which should NEVER happen since those 3 images are never rebuilt ... to me that scenario is a blatant docker logic bug ...
  
  whereas other times works OK and correctly displays
  
  ```
  loud_mongo is up-to-date
  loud_mail is up-to-date
  loud_devops is up-to-date
  Recreating loud_admin ... done
  Recreating loud_enduser ... done
  
  ```
  
  On the host of each server I have docker installed then as you can see in above yaml for container loud-devops I mount as volumes various docker dirs to enable the execution of docker in docker from inside container loud-devops
  
  A critical piece of evidence : on a server which has finished its rebuild cycle where it successfully rebuilt both images loudweb-enduser and loudweb-admin then incorrectly shows
  
  ```
  docker-compose -f /docker-compose.yml up -d 
  loud_mail is up-to-date
  loud_mongo is up-to-date
  loud_devops is up-to-date
  loud_admin is up-to-date
  loud_enduser is up-to-date
  
  ```
  
  where the `docker-compose up` command gets executed by the system while inside container loud-devops ... IF I then manually login to that same server then manually issue the same command it works correctly :
  
  ```
  docker-compose -f /docker-compose.yml up -d 
  loud_devops is up-to-date
  loud_mail is up-to-date
  loud_mongo is up-to-date
  Recreating loud_admin ... done
  Recreating loud_enduser ... done
  
  ```
  
  **Tags:** docker, docker-compose, docker-in-docker
  
  > **Comments:**
  > **username_1**: Do you tag the built images? this might happen if you build & tag an image with the tag which already exists = rewrite the previous version, then when you do docker-compose up it won't recreate because the version is the same. I would like to know if --force-rebuild and/or --always-recreate-deps solves the issue and I would also like to see the docker-compose file, I might be able to advice further afterwards  
  > **username_0**: @TomášTibenský  to respond to your questions - the rebuild logic is outside the docker-compose.yaml so I have never used nor believe I can use --force-rebuild or --always-recreate-deps  
  > **username_1**: what about running docker-compose -f /docker-compose.yml pull and then docker-compose -f /docker-compose.yml up -d ?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2018-06-05. Score: 0 | Accepted answer
  
  Solution was to put a docker pull into my launch script ... as in
  
  ```
  docker-compose -f ${GKE_COMPOSE_YAML} pull
  
  ```
  
  just before issuing
  
  ```
  docker-compose -f ${GKE_COMPOSE_YAML} up -d
  
  ```
  
  which seems strange since most of the time it works OK without issuing pull ... I am using a local docker registry
  
  
  ---
  
  