Answers: <p>I finally found out the problem: the fluent container needs some time to start up; because it's started with <code>--detach</code> the subsequent containers try to connect before fluentd is actually running, only the 3rd instance of the flask application connects when fluentd is already up. So, delaying the startup of the remaining containers solves my problem.</p>
  
Answers: CC BY-SA 4.0
Answers: 2019-08-20T19:01:01.967Z
Answers: 2019-08-20T19:01:01.967Z
Answers: user2683038
Body: <p>I am facing a strange problem with logging when I set up a network of three docker containers each running a small flask app.</p>
  
  <p>My python application looks as follows:</p>
  
  <pre><code>from flask import Flask, request
  import sys
  import logging
  import logging.handlers
  import os
  
  app = Flask(__name__)
  
  logger = logging.getLogger('Logger')
  logger.setLevel(logging.DEBUG)
  log_handler = logging.StreamHandler(sys.stdout)
  logger.addHandler(log_handler)
  
  service_name=""
  if 'SERVICE_NAME' in os.environ:
      service_name = os.environ['SERVICE_NAME']
  
  logger = logging.LoggerAdapter(logger, {"serviceName" : service_name})
  
  @app.route("/ping")
  def ping():
      logger.info(f'answering ping request')
  
      return f'{service_name} OK'
  
  if __name__ == "__main__":
      logger.info("Started")
  
      app.run(debug=True, host='0.0.0.0')
  </code></pre>
  
  <p>I start three instances of this in a docker network behind nginx:</p>
  
  <pre><code>docker network create tNet
  
  docker run -it -d -p 24224:24224 -v `pwd`:/fluentd/etc -e FLUENTD_CONF=fluentd.conf --name=fluentd_logging fluent/fluentd
  
  docker run -d --log-driver=fluentd --name ms1 -e SERVICE_NAME="ms1" --network tNet testflask
  docker run -d --log-driver=fluentd --name ms2 -e SERVICE_NAME="ms2" --network tNet testflask
  docker run -d --log-driver=fluentd --name ms3 -e SERVICE_NAME="ms3" --network tNet testflask
  
  docker run -d --name=nginx --network tNet --volume `pwd`/nginx.conf:/etc/nginx/conf.d/default.conf -p 2000:2000 -p 3000:3000 nginx
  </code></pre>
  
  <p>The nginx configuration is as follows:</p>
  
  <pre><code>upstream app {
      server ms1:5000;
      server ms2:5000;
      server ms3:5000;
  }
  
  server {
      listen 2000;
  
      location / {
          proxy_pass http://app;
      }
  }
  </code></pre>
  
  <p>My fluent configuration looks as follows:</p>
  
  <pre><code>&lt;source&gt;
      @type forward
  &lt;/source&gt;
  
  &lt;match&gt;
      @type stdout
  &lt;/match&gt;
  </code></pre>
  
  <p>When I use curl I can see that each instance of the service is responding:</p>
  
  <pre><code>curl localhost:2000/ping
  ms1 OK
  curl localhost:2000/ping
  ms2 OK
  curl localhost:2000/ping
  ms3 OK
  </code></pre>
  
  <p>When I however inspect the logs I can only see logs from the 3rd instance of the service:</p>
  
  <pre><code>docker logs &lt;fluent container ID&gt; | grep -Eis "answering ping"
  
  19-08-06 15:34:46.000000000 +0000 be338a3d5aa0: {"container_id":"be338a3d5aa0b8920366fdf1ba2a267082cee47e681adcea16b6c51c9c82cf0d","container_name":"/ms3","source":"stdout","log":"answering ping request"}
  </code></pre>
  
  <p>Has anyone an idea? Thank you in advance.</p>
  
Tags: <microservices><logging><flask>
Title: Problem with logging in docker when using fluentd for logging of three flask apps running in a docker network behind an nginx
UserDisplayName: user2683038
ThreadText: # Problem with logging in docker when using fluentd for logging of three flask apps running in a docker network behind an nginx
  
  (Asked by: username_0 on 2019-08-06)
  
  I am facing a strange problem with logging when I set up a network of three docker containers each running a small flask app.
  
  My python application looks as follows:
  
  ```
  from flask import Flask, request
  import sys
  import logging
  import logging.handlers
  import os
  
  app = Flask(__name__)
  
  logger = logging.getLogger('Logger')
  logger.setLevel(logging.DEBUG)
  log_handler = logging.StreamHandler(sys.stdout)
  logger.addHandler(log_handler)
  
  service_name=""
  if 'SERVICE_NAME' in os.environ:
      service_name = os.environ['SERVICE_NAME']
  
  logger = logging.LoggerAdapter(logger, {"serviceName" : service_name})
  
  @app.route("/ping")
  def ping():
      logger.info(f'answering ping request')
  
      return f'{service_name} OK'
  
  if __name__ == "__main__":
      logger.info("Started")
  
      app.run(debug=True, host='0.0.0.0')
  
  ```
  
  I start three instances of this in a docker network behind nginx:
  
  ```
  docker network create tNet
  
  docker run -it -d -p 24224:24224 -v `pwd`:/fluentd/etc -e FLUENTD_CONF=fluentd.conf --name=fluentd_logging fluent/fluentd
  
  docker run -d --log-driver=fluentd --name ms1 -e SERVICE_NAME="ms1" --network tNet testflask
  docker run -d --log-driver=fluentd --name ms2 -e SERVICE_NAME="ms2" --network tNet testflask
  docker run -d --log-driver=fluentd --name ms3 -e SERVICE_NAME="ms3" --network tNet testflask
  
  docker run -d --name=nginx --network tNet --volume `pwd`/nginx.conf:/etc/nginx/conf.d/default.conf -p 2000:2000 -p 3000:3000 nginx
  
  ```
  
  The nginx configuration is as follows:
  
  ```
  upstream app {
      server ms1:5000;
      server ms2:5000;
      server ms3:5000;
  }
  
  server {
      listen 2000;
  
      location / {
          proxy_pass http://app;
      }
  }
  
  ```
  
  My fluent configuration looks as follows:
  
  ```
  <source>
      @type forward
  </source>
  
  <match>
      @type stdout
  </match>
  
  ```
  
  When I use curl I can see that each instance of the service is responding:
  
  ```
  curl localhost:2000/ping
  ms1 OK
  curl localhost:2000/ping
  ms2 OK
  curl localhost:2000/ping
  ms3 OK
  
  ```
  
  When I however inspect the logs I can only see logs from the 3rd instance of the service:
  
  ```
  docker logs <fluent container ID> | grep -Eis "answering ping"
  
  19-08-06 15:34:46.000000000 +0000 be338a3d5aa0: {"container_id":"be338a3d5aa0b8920366fdf1ba2a267082cee47e681adcea16b6c51c9c82cf0d","container_name":"/ms3","source":"stdout","log":"answering ping request"}
  
  ```
  
  Has anyone an idea? Thank you in advance.
  
  **Tags:** microservices, logging, flask
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2019-08-20. Score: 0 | Accepted answer
  
  I finally found out the problem: the fluent container needs some time to start up; because it's started with `--detach` the subsequent containers try to connect before fluentd is actually running, only the 3rd instance of the flask application connects when fluentd is already up. So, delaying the startup of the remaining containers solves my problem.
  
  
  ---
  
  