Answers: <p>I can think of only one case where overhead of running workers in separate containers is justified: if your setup uses docker swarm for clustered deployment. This way you will get all of the HA benefits.</p>
  
  <p>Otherwise I don't see a reason to complicate such tasks, especially if they must use strictly the same codebase (which leads me to believe that they share one domain of concerns, so to say). It might seem elegant, but I just don't see enough benefits</p>
  
  <p>p.s. any split of such kind will force you to manage interconnection and complicate setup for local development</p>
  
Answers: CC BY-SA 3.0
Answers: 2018-01-16T17:07:58.550Z
Answers: I guess the complexity can be handled well in my case as I already employ docker-compose. Nonetheless, your point is valid since the load can currently be handled with a single instance and no need to make things more complex than they have to be
Answers: Moritz
Answers: CC BY-SA 3.0
Answers: 2018-01-16T13:25:42.737Z
Answers: 2018-01-16T13:25:42.737Z
Answers: Wintermute
Body: <p>My setup is based on a Flask app that has a PostgreSQL and Redis database. In addition, a worker process is started to handle asynchronous tasks such as sending emails. I use Gunicorn to create multiple app processes. Passing of jobs from the app processes to the worker process is done via Redis.</p>
  
  <p>The question is, should the worker process be started in a separate container despite being based on the same code? Currently I use a start script with the relevant bit:</p>
  
  <pre><code>gunicorn --bind=0.0.0.0:8000 --workers=3 manage:app &amp;
  python -u manage.py run_worker
  </code></pre>
  
  <p>Also if they are to be separated, how can I ensure that they reuse the common file system container? Additionally, when scaling either Gunicorn with the main app or the worker processes, should additional instances of the container be created or the number of processes within a container be increased?</p>
  
  <p>Related: <a href="https://devops.stackexchange.com/q/2089/5879">What are the advantages of dockerizing nginx and php in different containers?</a></p>
  
Tags: <docker><containers>
Title: Should worker processes of web applications be placed in separate containers?
UserDisplayName: Moritz
ThreadText: # Should worker processes of web applications be placed in separate containers?
  
  (Asked by: username_0 on 2018-01-16)
  
  My setup is based on a Flask app that has a PostgreSQL and Redis database. In addition, a worker process is started to handle asynchronous tasks such as sending emails. I use Gunicorn to create multiple app processes. Passing of jobs from the app processes to the worker process is done via Redis.
  
  The question is, should the worker process be started in a separate container despite being based on the same code? Currently I use a start script with the relevant bit:
  
  ```
  gunicorn --bind=0.0.0.0:8000 --workers=3 manage:app &
  python -u manage.py run_worker
  
  ```
  
  Also if they are to be separated, how can I ensure that they reuse the common file system container? Additionally, when scaling either Gunicorn with the main app or the worker processes, should additional instances of the container be created or the number of processes within a container be increased?
  
  Related: [What are the advantages of dockerizing nginx and php in different containers?](https://devops.stackexchange.com/q/2089/5879)
  
  **Tags:** docker, containers
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-01-16. Score: 6 | Accepted answer
  
  I can think of only one case where overhead of running workers in separate containers is justified: if your setup uses docker swarm for clustered deployment. This way you will get all of the HA benefits.
  
  Otherwise I don't see a reason to complicate such tasks, especially if they must use strictly the same codebase (which leads me to believe that they share one domain of concerns, so to say). It might seem elegant, but I just don't see enough benefits
  
  p.s. any split of such kind will force you to manage interconnection and complicate setup for local development
  
  > **Comments:**
  > **username_0**: I guess the complexity can be handled well in my case as I already employ docker-compose. Nonetheless, your point is valid since the load can currently be handled with a single instance and no need to make things more complex than they have to be  
  
  ---
  
  