Answers: <p>I ended up following this path</p>
  <p>Rather than use cron, i used a scheduling framework (in my case Bree.js) which allowed me to &quot;jobify&quot; the tasks and also execute these tasks in separate processes - this allowed me the vertical expansion needed (ie: using more processor cores)</p>
  <p>A docker container contains the dispatch (scheduling) and jobs - so once the container is started, jobs (and associated processes) are created/managed within that container</p>
  <p>Eventually, when I need to scale horizontally, the &quot;dispatch&quot; function will morph to running a job whose sole purpose is to post an event  (eg: Azure service bus).  I will have an orchestrator running various containers who read from the queue and process accordingly.</p>
  
Answers: CC BY-SA 4.0
Answers: 2021-01-25T23:48:57.220Z
Answers: 2021-01-25T23:48:57.220Z
Answers: 300baud
Body: <p>I have a node.js ETL application that transforms data from 1..n locations and from each location there will be 0..m transactions.
  The application is started via cron on a configurable interval, currently 15 minutes.
  Currently, the pattern employed is to use Promise.map({concurrency:x}) at both the location and transaction level to obtain a combined degree of concurrency to achieve the maximum throughput (constrained by DB I/O.  </p>
  
  <p>Right now the app runs on a single Ubuntu VM, processing 6 locations and ~ 1000 transactions/location.  So minimal data processing overhead.  The pattern works.
  But when there are 1000+ locations and possibly more transactions per interval, the pattern will still work of course, but the time to complete all locations will grow.
  Being node, the process is single threaded and so the pattern is not leveraging the available cores.</p>
  
  <p>So I’m thinking about how to change the pattern to scale.  Options I’m considering (not mutually exclusive)</p>
  
  <ul>
  <li>containerise the app, start one for every n locations - based on some manual determination of # of locations to process per container. </li>
  <li>remain non containerised but use fork() to leverage additional CPU’s and partition on n locations</li>
  </ul>
  
  <p>Based on my reading to date, my approach is:</p>
  
  <ul>
  <li>forget “fork"</li>
  <li>containerise the app
  
  <ul>
  <li>spin up x apps, x being determined based on manual determination since the parameters are known vs dynamic</li>
  <li>each container would be responsible for n locations</li>
  <li>each app would set its own “cron” to orchestrate its batch processing</li>
  </ul></li>
  </ul>
  
  <p>Not being an expert in docker (but reasonably familiar) and certainly a newbie to orchestration, I would appreciate any guidance on how containers / orchestration might help.</p>
  
Tags: <docker><kubernetes><node.js><orchestration><scalability>
Title: Pattern(s) to scale node.js based ETL
UserDisplayName: 300baud
ThreadText: # Pattern(s) to scale node.js based ETL
  
  (Asked by: username_0 on 2020-05-29)
  
  I have a node.js ETL application that transforms data from 1..n locations and from each location there will be 0..m transactions.
  The application is started via cron on a configurable interval, currently 15 minutes.
  Currently, the pattern employed is to use Promise.map({concurrency:x}) at both the location and transaction level to obtain a combined degree of concurrency to achieve the maximum throughput (constrained by DB I/O.
  
  Right now the app runs on a single Ubuntu VM, processing 6 locations and ~ 1000 transactions/location. So minimal data processing overhead. The pattern works.
  But when there are 1000+ locations and possibly more transactions per interval, the pattern will still work of course, but the time to complete all locations will grow.
  Being node, the process is single threaded and so the pattern is not leveraging the available cores.
  
  So I’m thinking about how to change the pattern to scale. Options I’m considering (not mutually exclusive)
  
  * containerise the app, start one for every n locations - based on some manual determination of # of locations to process per container.
  * remain non containerised but use fork() to leverage additional CPU’s and partition on n locations
  
  Based on my reading to date, my approach is:
  
  * forget “fork"
  * containerise the app
    + spin up x apps, x being determined based on manual determination since the parameters are known vs dynamic
    + each container would be responsible for n locations
    + each app would set its own “cron” to orchestrate its batch processing
  
  Not being an expert in docker (but reasonably familiar) and certainly a newbie to orchestration, I would appreciate any guidance on how containers / orchestration might help.
  
  **Tags:** docker, kubernetes, node.js, orchestration, scalability
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2021-01-25. Score: 1
  
  I ended up following this path
  
  Rather than use cron, i used a scheduling framework (in my case Bree.js) which allowed me to "jobify" the tasks and also execute these tasks in separate processes - this allowed me the vertical expansion needed (ie: using more processor cores)
  
  A docker container contains the dispatch (scheduling) and jobs - so once the container is started, jobs (and associated processes) are created/managed within that container
  
  Eventually, when I need to scale horizontally, the "dispatch" function will morph to running a job whose sole purpose is to post an event (eg: Azure service bus). I will have an orchestrator running various containers who read from the queue and process accordingly.
  
  
  ---
  
  