Answers.Body: <p>Just noting that this is no longer the case (referring to the <a href="https://civicrm.stackexchange.com/a/33153/35">original answer</a>) - the jobs are processed in chunks via ajax, or in the background if the server is configured that way</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-24T08:30:44.217Z
Answers.Text: Just to clarify "this is no longer the case" refer to the long-running [ajax] POST request for imports with multiple [ajax] GET poll requests? i.e. those are now multiple ajax POST requests, with a few (how many?) items in each request?
Answers.UserDisplayName: artfulrobot
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-26T07:15:06.043Z
Answers.Text: The number per request is driven by a setting which appears to be on the misc setting page https://github.com/civicrm/civicrm-core/blob/c56953d6e41f4695a531a8db91a8c6f0eccf7e7f/CRM/Import/Parser.php#L623
Answers.UserDisplayName: eileen
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-22T22:34:25.110Z
Answers.LastActivityDate: 2024-04-26T11:12:54.633Z
Answers.LastEditDate: 2024-04-26T11:12:54.633Z
Answers.UserDisplayName: eileen
Answers.Body: <p>It depends on the job.</p>
  
  <p>Import jobs (Contacts, Contributions...) work by having one long-running POST request, and then your browser polls for updates on progress.</p>
  
  <p>This means your browser/server/interenet connection need to be able to cope with the time that POST request requires. Whether a job completes after apparently timing out depends on how PHP is set up on your site.</p>
  
  <p>For example:</p>
  
  <ul>
  <li><p>Using Apache + <code>mod_php</code>: I believe that if the script times out on the POST request, processing stops at the point it got to.</p></li>
  <li><p>Using nginx + PHP FPM: nginx will give up and issue a Gateway Timeout according to its configuration, but the PHP process may still carry on regardless, until completion.</p></li>
  </ul>
  
  <p><strong>So for import jobs, it's best to configure your server's timeouts to exceed the time taken for the biggest import you're likely to need to do, and/or break your imports down into smaller chunks that can be handled within those timeouts</strong></p>
  
  <p>Other jobs (dedupe merge, and applying database upgrades are examples) may use CiviCRM's queue system. This has two ways to get processed.</p>
  
  <ol>
  <li><p>A server-side CLI script may chug through the entire queue without any timeouts.</p></li>
  <li><p>A browser-side process with a progress bar works differently and requires the browser to process each step of the queue. If a single step ("task") takes longer than the configured timeouts, you're in trouble. If you close your browser, the next step won't run and the last step may (or may not) fail. If you close your browser and re-open it, you may start running the tasks more than once, in parallel (so don't do this!).</p></li>
  </ol>
  
  <p>See my newly added <a href="https://docs.civicrm.org/dev/en/latest/framework/queues/" rel="nofollow noreferrer">Queue Reference</a> in the developer docs.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-24T12:16:07.000Z
Answers.LastActivityDate: 2019-09-24T12:16:07.000Z
Answers.UserDisplayName: artfulrobot
Body: <p>I'm trying to understand what happens with an import.</p>
  
  <p>A file gets uploaded in a POST operation, then there's GET requests polling(?) of <code>civicrm/ajax/status?id=somelongid</code></p>
  
  <p>Which request does the work? Is is the POST?</p>
  
  <p>Is it that the POST is a really long running script (server timeouts permitting) but that it's triggered in the background? (EDIT: yes, that's true.) Where does the <code>somelongid</code> come from for the GET polling? <del>Or do the GET requests trigger processing the next batch?</del> Does the process continue without the status requests (e.g. if you close the browser).</p>
  
  <p>The reason for wanting to understand is primarily that I have a timeout issue and I don't want to increase the global timeout for all requests, so I change the timeout for certain paths in nginx config. But also I'd just like to know and couldn't find it in the docs.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-23T10:53:49.987Z
Comments.Text: Would be very useful to understand this. I get time out messages from imports but they seem to have processed fine.
Comments.UserDisplayName: Mick Kahn
Tags: <import><documentation>
Title: How are long jobs handled, e.g. Contact import
ThreadText: # How are long jobs handled, e.g. Contact import
  
  (Asked by: username_0 on 2019-07-23)
  
  I'm trying to understand what happens with an import.
  
  A file gets uploaded in a POST operation, then there's GET requests polling(?) of `civicrm/ajax/status?id=somelongid`
  
  Which request does the work? Is is the POST?
  
  Is it that the POST is a really long running script (server timeouts permitting) but that it's triggered in the background? (EDIT: yes, that's true.) Where does the `somelongid` come from for the GET polling? ~~Or do the GET requests trigger processing the next batch?~~ Does the process continue without the status requests (e.g. if you close the browser).
  
  The reason for wanting to understand is primarily that I have a timeout issue and I don't want to increase the global timeout for all requests, so I change the timeout for certain paths in nginx config. But also I'd just like to know and couldn't find it in the docs.
  
  **Tags:** import, documentation
  
  > **Comments:**
  > **username_1**: Would be very useful to understand this. I get time out messages from imports but they seem to have processed fine.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2024-04-22. Score: 2 | Accepted answer
  
  Just noting that this is no longer the case (referring to the [original answer](https://civicrm.stackexchange.com/a/33153/35)) - the jobs are processed in chunks via ajax, or in the background if the server is configured that way
  
  > **Comments:**
  > **username_0**: Just to clarify "this is no longer the case" refer to the long-running [ajax] POST request for imports with multiple [ajax] GET poll requests? i.e. those are now multiple ajax POST requests, with a few (how many?) items in each request?  
  > **username_2**: The number per request is driven by a setting which appears to be on the misc setting page https://github.com/civicrm/civicrm-core/blob/c56953d6e41f4695a531a8db91a8c6f0eccf7e7f/CRM/Import/Parser.php#L623  
  
  ---
  
  ### Answer by username_0 on 2019-09-24. Score: 2
  
  It depends on the job.
  
  Import jobs (Contacts, Contributions...) work by having one long-running POST request, and then your browser polls for updates on progress.
  
  This means your browser/server/interenet connection need to be able to cope with the time that POST request requires. Whether a job completes after apparently timing out depends on how PHP is set up on your site.
  
  For example:
  
  * Using Apache + `mod_php`: I believe that if the script times out on the POST request, processing stops at the point it got to.
  * Using nginx + PHP FPM: nginx will give up and issue a Gateway Timeout according to its configuration, but the PHP process may still carry on regardless, until completion.
  
  **So for import jobs, it's best to configure your server's timeouts to exceed the time taken for the biggest import you're likely to need to do, and/or break your imports down into smaller chunks that can be handled within those timeouts**
  
  Other jobs (dedupe merge, and applying database upgrades are examples) may use CiviCRM's queue system. This has two ways to get processed.
  
  1. A server-side CLI script may chug through the entire queue without any timeouts.
  2. A browser-side process with a progress bar works differently and requires the browser to process each step of the queue. If a single step ("task") takes longer than the configured timeouts, you're in trouble. If you close your browser, the next step won't run and the last step may (or may not) fail. If you close your browser and re-open it, you may start running the tasks more than once, in parallel (so don't do this!).
  
  See my newly added [Queue Reference](https://docs.civicrm.org/dev/en/latest/framework/queues/) in the developer docs.
  
  
  ---
  
  