Answers.Body: <p>That's almost certainly a timeout issue - you can increase your timeout in <code>php.ini</code>, but I wouldn't use the UI at all for a million contacts.  Consider using a command-line import, where there should be no timeouts.</p>
  
  <p>There are several approaches you can take.</p>
  
  <ul>
  <li><p>Use the <a href="https://civicrm.stackexchange.com/q/4723/12">command-line API CSV import</a>.  This is pretty solid but you need to import each file multiple times - once for contact data, once for postal addresses, once for phone, once for email.  If you script it, it's not bad.  I often use <a href="https://community.hitachivantara.com/s/article/data-integration-kettle" rel="nofollow noreferrer">Hitachi Kettle</a> to do this.</p></li>
  <li><p>Use the Data Processor plugin and the REST API.  There's a <a href="https://civicrm.org/blog/jaapjansma/insert-data-from-a-csv-file-into-civicrm" rel="nofollow noreferrer">recent blog post</a> detailing this approach (using Hitachi Kettle actually).  This lets you import each file once.  However, the REST API is going to do something like 10 records/second, command-line import is far faster but requires command-line access to the server.</p></li>
  <li><p>Write your own PHP script so you don't need to break the data up.</p></li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-25T02:40:02.130Z
Answers.Text: There's an option that I believe has a lower technical barrier to entry, but I can't find it.  I believe there's an extension that will let you save an import you create in the UI, and run it on the command line.  I can't find it but I'll ask the person I thought wrote it.
Answers.UserDisplayName: Jon G - Megaphone Tech
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-19T21:51:20.073Z
Answers.Text: jon - any thoughts on that last comment by you?
Answers.UserDisplayName: petednz - fuzion
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-19T22:25:27.547Z
Answers.Text: Unfortunately not.  Ideally we would modify Mathieu's `advimport` extension to use standard Civi import classes as one of its options, because it handles queuing.  But that's several hours of work, and there's more work because `advimport` has its own limitation on size (related to how it stores the data).  However, `advimport`'s problems can all be solved in the extension itself; fixing standard import requires access to the server's configuration for Apache and PHP.
Answers.UserDisplayName: Jon G - Megaphone Tech
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-25T02:39:17.880Z
Answers.LastActivityDate: 2019-09-25T02:39:17.880Z
Answers.UserDisplayName: Jon G - Megaphone Tech
Body: <p>I’m hoping to use CiviCRM to manage campaign contacts for voters across many counties and need to upload BOE data to do it. It’s about 210 MB in total, but I have it broken up into CSVs under 30 MB each. Unfortunately, I keep running into a 503 error when I try to use the import wizard with my smallest file (around 3 MB).</p>
  
  <p>I’m using CiviCRM 5.17.4 on Joomla 3.9.12. I’ve increased Joomla’s session timeout and added a bit of code to civicrm_settings.php, following a tip on this guide: <a href="https://wiki.civicrm.org/confluence/display/CRMDOC/Importing+Data" rel="nofollow noreferrer">https://wiki.civicrm.org/confluence/display/CRMDOC/Importing+Data</a></p>
  
  <p>I know uploads of this size are a tall order; is there anything that I’m missing or anything else that can be done? There are about a million registered voters here and I don’t think uploading them 4,000 at a time is feasible. </p>
  
  <p>EDIT: If this goes unanswered and someone else runs into the same problem, you can whip up a BAT file to cut up CSVs for you.</p>
  
  <p><a href="https://www.makeuseof.com/tag/how-to-split-a-huge-csv-excel-workbook-into-seperate-files/" rel="nofollow noreferrer">https://www.makeuseof.com/tag/how-to-split-a-huge-csv-excel-workbook-into-seperate-files/</a></p>
  
Tags: <joomla><import><data-import>
Title: 503 Error with large CSV import
ThreadText: # 503 Error with large CSV import
  
  (Asked by: username_0 on 2019-09-24)
  
  I’m hoping to use CiviCRM to manage campaign contacts for voters across many counties and need to upload BOE data to do it. It’s about 210 MB in total, but I have it broken up into CSVs under 30 MB each. Unfortunately, I keep running into a 503 error when I try to use the import wizard with my smallest file (around 3 MB).
  
  I’m using CiviCRM 5.17.4 on Joomla 3.9.12. I’ve increased Joomla’s session timeout and added a bit of code to civicrm\_settings.php, following a tip on this guide: <https://wiki.civicrm.org/confluence/display/CRMDOC/Importing+Data>
  
  I know uploads of this size are a tall order; is there anything that I’m missing or anything else that can be done? There are about a million registered voters here and I don’t think uploading them 4,000 at a time is feasible.
  
  EDIT: If this goes unanswered and someone else runs into the same problem, you can whip up a BAT file to cut up CSVs for you.
  
  <https://www.makeuseof.com/tag/how-to-split-a-huge-csv-excel-workbook-into-seperate-files/>
  
  **Tags:** joomla, import, data-import
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-09-25. Score: 2
  
  That's almost certainly a timeout issue - you can increase your timeout in `php.ini`, but I wouldn't use the UI at all for a million contacts. Consider using a command-line import, where there should be no timeouts.
  
  There are several approaches you can take.
  
  * Use the [command-line API CSV import](https://civicrm.stackexchange.com/q/4723/12). This is pretty solid but you need to import each file multiple times - once for contact data, once for postal addresses, once for phone, once for email. If you script it, it's not bad. I often use [Hitachi Kettle](https://community.hitachivantara.com/s/article/data-integration-kettle) to do this.
  * Use the Data Processor plugin and the REST API. There's a [recent blog post](https://civicrm.org/blog/jaapjansma/insert-data-from-a-csv-file-into-civicrm) detailing this approach (using Hitachi Kettle actually). This lets you import each file once. However, the REST API is going to do something like 10 records/second, command-line import is far faster but requires command-line access to the server.
  * Write your own PHP script so you don't need to break the data up.
  
  > **Comments:**
  > **username_1**: There's an option that I believe has a lower technical barrier to entry, but I can't find it.  I believe there's an extension that will let you save an import you create in the UI, and run it on the command line.  I can't find it but I'll ask the person I thought wrote it.  
  > **username_2**: jon - any thoughts on that last comment by you?  
  > **username_1**: Unfortunately not.  Ideally we would modify Mathieu's `advimport` extension to use standard Civi import classes as one of its options, because it handles queuing.  But that's several hours of work, and there's more work because `advimport` has its own limitation on size (related to how it stores the data).  However, `advimport`'s problems can all be solved in the extension itself; fixing standard import requires access to the server's configuration for Apache and PHP.  
  
  ---
  
  