Answers.Body: <p>Turns out the answer in my case is to use a <strong>JSON</strong> data source.</p>
  
  <p>Importing users as <code>.json</code> instead of <code>.csv</code> appears to be about 10 times faster: </p>
  
  <ul>
  <li>With <code>.csv</code> importing 480 users took 2.5 hours</li>
  <li>With <code>.json</code> importing 455 users took 15 minutes</li>
  </ul>
  
  <p>My JSON looked like this: </p>
  
  <pre><code>{"user": [
      {
          "username":"jane.doe@example.com",
          "email":"jane.doe@example.com",
          "firstName":"Jane",
          "lastName":"Doe",
          "password":"User pass phrase",
          "groupName":"User group name"
      },
      {
          "username":"joe.bloggs@example.com",
          "email":"joe.bloggs@example.com",
          "firstName":"Joe",
          "lastName":"Bloggs",
          "password":"User pass phrase",
          "groupName":"User group name"
      }
  ]}
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-08T12:13:59.290Z
Answers.LastActivityDate: 2020-05-08T12:13:59.290Z
Answers.UserDisplayName: Jonathan Schofield
Body: <p>I'm aware of <a href="https://craftcms.stackexchange.com/questions/33048/craft-cms-feed-me-slow-performance">this thread on Feed Me slow performance</a>, but I'm raising it as a separate thread in case there is anything particular about my circumstance that is pertinent.</p>
  
  <p>I'm running Craft 3.4.18 and trying to import a large number of <strong>users</strong> (about 1,500) into a specific user group using Feed Me: </p>
  
  <ol>
  <li>I'm using <code>.csv</code> sources on the local file system above web root</li>
  <li>I'm importing:
  
  <ul>
  <li>Username (as email address string)</li>
  <li>Email</li>
  <li>First name</li>
  <li>Last name</li>
  <li>Shared password</li>
  <li>User group string</li>
  </ul></li>
  <li>I'm batching imports in 500s but because things are so slow will now try smaller batches</li>
  <li>I have read and attended to the <a href="https://docs.craftcms.com/feed-me/v4/troubleshooting.html#performance" rel="nofollow noreferrer">performance</a> aspects of the Feed Me troubleshooting docs</li>
  <li>There is plenty of memory and execution time allocation to Craft and the instance is on a dedicated Linux server with 8Gb of RAM</li>
  </ol>
  
  <p>Imports are working without error, but paaaaaainfully slowly -- we're talking hours -- to the point of the import <em>process</em> being effectively useless*.</p>
  
  <p>I have looked into <a href="https://nystudio107.com/blog/robust-queue-job-handling-in-craft-cms#solution-1-async-queue-plugin" rel="nofollow noreferrer">Andrew Welch's #1 solution</a> to running the queue as a background process, but have bumped into <a href="https://github.com/ostark/craft-async-queue/issues/32" rel="nofollow noreferrer">installation issues</a> with the <strong>Async Queue</strong> plugin.</p>
  
  <p>Any suggestions much appreciated.</p>
  
  <hr>
  
  <p>I'd be happy in principle to do <strong>SQL inserts</strong> directly into the <code>craft_users</code> and <code>craft_usergroups_users</code> tables but I infer that's not an option since each hash of the shared password is unique and there is a unique UID to generate for each row too.</p>
  
  <hr>
  
  <p><strong>Update on #3 above:</strong></p>
  
  <p>I'm currently running a batch of 100 instead of 500 and that makes no difference to speed whatsoever.</p>
  
  <p>Next step is perhaps to try a JSON feed instead of CSV?</p>
  
  <hr>
  
  <p>*'Effectively useless' in the sense that although I have time to run extended imports right now, if it's going to generally take this long it's a non-starter for time-pressured imports.</p>
  
  <hr>
  
  <p><strong>Second update on #3 above:</strong></p>
  
  <p>An import batch of 100 I started over 2 hours ago has crept along incrementally, with me periodically reviewing its status during that time and seeing barely any change, and then…</p>
  
  <p>I have just revisited it and it suddenly went from 64% complete and unchanging to 100%. Now onto the next batch to see what happens.</p>
  
Tags: <plugin-feedme><queue>
Title: Painfully slow Feed Me import of User accounts
ThreadText: # Painfully slow Feed Me import of User accounts
  
  (Asked by: username_0 on 2020-05-07)
  
  I'm aware of [this thread on Feed Me slow performance](https://craftcms.stackexchange.com/questions/33048/craft-cms-feed-me-slow-performance), but I'm raising it as a separate thread in case there is anything particular about my circumstance that is pertinent.
  
  I'm running Craft 3.4.18 and trying to import a large number of **users** (about 1,500) into a specific user group using Feed Me:
  
  1. I'm using `.csv` sources on the local file system above web root
  2. I'm importing:
     * Username (as email address string)
     * Email
     * First name
     * Last name
     * Shared password
     * User group string
  3. I'm batching imports in 500s but because things are so slow will now try smaller batches
  4. I have read and attended to the [performance](https://docs.craftcms.com/feed-me/v4/troubleshooting.html#performance) aspects of the Feed Me troubleshooting docs
  5. There is plenty of memory and execution time allocation to Craft and the instance is on a dedicated Linux server with 8Gb of RAM
  
  Imports are working without error, but paaaaaainfully slowly -- we're talking hours -- to the point of the import *process* being effectively useless\*.
  
  I have looked into [Andrew Welch's #1 solution](https://nystudio107.com/blog/robust-queue-job-handling-in-craft-cms#solution-1-async-queue-plugin) to running the queue as a background process, but have bumped into [installation issues](https://github.com/ostark/craft-async-queue/issues/32) with the **Async Queue** plugin.
  
  Any suggestions much appreciated.
  
  ---
  
  I'd be happy in principle to do **SQL inserts** directly into the `craft_users` and `craft_usergroups_users` tables but I infer that's not an option since each hash of the shared password is unique and there is a unique UID to generate for each row too.
  
  ---
  
  **Update on #3 above:**
  
  I'm currently running a batch of 100 instead of 500 and that makes no difference to speed whatsoever.
  
  Next step is perhaps to try a JSON feed instead of CSV?
  
  ---
  
  \*'Effectively useless' in the sense that although I have time to run extended imports right now, if it's going to generally take this long it's a non-starter for time-pressured imports.
  
  ---
  
  **Second update on #3 above:**
  
  An import batch of 100 I started over 2 hours ago has crept along incrementally, with me periodically reviewing its status during that time and seeing barely any change, and then…
  
  I have just revisited it and it suddenly went from 64% complete and unchanging to 100%. Now onto the next batch to see what happens.
  
  **Tags:** plugin-feedme, queue
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2020-05-08. Score: 1
  
  Turns out the answer in my case is to use a **JSON** data source.
  
  Importing users as `.json` instead of `.csv` appears to be about 10 times faster:
  
  * With `.csv` importing 480 users took 2.5 hours
  * With `.json` importing 455 users took 15 minutes
  
  My JSON looked like this:
  
  ```
  {"user": [
      {
          "username":"jane.doe@example.com",
          "email":"jane.doe@example.com",
          "firstName":"Jane",
          "lastName":"Doe",
          "password":"User pass phrase",
          "groupName":"User group name"
      },
      {
          "username":"joe.bloggs@example.com",
          "email":"joe.bloggs@example.com",
          "firstName":"Joe",
          "lastName":"Bloggs",
          "password":"User pass phrase",
          "groupName":"User group name"
      }
  ]}
  
  ```
  
  
  ---
  
  