Answers.Body: <p>Seems the answer that worked for me was to not use Structure section types for large amounts of entries. I ended up recreating the Structure section as a Channel and re-imported the entries. I feel as though it should be mentioned in the documentation that you should not use Structures if you plan on having many children entries, even at just two levels. Perhaps others would assume this is the case but I had no way of knowing.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-09T19:17:20.767Z
Answers.LastActivityDate: 2019-12-09T19:17:20.767Z
Answers.UserDisplayName: Moucky
Answers.Body: <p>If one single update query (on one record) is taking over 2 minutes, I would guess that the bottle neck is the database. You could check if this is an incident or if these update queries are structurally slow.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-20T19:13:45.567Z
Answers.Text: Thanks for replying. What you're saying is my concern, there is some issue with the database having lots of entries(even though I didn't think it was enough to cause trouble). This log is from the third Feed Me step, before any importing even happens, so it's querying the database for the entries before I map the fields. I'm not sure how to check any further what's causing the problem.
Answers.UserDisplayName: Moucky
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-21T20:42:45.867Z
Answers.Text: What happens if you execute the INSERT-query manually on the database through the command line or a database tool? The 'ON DUPLICATE'-part is important as well. 
  
  It checks if the key already exists and performs an update instead of an insert if the key already exists. That does not make it faster but is is probably the fastest solution. You could try adding an index on that table, in case there is no index yet.
Answers.UserDisplayName: Niek
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-21T22:02:45.190Z
Answers.Text: Is it the SELECT query or the INSERT one that's taking a long time? My understanding was that the times shown are "Since Previous" meaning the SELECT query would be the slow one. Anyway I just tried executing the SELECT and INSERT queries manually(using PhpMyAdmin) and the were both very quick. SELECT was 0.0004 seconds and the INSERT was even less.
Answers.UserDisplayName: Moucky
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-20T09:18:59.037Z
Answers.LastActivityDate: 2019-11-20T09:18:59.037Z
Answers.UserDisplayName: Niek
Answers.Body: <p>I would recommend running the Craft queue in a background process <a href="https://nystudio107.com/blog/robust-queue-job-handling-in-craft-cms" rel="nofollow noreferrer">as detailed here</a>. This way it won't bog down your control panel and can just chug away in the background.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-19T18:53:11.777Z
Answers.Text: Thanks for the recommendation! I'll have a look at doing that, do you think that could be what the issue is?
Answers.UserDisplayName: Moucky
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-20T14:50:44.023Z
Answers.Text: It's hard to say but 15,000 entries is a lot to import, so it will likely take a while (15 minutes-ish is my guess) depending on how many fields in needs to compare.
Answers.UserDisplayName: dmatthams
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-20T19:07:02.897Z
Answers.Text: The actual amount of entries I was trying to import there was only 58, the amount current entries on the site is around 15,000. I think there will be a total of about 30,000 when all the importing is complete but I do them per catalogue. Some have less than 100, others may have about 300.
Answers.UserDisplayName: Moucky
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-19T15:06:10.087Z
Answers.LastActivityDate: 2019-11-19T15:06:10.087Z
Answers.UserDisplayName: dmatthams
Body: <p>I am wondering if I have structured my data badly and was hoping someone could advise on an alternative. Essentially I have a structure with two types, a catalogue and the catalogues children. Some catalogues may have up to 400 children.</p>
  
  <p>When I began importing the data using Feed Me I didn't have any problem, but after a while it was becoming nearly impossible to use and I'm only half way through. I checked the logs in the debug bar and found this was holding things up. Perhaps it's of some help?</p>
  
  <p>Any advice here would be really appreciated!</p>
  
  <p><a href="https://i.sstatic.net/ArJlf.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/ArJlf.png" alt="enter image description here"></a></p>
  
  <p>Edit:</p>
  
  <p>For more clarification, this log is from the screen where you map the fields. There may be around 15,000 entries so far, most of which are the children entries. I am just wondering if there is something wrong with using structures this way or is this just because of the large amount of entries in the database? I have read that people are running Craft sites with over 100,000 entries so I didn't think this would be a problem.</p>
  
Tags: <craft3><plugin-feedme>
Title: Craft CMS/Feed Me slow performance
ThreadText: # Craft CMS/Feed Me slow performance
  
  (Asked by: username_0 on 2019-11-18)
  
  I am wondering if I have structured my data badly and was hoping someone could advise on an alternative. Essentially I have a structure with two types, a catalogue and the catalogues children. Some catalogues may have up to 400 children.
  
  When I began importing the data using Feed Me I didn't have any problem, but after a while it was becoming nearly impossible to use and I'm only half way through. I checked the logs in the debug bar and found this was holding things up. Perhaps it's of some help?
  
  Any advice here would be really appreciated!
  
  [![enter image description here](https://i.sstatic.net/ArJlf.png)](https://i.sstatic.net/ArJlf.png)
  
  Edit:
  
  For more clarification, this log is from the screen where you map the fields. There may be around 15,000 entries so far, most of which are the children entries. I am just wondering if there is something wrong with using structures this way or is this just because of the large amount of entries in the database? I have read that people are running Craft sites with over 100,000 entries so I didn't think this would be a problem.
  
  **Tags:** craft3, plugin-feedme
  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_0 on 2019-12-09. Score: 1 | Accepted answer
  
  Seems the answer that worked for me was to not use Structure section types for large amounts of entries. I ended up recreating the Structure section as a Channel and re-imported the entries. I feel as though it should be mentioned in the documentation that you should not use Structures if you plan on having many children entries, even at just two levels. Perhaps others would assume this is the case but I had no way of knowing.
  
  
  ---
  
  ### Answer by username_1 on 2019-11-20. Score: 1
  
  If one single update query (on one record) is taking over 2 minutes, I would guess that the bottle neck is the database. You could check if this is an incident or if these update queries are structurally slow.
  
  > **Comments:**
  > **username_0**: Thanks for replying. What you're saying is my concern, there is some issue with the database having lots of entries(even though I didn't think it was enough to cause trouble). This log is from the third Feed Me step, before any importing even happens, so it's querying the database for the entries before I map the fields. I'm not sure how to check any further what's causing the problem.  
  > **username_1**: What happens if you execute the INSERT-query manually on the database through the command line or a database tool? The 'ON DUPLICATE'-part is important as well.   It checks if the key already exists and performs an update instead of an insert if the key already exists. That does not make it faster but is is probably the fastest solution. You could try adding an index on that table, in case there is no index yet.  
  > **username_0**: Is it the SELECT query or the INSERT one that's taking a long time? My understanding was that the times shown are "Since Previous" meaning the SELECT query would be the slow one. Anyway I just tried executing the SELECT and INSERT queries manually(using PhpMyAdmin) and the were both very quick. SELECT was 0.0004 seconds and the INSERT was even less.  
  
  ---
  
  ### Answer by username_2 on 2019-11-19. Score: 1
  
  I would recommend running the Craft queue in a background process [as detailed here](https://nystudio107.com/blog/robust-queue-job-handling-in-craft-cms). This way it won't bog down your control panel and can just chug away in the background.
  
  > **Comments:**
  > **username_0**: Thanks for the recommendation! I'll have a look at doing that, do you think that could be what the issue is?  
  > **username_2**: It's hard to say but 15,000 entries is a lot to import, so it will likely take a while (15 minutes-ish is my guess) depending on how many fields in needs to compare.  
  > **username_0**: The actual amount of entries I was trying to import there was only 58, the amount current entries on the site is around 15,000. I think there will be a total of about 30,000 when all the importing is complete but I do them per catalogue. Some have less than 100, others may have about 300.  
  
  ---
  
  