Answers.Body: <p>I'm not sure why you would do two different servers.</p>
  
  <p>Your front end can be cached with Blitz / NGINX which will take a load off the server and admin.</p>
  
  <p>How often are you aggregating the data?</p>
  
  <p>I've just imported 10k records from a source and I would recommend writing your own module to handle the imports as commands. From there you could set up a cron which would run at a scheduled interval. These commands wouldn't run via the web server so you wouldn't be restricted to memory / timeout limits.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-29T05:33:09.037Z
Answers.Text: Thanks Leevi.  Not two servers - one beefy server with multiple Craft installs (if necessary). As for importing data... most feeds run only once or twice a day, with perhaps 3 feeds running as often as every 2-3 minutes. Incidentally, those frequent feeds may have as many as 5K-6K record updates each time.  I'm cool with writing scripts and using cron for imports, but wasn't sure how wise it would be to write/overwrite this kind of bulk data into the Craft DB from outside Craft (especially with such frequent 24/7/365 updates).
Answers.UserDisplayName: JJ.
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-29T04:32:00.220Z
Answers.LastActivityDate: 2020-04-29T04:43:00.453Z
Answers.LastEditDate: 2020-04-29T04:43:00.453Z
Answers.UserDisplayName: Leevi Graham
Body: <p>I'm working on my second Craft project ever. (First was an ultra-simple site just to learn the basics.)</p>
  
  <p>With this new project I need to accomplish 2 things...</p>
  
  <p>(1) - Private - Import/aggregate mass data from external sources based on my specific search parameters (think YouTube, CNN, BBC, Instagram, Reddit, Imgur, Pinterest, etc).  Then slice, dice, curate, categorize, headline, and editorialize that data to create content-rich, topical feeds. </p>
  
  <p>(2) - Public - Present those content-rich feeds as dense Topical Resources for my website visitors.</p>
  
  <p>Craft is a brilliant solution for building a robust (and lightning fast) Public front end.  </p>
  
  <p>But based on how Craft operates underneath the hood... What's the Best Practice for managing the Private raw data behind the scenes?</p>
  
  <p>Is it smarter to have ONE Craft Install with one DB - and separate tables for the imported (but not yet published) data?</p>
  
  <p>OR</p>
  
  <p>Is it wiser to do TWO entirely separate Craft Installs - one for the Private back-end work (importing, compiling, editing, building newly curated feeds) - and another to grab and serve the Public (post-curation) results?</p>
  
Tags: <craft3><database><performance><plugin-feedme><feed>
Title: Import / Aggregation DB - with 50 Million Records - Best Practice?
ThreadText: # Import / Aggregation DB - with 50 Million Records - Best Practice?
  
  (Asked by: username_0 on 2020-04-29)
  
  I'm working on my second Craft project ever. (First was an ultra-simple site just to learn the basics.)
  
  With this new project I need to accomplish 2 things...
  
  (1) - Private - Import/aggregate mass data from external sources based on my specific search parameters (think YouTube, CNN, BBC, Instagram, Reddit, Imgur, Pinterest, etc). Then slice, dice, curate, categorize, headline, and editorialize that data to create content-rich, topical feeds.
  
  (2) - Public - Present those content-rich feeds as dense Topical Resources for my website visitors.
  
  Craft is a brilliant solution for building a robust (and lightning fast) Public front end.
  
  But based on how Craft operates underneath the hood... What's the Best Practice for managing the Private raw data behind the scenes?
  
  Is it smarter to have ONE Craft Install with one DB - and separate tables for the imported (but not yet published) data?
  
  OR
  
  Is it wiser to do TWO entirely separate Craft Installs - one for the Private back-end work (importing, compiling, editing, building newly curated feeds) - and another to grab and serve the Public (post-curation) results?
  
  **Tags:** craft3, database, performance, plugin-feedme, feed
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-04-29. Score: 1
  
  I'm not sure why you would do two different servers.
  
  Your front end can be cached with Blitz / NGINX which will take a load off the server and admin.
  
  How often are you aggregating the data?
  
  I've just imported 10k records from a source and I would recommend writing your own module to handle the imports as commands. From there you could set up a cron which would run at a scheduled interval. These commands wouldn't run via the web server so you wouldn't be restricted to memory / timeout limits.
  
  > **Comments:**
  > **username_0**: Thanks username_1.  Not two servers - one beefy server with multiple Craft installs (if necessary). As for importing data... most feeds run only once or twice a day, with perhaps 3 feeds running as often as every 2-3 minutes. Incidentally, those frequent feeds may have as many as 5K-6K record updates each time.  I'm cool with writing scripts and using cron for imports, but wasn't sure how wise it would be to write/overwrite this kind of bulk data into the Craft DB from outside Craft (especially with such frequent 24/7/365 updates).  
  
  ---
  
  