title: SESSION or SQL?
body: <p>I have been using a robot I've coded from quite a while now and it works pretty well.
  It goes to a domain, parse the index, save all internal links in a session var (lets call it array1), then refresh itself, move the index to an other session var (array2) and deletes it from array1, parses the next page in array1, check if the new internal links it finds are already in either of the two arrays, if not it saves them in array1, and so on. It basically makes a list of all pages within a domain.</p>
  
  <p>This bot has been crawling some pretty big websites (20k+ pages) and it did fine, it just took some time, but that does not bother me. Now I want it to crawl some even bigger websites (200k+ pages), and I would like to have your opinion on the best way to handle data.</p>
  
  <p>Should I carry on with sessions ? I know sessions use disk space, and even though my raspberry pi is quite stable, is it gonna be able to handle 20Mb+ variables ?
  Should I keep all URL's in a sql tables ? In which case how well can sql handle this amount of data?</p>
  
  <p>Thank you.</p>
  
  <p>(I have searched the web but nothing seems to be really close to what I'm experiencing)</p>