Python web crawler multithreading and multiprocessing
Look into grequests, it doesn't do actual muti-threading or multiprocessing, but it scales much better than both.
Briefly idea, My web crawler have 2 main jobs. Collector and Crawler, The collector will collecting all of the url items for each sites and storing non duplicated url. The crawler will grab the urls from storage, extract needed data and store its back. 

2 Machines

Bot machine -> 8 core, Physical Linux OS (No VM on this machine)
Storage machine -> mySql with clustering (VM for clustering), 2 databases (url and data); url database on port 1 and data port 2


Objective: Crawled 100 sites and try to decrease the bottle neck situation

First case: Collector *request(urllib) all sites , collect the url
  items for each sites and * insert if it's non duplicated url to
  Storage machine on port 1. Crawler *get the url from storage port 1 ,
  *request site and extract needed data and *store it's back on port 2

This cause the connection bottle neck for both request web sites and mySql connection

Second case: Instead of inserting across the machine, Collector store
  the url on my own mini database file system.There is no *read a huge
  file(use os command technic) just *write (append) and *remove header
  of the file.

This cause the connection request web sites and I/O (read,write) bottle neck (may be)
Both case also have the CPU bound cause of collecting and crawling 100 sites
As I heard for I/O bound use multithreading, CPU bound use multiprocessing
How about both ? scrappy ? any idea or suggestion ?