title: Multithreading in Python/BeautifulSoup scraping doesn't speed up at all
body: <p>I have a csv file ("SomeSiteValidURLs.csv") which listed all the links I need to scrape. The code is working and will go through the urls in the csv, scrape the information and record/save in another csv file ("Output.csv"). However, since I am planning to do it for a large portion of the site (for >10,000,000 pages), speed is important. For each link, it takes about 1s to crawl and save the info into the csv, which is too slow for the magnitude of the project. So I have incorporated the multithreading module and to my surprise it doesn't speed up at all, it still takes 1s person link. Did I do something wrong? Is there other way to speed up the processing speed?</p>
  
  <p>Without multithreading:</p>
  
  <pre><code>import urllib2
  import csv
  from bs4 import BeautifulSoup
  import threading
  
  def crawlToCSV(FileName):
  
      with open(FileName, "rb") as f:
          for URLrecords in f:
  
              OpenSomeSiteURL = urllib2.urlopen(URLrecords)
              Soup_SomeSite = BeautifulSoup(OpenSomeSiteURL, "lxml")
              OpenSomeSiteURL.close()
  
              tbodyTags = Soup_SomeSite.find("tbody")
              trTags = tbodyTags.find_all("tr", class_="result-item ")
  
              placeHolder = []
  
              for trTag in trTags:
                  tdTags = trTag.find("td", class_="result-value")
                  tdTags_string = tdTags.string
                  placeHolder.append(tdTags_string)
  
              with open("Output.csv", "ab") as f:
                  writeFile = csv.writer(f)
                  writeFile.writerow(placeHolder)
  
  crawltoCSV("SomeSiteValidURLs.csv")
  </code></pre>
  
  <p>With multithreading:</p>
  
  <pre><code>import urllib2
  import csv
  from bs4 import BeautifulSoup
  import threading
  
  def crawlToCSV(FileName):
  
      with open(FileName, "rb") as f:
          for URLrecords in f:
  
              OpenSomeSiteURL = urllib2.urlopen(URLrecords)
              Soup_SomeSite = BeautifulSoup(OpenSomeSiteURL, "lxml")
              OpenSomeSiteURL.close()
  
              tbodyTags = Soup_SomeSite.find("tbody")
              trTags = tbodyTags.find_all("tr", class_="result-item ")
  
              placeHolder = []
  
              for trTag in trTags:
                  tdTags = trTag.find("td", class_="result-value")
                  tdTags_string = tdTags.string
                  placeHolder.append(tdTags_string)
  
              with open("Output.csv", "ab") as f:
                  writeFile = csv.writer(f)
                  writeFile.writerow(placeHolder)
  
  fileName = "SomeSiteValidURLs.csv"
  
  if __name__ == "__main__":
      t = threading.Thread(target=crawlToCSV, args=(fileName, ))
      t.start()
      t.join()
  </code></pre>