title: Scrapy writing to a file takes very long and writes only a portion of scraped text
body: <p>I am writing to a csv file using scrapy. I have parsed the website I want to scrape and saved the contents in a item container</p>
  <pre><code>yield(self.post_item)  
  </code></pre>
  <p>The forum that I am scraping is quite large. The code that scrapes and writes seems to be working fine but it is taking very long to write to a csv file. I am writing it using <code> scrapy crawl posts -o file.csv</code>. I forced close the script that was running but the csv file is still writing new contents (very slowly tho).</p>
  <p>What could be causing this and how can I get around this?</p>
  <p>Edit: here is my spider</p>
  <pre><code>class SpiderSpider(scrapy.Spider):
      name = 'posts'
      # allowed_domains = ['dcurbanmom.com']
  
      page_count = 1
      forum_count = 0
  
      #Create an item container to store all this data
      post_item = PostsItem()
      post_subject_answer = &quot;&quot;
      post_subject_first_post_date = &quot;&quot;
      post_subject_views = &quot;&quot;
      post_subject_last_message_date = &quot;&quot;
      total_post = 0
      start_urls = [
          'https://www.dcurbanmom.com/jforum/forums/show/32.page'
          # 'https://www.dcurbanmom.com/jforum/posts/list/643613.page'
      ]
  
      # Grabs the list of threads in the DCPS forum
      def parse(self, response):
          count = 0
          for next_forum in response.xpath('//span[@class=&quot;topictitle&quot;]'):
              if(self.page_count &gt; 10):
                  return
              next_forum_link = next_forum.xpath('.//a/@href')
              next_forum_url = response.urljoin(next_forum_link.extract_first())
   
              answers = next_forum.xpath('.//ancestor::td[1]/following-sibling::td[1]/span/text()')
              self.post_subject_answer = answers.get()
              self.total_post += int(answers.get())
              
              first_post = next_forum.xpath('.//ancestor::td[1]/following-sibling::td[2]/span/text()')
              self.post_subject_first_post_date = first_post.get()
  
              views = next_forum.xpath('.//ancestor::td[1]/following-sibling::td[3]/span/text()')
              self.post_subject_views = views.get()
  
              last_message = next_forum.xpath('.//ancestor::td[1]/following-sibling::td[4]/span/text()')
              self.post_subject_last_message_date = last_message.get()
  
              yield scrapy.Request(url = next_forum_url, callback=self.parse_contents)
              
              count += 1
              self.forum_count += 1
  
          next_page = response.xpath('//div[@class=&quot;pagination&quot;]//a[contains(text(),&quot;►&quot;)]')
          if (len(next_page) &gt; 0):
              next_page = next_page[0].xpath('./@href')
              next_page_url = response.urljoin(next_page.extract_first())
              self.page_count += 1
              yield scrapy.Request(url = next_page_url, callback=self.parse)
                        
          print(&quot;Total discussions traversed: &quot; + str(self.forum_count))
          print(&quot;Total pages traversed: &quot; + str(self.page_count))
          print(&quot;Total posts scraped: &quot;, self.total_post)
          
      #Parses individual discussion thread 
      def parse_contents(self, response):
          all_posts = response.xpath('//table[@class=&quot;forumline&quot;]//tr')
          post_date = &quot;&quot;
          post_text = &quot;&quot;
          post_subject = &quot;&quot;
          post_author = &quot;&quot;
          urls = []
          href_links = {}
          for post in all_posts:
              #Extract the date from post info
              post_date_response = post.xpath('.//div[@class=&quot;postinfo&quot;]//div[@class=&quot;date&quot;]/text()').getall()
              if(len(post_date_response)&gt;0):
                  post_date = post_date_response[-1].strip()   
              
              #Extract the post subject
              post_subject_response = post.xpath('.//div[@class=&quot;postinfo&quot;]//div[@class=&quot;subject&quot;]//a/text()')
              if(len(post_subject_response)&gt;0):
                  post_subject = post_subject_response.extract_first() 
              
              author = post.xpath('.//span[@class=&quot;genmed&quot;]/b/text()').get()
          
              if(author is not None):
                  post_author = author
  
              post_text_response = post.xpath(&quot;.//div[@class='postbody']/br/following-sibling::text()[1] | .//div[@class='postbody']/br/following-sibling::a[1]/text() | .//div[@class='postbody']/text() | .//div[@class='postbody']/a/text()&quot;)
              
              if(len(post_text_response.getall())&gt;0):
                  post_text = &quot;&quot;.join(re.sub('\r','',x) for x in post_text_response.getall()).strip()
                  url_pattern = r'(http(s)?:\/\/.)?(www\.)?[-a-zA-Z0-9@:%._\+~#=]{2,256}\.[a-z]{2,6}\b([-a-zA-Z0-9@:%_\+.~#?&amp;//=]*)'
                  urls = re.finditer(url_pattern, post_text)
                  url_list = [x.group() for x in urls]
              
              href_links_response = post.xpath(&quot;.//div[@class='postbody']//following-sibling::a[1] | .//div[@class='postbody']/a&quot;)
  
              if(len(href_links_response)&gt;0):
                  #Creat a list of texts inside the hyperlinked urls
                  a_text_list = [x.xpath(&quot;./text()&quot;).extract_first() for x in href_links_response]
  
                  #In the URL list (that was copied separatley by the poster), keep only the links that are not in the hyperlink
                  url_in_message_only = [x for x in url_list if x not in a_text_list]
                  
                  href_links = [x.xpath(&quot;./@href&quot;).extract_first() for x in href_links_response]
  
              # Populate the item container
              if(bool(re.search(r'^\s*$', post_text))==False):
                  self.post_item['post_date'] = post_date
                  self.post_item['post_message'] = post_text
                  self.post_item['post_subject'] = post_subject
                  self.post_item['post_author'] = post_author
                  self.post_item['post_hyperlinks'] = href_links
                  self.post_item['post_non_hyperlinks'] = url_in_message_only
                  self.post_item['post_subject_answers'] = self.post_subject_answer
                  self.post_item['post_subject_first_post_date'] = self.post_subject_first_post_date
                  self.post_item['post_subject_views'] = self.post_subject_views
                  self.post_item['post_subject_last_message_date'] = self.post_subject_last_message_date
                  
                  post_date = &quot;&quot;
                  post_text = &quot;&quot;
                  post_subject = &quot;&quot;
                  post_author = &quot;&quot;
                  url_in_message_only = &quot;&quot;
                  href_links = &quot;&quot;
                  yield(self.post_item)  
          # Go to next page
          next_page = response.xpath('//div[@class=&quot;pagination&quot;]//a[contains(text(),&quot;►&quot;)]')
          if len(next_page) &gt; 0:
              next_page = next_page[0].xpath('./@href')
              next_page_url = response.urljoin(next_page.extract_first())
              yield scrapy.Request(url = next_page_url, callback=self.parse_contents)
  </code></pre>
  <p>And here is settings.py</p>
  <pre><code># Scrapy settings for scraper project
  #
  # For simplicity, this file contains only settings considered important or
  # commonly used. You can find more settings consulting the documentation:
  #
  #     https://docs.scrapy.org/en/latest/topics/settings.html
  #     https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
  #     https://docs.scrapy.org/en/latest/topics/spider-middleware.html
  
  BOT_NAME = 'scraper'
  
  SPIDER_MODULES = ['scraper.spiders']
  NEWSPIDER_MODULE = 'scraper.spiders'
  
  # Crawl responsibly by identifying yourself (and your website) on the user-agent
  #USER_AGENT = 'scraper (+http://www.yourdomain.com)'
  
  # Obey robots.txt rules
  ROBOTSTXT_OBEY = False
  LOG_ENABLED = False
  
  # Configure maximum concurrent requests performed by Scrapy (default: 16)
  #CONCURRENT_REQUESTS = 32
  #Wait 250ms between each request
  DOWNLOAD_DELAY = 0.25 
  # Configure a delay for requests for the same website (default: 0)
  # See https://docs.scrapy.org/en/latest/topics/settings.html#download-delay
  # See also autothrottle settings and docs
  #DOWNLOAD_DELAY = 3
  # The download delay setting will honor only one of:
  #CONCURRENT_REQUESTS_PER_DOMAIN = 16
  #CONCURRENT_REQUESTS_PER_IP = 16
  
  # Disable cookies (enabled by default)
  #COOKIES_ENABLED = False
  
  # Disable Telnet Console (enabled by default)
  #TELNETCONSOLE_ENABLED = False
  
  # Override the default request headers:
  #DEFAULT_REQUEST_HEADERS = {
  #   'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
  #   'Accept-Language': 'en',
  #}
  
  # Enable or disable spider middlewares
  # See https://docs.scrapy.org/en/latest/topics/spider-middleware.html
  #SPIDER_MIDDLEWARES = {
  #    'scraper.middlewares.ScraperSpiderMiddleware': 543,
  #}
  
  # Enable or disable downloader middlewares
  # See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
  #DOWNLOADER_MIDDLEWARES = {
  #    'scraper.middlewares.ScraperDownloaderMiddleware': 543,
  #}
  
  # Enable or disable extensions
  # See https://docs.scrapy.org/en/latest/topics/extensions.html
  #EXTENSIONS = {
  #    'scrapy.extensions.telnet.TelnetConsole': None,
  #}
  
  # Configure item pipelines
  # See https://docs.scrapy.org/en/latest/topics/item-pipeline.html
  #ITEM_PIPELINES = {
  #    'scraper.pipelines.ScraperPipeline': 300,
  #}
  
  # Enable and configure the AutoThrottle extension (disabled by default)
  # See https://docs.scrapy.org/en/latest/topics/autothrottle.html
  #AUTOTHROTTLE_ENABLED = True
  # The initial download delay
  #AUTOTHROTTLE_START_DELAY = 5
  # The maximum download delay to be set in case of high latencies
  #AUTOTHROTTLE_MAX_DELAY = 60
  # The average number of requests Scrapy should be sending in parallel to
  # each remote server
  #AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
  # Enable showing throttling stats for every response received:
  #AUTOTHROTTLE_DEBUG = False
  
  # Enable and configure HTTP caching (disabled by default)
  # See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
  #HTTPCACHE_ENABLED = True
  #HTTPCACHE_EXPIRATION_SECS = 0
  #HTTPCACHE_DIR = 'httpcache'
  #HTTPCACHE_IGNORE_HTTP_CODES = []
  #HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'
  
  FEED_EXPORT_ENCODING = 'utf-8'
  
  #Rotating proxies - by scrapy-rotating-proxies
  # ROTATING_PROXY_LIST_PATH = 'http.txt'
  
  DOWNLOADER_MIDDLEWARES = {
      'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,
      'scrapy_user_agents.middlewares.RandomUserAgentMiddleware': 400,
      # ...
      # 'rotating_proxies.middlewares.RotatingProxyMiddleware': 800,
      # 'rotating_proxies.middlewares.BanDetectionMiddleware': 800,
      # ...
  }
  
  RANDOM_UA_PER_PROXY = True
  </code></pre>