<h3 dir="auto">Description</h3>
<p dir="auto">I'm running a Scrapy crawler from PyCharm's Python Console:</p>
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/5827775/91909783-378c4800-ec84-11ea-83fc-849e6acb383b.png"><img src="https://user-images.githubusercontent.com/5827775/91909783-378c4800-ec84-11ea-83fc-849e6acb383b.png" alt="image" style="max-width: 100%;"></a></p>
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/5827775/91909802-43780a00-ec84-11ea-9155-cc636e1180e6.png"><img src="https://user-images.githubusercontent.com/5827775/91909802-43780a00-ec84-11ea-9155-cc636e1180e6.png" alt="image" style="max-width: 100%;"></a></p>
<p dir="auto">In my code (below), I export the scraped content to CSV files through <code class="notranslate">CsvItemExporter</code>. When run from PyCharm's Python Console (using both configurations above), the scraper runs fine, but doesn't write to the CSV files; they are 0 bytes long after the crawler runs. However, when I run scrapy from the command line (<code class="notranslate">scrapy crawl disasters</code>) or from PyCharm's debugger, suddenly it writes to the CSV files as intended.</p>
<p dir="auto">I tried this using <code class="notranslate">XmlItemExporter</code> as well. Same results.</p>
<p dir="auto">How can I run this using PyCharm's Python Console?</p>
<p dir="auto">Here is my code:</p>
<p dir="auto"><code class="notranslate">spiders/disasters.py</code></p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="from datetime import datetime as dt
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule
from asnscraper.items import DisasterRaw, AirportRaw
import re


RE_LISTING = r'.*/dblist\.php\?(Year)=(1920)$'
RE_RECORD = r'.*/record\.php\?id=((\d{8})-(\d+))$'
RE_AIRPORT = r'.*/airport\.php\?id=(\w+)$'


class DisasterSpider(CrawlSpider):
    name = 'disasters'
    allowed_domains = ['aviation-safety.net']
    start_urls = ['https://aviation-safety.net/database/']
    rules = [
        Rule(LinkExtractor(allow=RE_LISTING), callback='parse_list', follow=True),
        Rule(LinkExtractor(allow=RE_RECORD), callback='parse_record', follow=True),
        Rule(LinkExtractor(allow=RE_AIRPORT), callback='parse_airport', follow=False)
    ]

    @staticmethod
    def parse_list(response):
        m = re.match(RE_LISTING, response.url)
        print(f&quot;Database listing WHERE '{m[1]}' = '{m[2]}'&quot;)

    @staticmethod
    def parse_airport(response):
        id = re.match(RE_AIRPORT, response.url)
        airportraw = AirportRaw()
        airportfields = response.xpath(&quot;//a[@name='general']/div[@class='infobox']//table[1]//tr&quot;)
        airportname = response.xpath(&quot;//a[@name='general']/div[@class='infobox']/span/text()&quot;)
        airportraw['id'] = id[1]
        airportraw['name'] = airportname
        airportraw['allfields'] = airportfields
        print(f&quot;Airport listing WHERE 'id' = '{airportraw['id']}'&quot;)
        return airportraw

    @staticmethod
    def parse_record(response):
        id = re.match(RE_RECORD, response.url)
        disasterraw = DisasterRaw()
        disasterfields = response.xpath(&quot;//div[@class='innertube']//table[1]//tr&quot;)
        disasterraw['id'] = id[1]
        disasterraw['date'] = dt.strptime(id[2], '%Y%m%d')
        disasterraw['allfields'] = disasterfields
        print(f&quot;Acident listing WHERE 'id' = '{disasterraw['id']}'&quot;)
        return disasterraw"><pre class="notranslate"><span class="pl-k">from</span> <span class="pl-s1">datetime</span> <span class="pl-k">import</span> <span class="pl-s1">datetime</span> <span class="pl-k">as</span> <span class="pl-s1">dt</span>
<span class="pl-k">from</span> <span class="pl-s1">scrapy</span>.<span class="pl-s1">linkextractors</span> <span class="pl-k">import</span> <span class="pl-v">LinkExtractor</span>
<span class="pl-k">from</span> <span class="pl-s1">scrapy</span>.<span class="pl-s1">spiders</span> <span class="pl-k">import</span> <span class="pl-v">CrawlSpider</span>, <span class="pl-v">Rule</span>
<span class="pl-k">from</span> <span class="pl-s1">asnscraper</span>.<span class="pl-s1">items</span> <span class="pl-k">import</span> <span class="pl-v">DisasterRaw</span>, <span class="pl-v">AirportRaw</span>
<span class="pl-k">import</span> <span class="pl-s1">re</span>


<span class="pl-v">RE_LISTING</span> <span class="pl-c1">=</span> <span class="pl-s">r'.*/dblist\.php\?(Year)=(1920)$'</span>
<span class="pl-v">RE_RECORD</span> <span class="pl-c1">=</span> <span class="pl-s">r'.*/record\.php\?id=((\d{8})-(\d+))$'</span>
<span class="pl-v">RE_AIRPORT</span> <span class="pl-c1">=</span> <span class="pl-s">r'.*/airport\.php\?id=(\w+)$'</span>


<span class="pl-k">class</span> <span class="pl-v">DisasterSpider</span>(<span class="pl-v">CrawlSpider</span>):
    <span class="pl-s1">name</span> <span class="pl-c1">=</span> <span class="pl-s">'disasters'</span>
    <span class="pl-s1">allowed_domains</span> <span class="pl-c1">=</span> [<span class="pl-s">'aviation-safety.net'</span>]
    <span class="pl-s1">start_urls</span> <span class="pl-c1">=</span> [<span class="pl-s">'https://aviation-safety.net/database/'</span>]
    <span class="pl-s1">rules</span> <span class="pl-c1">=</span> [
        <span class="pl-v">Rule</span>(<span class="pl-v">LinkExtractor</span>(<span class="pl-s1">allow</span><span class="pl-c1">=</span><span class="pl-v">RE_LISTING</span>), <span class="pl-s1">callback</span><span class="pl-c1">=</span><span class="pl-s">'parse_list'</span>, <span class="pl-s1">follow</span><span class="pl-c1">=</span><span class="pl-c1">True</span>),
        <span class="pl-v">Rule</span>(<span class="pl-v">LinkExtractor</span>(<span class="pl-s1">allow</span><span class="pl-c1">=</span><span class="pl-v">RE_RECORD</span>), <span class="pl-s1">callback</span><span class="pl-c1">=</span><span class="pl-s">'parse_record'</span>, <span class="pl-s1">follow</span><span class="pl-c1">=</span><span class="pl-c1">True</span>),
        <span class="pl-v">Rule</span>(<span class="pl-v">LinkExtractor</span>(<span class="pl-s1">allow</span><span class="pl-c1">=</span><span class="pl-v">RE_AIRPORT</span>), <span class="pl-s1">callback</span><span class="pl-c1">=</span><span class="pl-s">'parse_airport'</span>, <span class="pl-s1">follow</span><span class="pl-c1">=</span><span class="pl-c1">False</span>)
    ]

    <span class="pl-en">@<span class="pl-s1">staticmethod</span></span>
    <span class="pl-k">def</span> <span class="pl-en">parse_list</span>(<span class="pl-s1">response</span>):
        <span class="pl-s1">m</span> <span class="pl-c1">=</span> <span class="pl-s1">re</span>.<span class="pl-en">match</span>(<span class="pl-v">RE_LISTING</span>, <span class="pl-s1">response</span>.<span class="pl-s1">url</span>)
        <span class="pl-en">print</span>(<span class="pl-s">f"Database listing WHERE '<span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">m</span>[<span class="pl-c1">1</span>]<span class="pl-kos">}</span></span>' = '<span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">m</span>[<span class="pl-c1">2</span>]<span class="pl-kos">}</span></span>'"</span>)

    <span class="pl-en">@<span class="pl-s1">staticmethod</span></span>
    <span class="pl-k">def</span> <span class="pl-en">parse_airport</span>(<span class="pl-s1">response</span>):
        <span class="pl-s1">id</span> <span class="pl-c1">=</span> <span class="pl-s1">re</span>.<span class="pl-en">match</span>(<span class="pl-v">RE_AIRPORT</span>, <span class="pl-s1">response</span>.<span class="pl-s1">url</span>)
        <span class="pl-s1">airportraw</span> <span class="pl-c1">=</span> <span class="pl-v">AirportRaw</span>()
        <span class="pl-s1">airportfields</span> <span class="pl-c1">=</span> <span class="pl-s1">response</span>.<span class="pl-en">xpath</span>(<span class="pl-s">"//a[@name='general']/div[@class='infobox']//table[1]//tr"</span>)
        <span class="pl-s1">airportname</span> <span class="pl-c1">=</span> <span class="pl-s1">response</span>.<span class="pl-en">xpath</span>(<span class="pl-s">"//a[@name='general']/div[@class='infobox']/span/text()"</span>)
        <span class="pl-s1">airportraw</span>[<span class="pl-s">'id'</span>] <span class="pl-c1">=</span> <span class="pl-s1">id</span>[<span class="pl-c1">1</span>]
        <span class="pl-s1">airportraw</span>[<span class="pl-s">'name'</span>] <span class="pl-c1">=</span> <span class="pl-s1">airportname</span>
        <span class="pl-s1">airportraw</span>[<span class="pl-s">'allfields'</span>] <span class="pl-c1">=</span> <span class="pl-s1">airportfields</span>
        <span class="pl-en">print</span>(<span class="pl-s">f"Airport listing WHERE 'id' = '<span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">airportraw</span>[<span class="pl-s">'id'</span>]<span class="pl-kos">}</span></span>'"</span>)
        <span class="pl-k">return</span> <span class="pl-s1">airportraw</span>

    <span class="pl-en">@<span class="pl-s1">staticmethod</span></span>
    <span class="pl-k">def</span> <span class="pl-en">parse_record</span>(<span class="pl-s1">response</span>):
        <span class="pl-s1">id</span> <span class="pl-c1">=</span> <span class="pl-s1">re</span>.<span class="pl-en">match</span>(<span class="pl-v">RE_RECORD</span>, <span class="pl-s1">response</span>.<span class="pl-s1">url</span>)
        <span class="pl-s1">disasterraw</span> <span class="pl-c1">=</span> <span class="pl-v">DisasterRaw</span>()
        <span class="pl-s1">disasterfields</span> <span class="pl-c1">=</span> <span class="pl-s1">response</span>.<span class="pl-en">xpath</span>(<span class="pl-s">"//div[@class='innertube']//table[1]//tr"</span>)
        <span class="pl-s1">disasterraw</span>[<span class="pl-s">'id'</span>] <span class="pl-c1">=</span> <span class="pl-s1">id</span>[<span class="pl-c1">1</span>]
        <span class="pl-s1">disasterraw</span>[<span class="pl-s">'date'</span>] <span class="pl-c1">=</span> <span class="pl-s1">dt</span>.<span class="pl-en">strptime</span>(<span class="pl-s1">id</span>[<span class="pl-c1">2</span>], <span class="pl-s">'%Y%m%d'</span>)
        <span class="pl-s1">disasterraw</span>[<span class="pl-s">'allfields'</span>] <span class="pl-c1">=</span> <span class="pl-s1">disasterfields</span>
        <span class="pl-en">print</span>(<span class="pl-s">f"Acident listing WHERE 'id' = '<span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">disasterraw</span>[<span class="pl-s">'id'</span>]<span class="pl-kos">}</span></span>'"</span>)
        <span class="pl-k">return</span> <span class="pl-s1">disasterraw</span></pre></div>
<p dir="auto"><code class="notranslate">items.py</code></p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import scrapy


class DisasterRaw(scrapy.Item):
    id = scrapy.Field()
    allfields = scrapy.Field()
    date = scrapy.Field()
    ap_from = scrapy.Field()
    ap_to = scrapy.Field()


class Disaster(scrapy.Item):
    id = scrapy.Field()
    datetime = scrapy.Field()
    type = scrapy.Field()
    operator = scrapy.Field()
    aircraft = scrapy.Field()
    crew_deaths = scrapy.Field()
    crew_total = scrapy.Field()
    passenger_deaths = scrapy.Field()
    passenger_total = scrapy.Field()
    ground_deaths = scrapy.Field()
    souls_deaths = scrapy.Field()
    souls_total = scrapy.Field()
    damage = scrapy.Field()
    fate = scrapy.Field()
    location = scrapy.Field()
    flightphase = scrapy.Field()
    nature = scrapy.Field()
    ap_from = scrapy.Field()
    ap_to = scrapy.Field()
    flightnumber = scrapy.Field()
    narrative = scrapy.Field()


class AirportRaw(scrapy.Item):
    id = scrapy.Field()
    name = scrapy.Field()
    allfields = scrapy.Field()


class Airport(scrapy.Item):
    id = scrapy.Field()
    name = scrapy.Field()
    country = scrapy.Field()
    iata = scrapy.Field()
    icao = scrapy.Field()
    elevation = scrapy.Field()
    elevation_unit = scrapy.Field()
    dateopened = scrapy.Field()"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">scrapy</span>


<span class="pl-k">class</span> <span class="pl-v">DisasterRaw</span>(<span class="pl-s1">scrapy</span>.<span class="pl-v">Item</span>):
    <span class="pl-s1">id</span> <span class="pl-c1">=</span> <span class="pl-s1">scrapy</span>.<span class="pl-v">Field</span>()
    <span class="pl-s1">allfields</span> <span class="pl-c1">=</span> <span class="pl-s1">scrapy</span>.<span class="pl-v">Field</span>()
    <span class="pl-s1">date</span> <span class="pl-c1">=</span> <span class="pl-s1">scrapy</span>.<span class="pl-v">Field</span>()
    <span class="pl-s1">ap_from</span> <span class="pl-c1">=</span> <span class="pl-s1">scrapy</span>.<span class="pl-v">Field</span>()
    <span class="pl-s1">ap_to</span> <span class="pl-c1">=</span> <span class="pl-s1">scrapy</span>.<span class="pl-v">Field</span>()


<span class="pl-k">class</span> <span class="pl-v">Disaster</span>(<span class="pl-s1">scrapy</span>.<span class="pl-v">Item</span>):
    <span class="pl-s1">id</span> <span class="pl-c1">=</span> <span class="pl-s1">scrapy</span>.<span class="pl-v">Field</span>()
    <span class="pl-s1">datetime</span> <span class="pl-c1">=</span> <span class="pl-s1">scrapy</span>.<span class="pl-v">Field</span>()
    <span class="pl-s1">type</span> <span class="pl-c1">=</span> <span class="pl-s1">scrapy</span>.<span class="pl-v">Field</span>()
    <span class="pl-s1">operator</span> <span class="pl-c1">=</span> <span class="pl-s1">scrapy</span>.<span class="pl-v">Field</span>()
    <span class="pl-s1">aircraft</span> <span class="pl-c1">=</span> <span class="pl-s1">scrapy</span>.<span class="pl-v">Field</span>()
    <span class="pl-s1">crew_deaths</span> <span class="pl-c1">=</span> <span class="pl-s1">scrapy</span>.<span class="pl-v">Field</span>()
    <span class="pl-s1">crew_total</span> <span class="pl-c1">=</span> <span class="pl-s1">scrapy</span>.<span class="pl-v">Field</span>()
    <span class="pl-s1">passenger_deaths</span> <span class="pl-c1">=</span> <span class="pl-s1">scrapy</span>.<span class="pl-v">Field</span>()
    <span class="pl-s1">passenger_total</span> <span class="pl-c1">=</span> <span class="pl-s1">scrapy</span>.<span class="pl-v">Field</span>()
    <span class="pl-s1">ground_deaths</span> <span class="pl-c1">=</span> <span class="pl-s1">scrapy</span>.<span class="pl-v">Field</span>()
    <span class="pl-s1">souls_deaths</span> <span class="pl-c1">=</span> <span class="pl-s1">scrapy</span>.<span class="pl-v">Field</span>()
    <span class="pl-s1">souls_total</span> <span class="pl-c1">=</span> <span class="pl-s1">scrapy</span>.<span class="pl-v">Field</span>()
    <span class="pl-s1">damage</span> <span class="pl-c1">=</span> <span class="pl-s1">scrapy</span>.<span class="pl-v">Field</span>()
    <span class="pl-s1">fate</span> <span class="pl-c1">=</span> <span class="pl-s1">scrapy</span>.<span class="pl-v">Field</span>()
    <span class="pl-s1">location</span> <span class="pl-c1">=</span> <span class="pl-s1">scrapy</span>.<span class="pl-v">Field</span>()
    <span class="pl-s1">flightphase</span> <span class="pl-c1">=</span> <span class="pl-s1">scrapy</span>.<span class="pl-v">Field</span>()
    <span class="pl-s1">nature</span> <span class="pl-c1">=</span> <span class="pl-s1">scrapy</span>.<span class="pl-v">Field</span>()
    <span class="pl-s1">ap_from</span> <span class="pl-c1">=</span> <span class="pl-s1">scrapy</span>.<span class="pl-v">Field</span>()
    <span class="pl-s1">ap_to</span> <span class="pl-c1">=</span> <span class="pl-s1">scrapy</span>.<span class="pl-v">Field</span>()
    <span class="pl-s1">flightnumber</span> <span class="pl-c1">=</span> <span class="pl-s1">scrapy</span>.<span class="pl-v">Field</span>()
    <span class="pl-s1">narrative</span> <span class="pl-c1">=</span> <span class="pl-s1">scrapy</span>.<span class="pl-v">Field</span>()


<span class="pl-k">class</span> <span class="pl-v">AirportRaw</span>(<span class="pl-s1">scrapy</span>.<span class="pl-v">Item</span>):
    <span class="pl-s1">id</span> <span class="pl-c1">=</span> <span class="pl-s1">scrapy</span>.<span class="pl-v">Field</span>()
    <span class="pl-s1">name</span> <span class="pl-c1">=</span> <span class="pl-s1">scrapy</span>.<span class="pl-v">Field</span>()
    <span class="pl-s1">allfields</span> <span class="pl-c1">=</span> <span class="pl-s1">scrapy</span>.<span class="pl-v">Field</span>()


<span class="pl-k">class</span> <span class="pl-v">Airport</span>(<span class="pl-s1">scrapy</span>.<span class="pl-v">Item</span>):
    <span class="pl-s1">id</span> <span class="pl-c1">=</span> <span class="pl-s1">scrapy</span>.<span class="pl-v">Field</span>()
    <span class="pl-s1">name</span> <span class="pl-c1">=</span> <span class="pl-s1">scrapy</span>.<span class="pl-v">Field</span>()
    <span class="pl-s1">country</span> <span class="pl-c1">=</span> <span class="pl-s1">scrapy</span>.<span class="pl-v">Field</span>()
    <span class="pl-s1">iata</span> <span class="pl-c1">=</span> <span class="pl-s1">scrapy</span>.<span class="pl-v">Field</span>()
    <span class="pl-s1">icao</span> <span class="pl-c1">=</span> <span class="pl-s1">scrapy</span>.<span class="pl-v">Field</span>()
    <span class="pl-s1">elevation</span> <span class="pl-c1">=</span> <span class="pl-s1">scrapy</span>.<span class="pl-v">Field</span>()
    <span class="pl-s1">elevation_unit</span> <span class="pl-c1">=</span> <span class="pl-s1">scrapy</span>.<span class="pl-v">Field</span>()
    <span class="pl-s1">dateopened</span> <span class="pl-c1">=</span> <span class="pl-s1">scrapy</span>.<span class="pl-v">Field</span>()</pre></div>
<p dir="auto"><code class="notranslate">pipelines.py</code></p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import os.path
from datetime import timedelta as td
from scrapy.exporters import CsvItemExporter
from scrapy.exceptions import DropItem
from asnscraper.items import Disaster, DisasterRaw, Airport, AirportRaw

import re

DATADIR = r'..\data'


class ExportPipeline:
    dbs = [Airport, Disaster]
    exporters = {}

    @staticmethod
    def dbname(db):
        return db.__name__.lower()

    @staticmethod
    def file_path(file, folder = DATADIR):
        return os.path.abspath(os.path.join(folder, file))

    def open_spider(self, spider):

        for db in self.dbs:
            db_name = self.dbname(db)
            print(f&quot;Exporting {db_name}s to {db_name}s.csv&quot;)
            e = CsvItemExporter(open(self.file_path(f&quot;{db_name}s.csv&quot;), 'wb'))
            self.exporters[db_name] = e
            self.exporters[db_name].start_exporting()

        return spider

    def close_spider(self, spider):
        for k, e in self.exporters.items():
            e.finish_exporting()

        return spider

    def process_item(self, item, spider):
        for db in self.dbs:
            if isinstance(item, db):
                db_name = self.dbname(db)
                self.exporters[db_name].export_item(item)
        return item


class AirportPipeline:

    def process_item(self, raw, spider):
        if not isinstance(raw, AirportRaw):
            return raw
        # df = raw['allfields']
        airport = Airport()
        airport['id'] = raw['id']
        airport['name'] = raw['name'].extract()[0].strip()
        return airport


class DisasterPipeline:

    def process_item(self, raw, spider):
        if not isinstance(raw, DisasterRaw):
            return raw
        df = raw['allfields']
        disaster = Disaster()
        disaster['id'] = raw['id']
        timestr = extract_field(field='Time', df=df)
        try:
            hours, minutes = re.match(r'(\d\d):(\d\d)', timestr).groups()
        except (AttributeError, TypeError):
            hours, minutes = (0, 0)
        finally:
            deltat = td(hours=(int(hours)), minutes=(int(minutes)))
            total_dt = raw['date'] + deltat
            disaster['datetime'] = total_dt
            disaster['crew_deaths'], disaster['crew_total'] = fatalities('Crew', df=df)
            disaster['passenger_deaths'], disaster['passenger_total'] = \
                fatalities('Passengers', df=df)
            disaster['souls_deaths'], disaster['souls_total'] = fatalities('Total', df=df)
            disaster['ground_deaths'] = fatalities_ground(df=df)
            disaster['damage'] = extract_field(field='Aircraft damage', df=df)
            disaster['fate'] = extract_field(field='Aircraft fate', df=df)
            disaster['aircraft'] = extract_field(field='Registration', df=df)
            disaster['location'] = extract_field(field='Location', df=df)
            disaster['flightphase'] = extract_field(field='Phase', df=df)
            disaster['nature'] = extract_field(field='Nature', df=df)
            disaster['ap_from'] = extract_field(field='Departure airport', df=df)
            disaster['ap_to'] = extract_field(field='Destination airport', df=df)
            disaster['flightnumber'] = extract_field(field='Flightnumber', df=df)
            return disaster


def extract_field(field, df):
    query = f'td[@class=&quot;caption&quot; and text()=&quot;{field}:&quot;]/following-sibling::td//text()'
    query2 = f'td[@class=&quot;caption&quot;]//*[text()=&quot;{field}:&quot;]/' \
             f'ancestor::td/following-sibling::td//text()'
    ret = df.xpath(query).extract() or df.xpath(query2).extract()
    if isinstance(ret, list):
        if len(ret) == 0:
            return
        ret = [str(r) for r in ret]
        return ''.join(ret).strip()
    if isinstance(ret, str):
        return ret.strip()
    return ret


def fatalities(field, df):
    fstr = extract_field(field=field, df=df)
    if fstr is None:
        return
    fatal = re.match(r'.*[Ff]atalities:\s*(\d*)\s*/\s*[Oo]ccupants:\s*(\d*).*', fstr)
    f_deaths = int(fatal[1]) if fatal[1] != '' else None
    f_total = int(fatal[2]) if fatal[2] != '' else None
    return (
     f_deaths, f_total)


def fatalities_ground(df):
    fstr = extract_field(field='Ground casualties', df=df)
    if fstr is None:
        return
    fatal = re.match(r'.*[Ff]atalities:\s*(\d*)\s*.*', fstr)
    if fatal is None:
        return
    f_deaths = int(fatal[1]) if fatal[1] != '' else None
    return f_deaths"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">os</span>.<span class="pl-s1">path</span>
<span class="pl-k">from</span> <span class="pl-s1">datetime</span> <span class="pl-k">import</span> <span class="pl-s1">timedelta</span> <span class="pl-k">as</span> <span class="pl-s1">td</span>
<span class="pl-k">from</span> <span class="pl-s1">scrapy</span>.<span class="pl-s1">exporters</span> <span class="pl-k">import</span> <span class="pl-v">CsvItemExporter</span>
<span class="pl-k">from</span> <span class="pl-s1">scrapy</span>.<span class="pl-s1">exceptions</span> <span class="pl-k">import</span> <span class="pl-v">DropItem</span>
<span class="pl-k">from</span> <span class="pl-s1">asnscraper</span>.<span class="pl-s1">items</span> <span class="pl-k">import</span> <span class="pl-v">Disaster</span>, <span class="pl-v">DisasterRaw</span>, <span class="pl-v">Airport</span>, <span class="pl-v">AirportRaw</span>

<span class="pl-k">import</span> <span class="pl-s1">re</span>

<span class="pl-v">DATADIR</span> <span class="pl-c1">=</span> <span class="pl-s">r'..\data'</span>


<span class="pl-k">class</span> <span class="pl-v">ExportPipeline</span>:
    <span class="pl-s1">dbs</span> <span class="pl-c1">=</span> [<span class="pl-v">Airport</span>, <span class="pl-v">Disaster</span>]
    <span class="pl-s1">exporters</span> <span class="pl-c1">=</span> {}

    <span class="pl-en">@<span class="pl-s1">staticmethod</span></span>
    <span class="pl-k">def</span> <span class="pl-en">dbname</span>(<span class="pl-s1">db</span>):
        <span class="pl-k">return</span> <span class="pl-s1">db</span>.<span class="pl-s1">__name__</span>.<span class="pl-en">lower</span>()

    <span class="pl-en">@<span class="pl-s1">staticmethod</span></span>
    <span class="pl-k">def</span> <span class="pl-en">file_path</span>(<span class="pl-s1">file</span>, <span class="pl-s1">folder</span> <span class="pl-c1">=</span> <span class="pl-v">DATADIR</span>):
        <span class="pl-k">return</span> <span class="pl-s1">os</span>.<span class="pl-s1">path</span>.<span class="pl-en">abspath</span>(<span class="pl-s1">os</span>.<span class="pl-s1">path</span>.<span class="pl-en">join</span>(<span class="pl-s1">folder</span>, <span class="pl-s1">file</span>))

    <span class="pl-k">def</span> <span class="pl-en">open_spider</span>(<span class="pl-s1">self</span>, <span class="pl-s1">spider</span>):

        <span class="pl-k">for</span> <span class="pl-s1">db</span> <span class="pl-c1">in</span> <span class="pl-s1">self</span>.<span class="pl-s1">dbs</span>:
            <span class="pl-s1">db_name</span> <span class="pl-c1">=</span> <span class="pl-s1">self</span>.<span class="pl-en">dbname</span>(<span class="pl-s1">db</span>)
            <span class="pl-en">print</span>(<span class="pl-s">f"Exporting <span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">db_name</span><span class="pl-kos">}</span></span>s to <span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">db_name</span><span class="pl-kos">}</span></span>s.csv"</span>)
            <span class="pl-s1">e</span> <span class="pl-c1">=</span> <span class="pl-v">CsvItemExporter</span>(<span class="pl-en">open</span>(<span class="pl-s1">self</span>.<span class="pl-en">file_path</span>(<span class="pl-s">f"<span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">db_name</span><span class="pl-kos">}</span></span>s.csv"</span>), <span class="pl-s">'wb'</span>))
            <span class="pl-s1">self</span>.<span class="pl-s1">exporters</span>[<span class="pl-s1">db_name</span>] <span class="pl-c1">=</span> <span class="pl-s1">e</span>
            <span class="pl-s1">self</span>.<span class="pl-s1">exporters</span>[<span class="pl-s1">db_name</span>].<span class="pl-en">start_exporting</span>()

        <span class="pl-k">return</span> <span class="pl-s1">spider</span>

    <span class="pl-k">def</span> <span class="pl-en">close_spider</span>(<span class="pl-s1">self</span>, <span class="pl-s1">spider</span>):
        <span class="pl-k">for</span> <span class="pl-s1">k</span>, <span class="pl-s1">e</span> <span class="pl-c1">in</span> <span class="pl-s1">self</span>.<span class="pl-s1">exporters</span>.<span class="pl-en">items</span>():
            <span class="pl-s1">e</span>.<span class="pl-en">finish_exporting</span>()

        <span class="pl-k">return</span> <span class="pl-s1">spider</span>

    <span class="pl-k">def</span> <span class="pl-en">process_item</span>(<span class="pl-s1">self</span>, <span class="pl-s1">item</span>, <span class="pl-s1">spider</span>):
        <span class="pl-k">for</span> <span class="pl-s1">db</span> <span class="pl-c1">in</span> <span class="pl-s1">self</span>.<span class="pl-s1">dbs</span>:
            <span class="pl-k">if</span> <span class="pl-en">isinstance</span>(<span class="pl-s1">item</span>, <span class="pl-s1">db</span>):
                <span class="pl-s1">db_name</span> <span class="pl-c1">=</span> <span class="pl-s1">self</span>.<span class="pl-en">dbname</span>(<span class="pl-s1">db</span>)
                <span class="pl-s1">self</span>.<span class="pl-s1">exporters</span>[<span class="pl-s1">db_name</span>].<span class="pl-en">export_item</span>(<span class="pl-s1">item</span>)
        <span class="pl-k">return</span> <span class="pl-s1">item</span>


<span class="pl-k">class</span> <span class="pl-v">AirportPipeline</span>:

    <span class="pl-k">def</span> <span class="pl-en">process_item</span>(<span class="pl-s1">self</span>, <span class="pl-s1">raw</span>, <span class="pl-s1">spider</span>):
        <span class="pl-k">if</span> <span class="pl-c1">not</span> <span class="pl-en">isinstance</span>(<span class="pl-s1">raw</span>, <span class="pl-v">AirportRaw</span>):
            <span class="pl-k">return</span> <span class="pl-s1">raw</span>
        <span class="pl-c"># df = raw['allfields']</span>
        <span class="pl-s1">airport</span> <span class="pl-c1">=</span> <span class="pl-v">Airport</span>()
        <span class="pl-s1">airport</span>[<span class="pl-s">'id'</span>] <span class="pl-c1">=</span> <span class="pl-s1">raw</span>[<span class="pl-s">'id'</span>]
        <span class="pl-s1">airport</span>[<span class="pl-s">'name'</span>] <span class="pl-c1">=</span> <span class="pl-s1">raw</span>[<span class="pl-s">'name'</span>].<span class="pl-en">extract</span>()[<span class="pl-c1">0</span>].<span class="pl-en">strip</span>()
        <span class="pl-k">return</span> <span class="pl-s1">airport</span>


<span class="pl-k">class</span> <span class="pl-v">DisasterPipeline</span>:

    <span class="pl-k">def</span> <span class="pl-en">process_item</span>(<span class="pl-s1">self</span>, <span class="pl-s1">raw</span>, <span class="pl-s1">spider</span>):
        <span class="pl-k">if</span> <span class="pl-c1">not</span> <span class="pl-en">isinstance</span>(<span class="pl-s1">raw</span>, <span class="pl-v">DisasterRaw</span>):
            <span class="pl-k">return</span> <span class="pl-s1">raw</span>
        <span class="pl-s1">df</span> <span class="pl-c1">=</span> <span class="pl-s1">raw</span>[<span class="pl-s">'allfields'</span>]
        <span class="pl-s1">disaster</span> <span class="pl-c1">=</span> <span class="pl-v">Disaster</span>()
        <span class="pl-s1">disaster</span>[<span class="pl-s">'id'</span>] <span class="pl-c1">=</span> <span class="pl-s1">raw</span>[<span class="pl-s">'id'</span>]
        <span class="pl-s1">timestr</span> <span class="pl-c1">=</span> <span class="pl-en">extract_field</span>(<span class="pl-s1">field</span><span class="pl-c1">=</span><span class="pl-s">'Time'</span>, <span class="pl-s1">df</span><span class="pl-c1">=</span><span class="pl-s1">df</span>)
        <span class="pl-k">try</span>:
            <span class="pl-s1">hours</span>, <span class="pl-s1">minutes</span> <span class="pl-c1">=</span> <span class="pl-s1">re</span>.<span class="pl-en">match</span>(<span class="pl-s">r'(\d\d):(\d\d)'</span>, <span class="pl-s1">timestr</span>).<span class="pl-en">groups</span>()
        <span class="pl-k">except</span> (<span class="pl-v">AttributeError</span>, <span class="pl-v">TypeError</span>):
            <span class="pl-s1">hours</span>, <span class="pl-s1">minutes</span> <span class="pl-c1">=</span> (<span class="pl-c1">0</span>, <span class="pl-c1">0</span>)
        <span class="pl-k">finally</span>:
            <span class="pl-s1">deltat</span> <span class="pl-c1">=</span> <span class="pl-en">td</span>(<span class="pl-s1">hours</span><span class="pl-c1">=</span>(<span class="pl-en">int</span>(<span class="pl-s1">hours</span>)), <span class="pl-s1">minutes</span><span class="pl-c1">=</span>(<span class="pl-en">int</span>(<span class="pl-s1">minutes</span>)))
            <span class="pl-s1">total_dt</span> <span class="pl-c1">=</span> <span class="pl-s1">raw</span>[<span class="pl-s">'date'</span>] <span class="pl-c1">+</span> <span class="pl-s1">deltat</span>
            <span class="pl-s1">disaster</span>[<span class="pl-s">'datetime'</span>] <span class="pl-c1">=</span> <span class="pl-s1">total_dt</span>
            <span class="pl-s1">disaster</span>[<span class="pl-s">'crew_deaths'</span>], <span class="pl-s1">disaster</span>[<span class="pl-s">'crew_total'</span>] <span class="pl-c1">=</span> <span class="pl-en">fatalities</span>(<span class="pl-s">'Crew'</span>, <span class="pl-s1">df</span><span class="pl-c1">=</span><span class="pl-s1">df</span>)
            <span class="pl-s1">disaster</span>[<span class="pl-s">'passenger_deaths'</span>], <span class="pl-s1">disaster</span>[<span class="pl-s">'passenger_total'</span>] <span class="pl-c1">=</span> \
                <span class="pl-en">fatalities</span>(<span class="pl-s">'Passengers'</span>, <span class="pl-s1">df</span><span class="pl-c1">=</span><span class="pl-s1">df</span>)
            <span class="pl-s1">disaster</span>[<span class="pl-s">'souls_deaths'</span>], <span class="pl-s1">disaster</span>[<span class="pl-s">'souls_total'</span>] <span class="pl-c1">=</span> <span class="pl-en">fatalities</span>(<span class="pl-s">'Total'</span>, <span class="pl-s1">df</span><span class="pl-c1">=</span><span class="pl-s1">df</span>)
            <span class="pl-s1">disaster</span>[<span class="pl-s">'ground_deaths'</span>] <span class="pl-c1">=</span> <span class="pl-en">fatalities_ground</span>(<span class="pl-s1">df</span><span class="pl-c1">=</span><span class="pl-s1">df</span>)
            <span class="pl-s1">disaster</span>[<span class="pl-s">'damage'</span>] <span class="pl-c1">=</span> <span class="pl-en">extract_field</span>(<span class="pl-s1">field</span><span class="pl-c1">=</span><span class="pl-s">'Aircraft damage'</span>, <span class="pl-s1">df</span><span class="pl-c1">=</span><span class="pl-s1">df</span>)
            <span class="pl-s1">disaster</span>[<span class="pl-s">'fate'</span>] <span class="pl-c1">=</span> <span class="pl-en">extract_field</span>(<span class="pl-s1">field</span><span class="pl-c1">=</span><span class="pl-s">'Aircraft fate'</span>, <span class="pl-s1">df</span><span class="pl-c1">=</span><span class="pl-s1">df</span>)
            <span class="pl-s1">disaster</span>[<span class="pl-s">'aircraft'</span>] <span class="pl-c1">=</span> <span class="pl-en">extract_field</span>(<span class="pl-s1">field</span><span class="pl-c1">=</span><span class="pl-s">'Registration'</span>, <span class="pl-s1">df</span><span class="pl-c1">=</span><span class="pl-s1">df</span>)
            <span class="pl-s1">disaster</span>[<span class="pl-s">'location'</span>] <span class="pl-c1">=</span> <span class="pl-en">extract_field</span>(<span class="pl-s1">field</span><span class="pl-c1">=</span><span class="pl-s">'Location'</span>, <span class="pl-s1">df</span><span class="pl-c1">=</span><span class="pl-s1">df</span>)
            <span class="pl-s1">disaster</span>[<span class="pl-s">'flightphase'</span>] <span class="pl-c1">=</span> <span class="pl-en">extract_field</span>(<span class="pl-s1">field</span><span class="pl-c1">=</span><span class="pl-s">'Phase'</span>, <span class="pl-s1">df</span><span class="pl-c1">=</span><span class="pl-s1">df</span>)
            <span class="pl-s1">disaster</span>[<span class="pl-s">'nature'</span>] <span class="pl-c1">=</span> <span class="pl-en">extract_field</span>(<span class="pl-s1">field</span><span class="pl-c1">=</span><span class="pl-s">'Nature'</span>, <span class="pl-s1">df</span><span class="pl-c1">=</span><span class="pl-s1">df</span>)
            <span class="pl-s1">disaster</span>[<span class="pl-s">'ap_from'</span>] <span class="pl-c1">=</span> <span class="pl-en">extract_field</span>(<span class="pl-s1">field</span><span class="pl-c1">=</span><span class="pl-s">'Departure airport'</span>, <span class="pl-s1">df</span><span class="pl-c1">=</span><span class="pl-s1">df</span>)
            <span class="pl-s1">disaster</span>[<span class="pl-s">'ap_to'</span>] <span class="pl-c1">=</span> <span class="pl-en">extract_field</span>(<span class="pl-s1">field</span><span class="pl-c1">=</span><span class="pl-s">'Destination airport'</span>, <span class="pl-s1">df</span><span class="pl-c1">=</span><span class="pl-s1">df</span>)
            <span class="pl-s1">disaster</span>[<span class="pl-s">'flightnumber'</span>] <span class="pl-c1">=</span> <span class="pl-en">extract_field</span>(<span class="pl-s1">field</span><span class="pl-c1">=</span><span class="pl-s">'Flightnumber'</span>, <span class="pl-s1">df</span><span class="pl-c1">=</span><span class="pl-s1">df</span>)
            <span class="pl-k">return</span> <span class="pl-s1">disaster</span>


<span class="pl-k">def</span> <span class="pl-en">extract_field</span>(<span class="pl-s1">field</span>, <span class="pl-s1">df</span>):
    <span class="pl-s1">query</span> <span class="pl-c1">=</span> <span class="pl-s">f'td[@class="caption" and text()="<span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">field</span><span class="pl-kos">}</span></span>:"]/following-sibling::td//text()'</span>
    <span class="pl-s1">query2</span> <span class="pl-c1">=</span> <span class="pl-s">f'td[@class="caption"]//*[text()="<span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">field</span><span class="pl-kos">}</span></span>:"]/'</span> \
             <span class="pl-s">f'ancestor::td/following-sibling::td//text()'</span>
    <span class="pl-s1">ret</span> <span class="pl-c1">=</span> <span class="pl-s1">df</span>.<span class="pl-en">xpath</span>(<span class="pl-s1">query</span>).<span class="pl-en">extract</span>() <span class="pl-c1">or</span> <span class="pl-s1">df</span>.<span class="pl-en">xpath</span>(<span class="pl-s1">query2</span>).<span class="pl-en">extract</span>()
    <span class="pl-k">if</span> <span class="pl-en">isinstance</span>(<span class="pl-s1">ret</span>, <span class="pl-s1">list</span>):
        <span class="pl-k">if</span> <span class="pl-en">len</span>(<span class="pl-s1">ret</span>) <span class="pl-c1">==</span> <span class="pl-c1">0</span>:
            <span class="pl-k">return</span>
        <span class="pl-s1">ret</span> <span class="pl-c1">=</span> [<span class="pl-en">str</span>(<span class="pl-s1">r</span>) <span class="pl-k">for</span> <span class="pl-s1">r</span> <span class="pl-c1">in</span> <span class="pl-s1">ret</span>]
        <span class="pl-k">return</span> <span class="pl-s">''</span>.<span class="pl-en">join</span>(<span class="pl-s1">ret</span>).<span class="pl-en">strip</span>()
    <span class="pl-k">if</span> <span class="pl-en">isinstance</span>(<span class="pl-s1">ret</span>, <span class="pl-s1">str</span>):
        <span class="pl-k">return</span> <span class="pl-s1">ret</span>.<span class="pl-en">strip</span>()
    <span class="pl-k">return</span> <span class="pl-s1">ret</span>


<span class="pl-k">def</span> <span class="pl-en">fatalities</span>(<span class="pl-s1">field</span>, <span class="pl-s1">df</span>):
    <span class="pl-s1">fstr</span> <span class="pl-c1">=</span> <span class="pl-en">extract_field</span>(<span class="pl-s1">field</span><span class="pl-c1">=</span><span class="pl-s1">field</span>, <span class="pl-s1">df</span><span class="pl-c1">=</span><span class="pl-s1">df</span>)
    <span class="pl-k">if</span> <span class="pl-s1">fstr</span> <span class="pl-c1">is</span> <span class="pl-c1">None</span>:
        <span class="pl-k">return</span>
    <span class="pl-s1">fatal</span> <span class="pl-c1">=</span> <span class="pl-s1">re</span>.<span class="pl-en">match</span>(<span class="pl-s">r'.*[Ff]atalities:\s*(\d*)\s*/\s*[Oo]ccupants:\s*(\d*).*'</span>, <span class="pl-s1">fstr</span>)
    <span class="pl-s1">f_deaths</span> <span class="pl-c1">=</span> <span class="pl-en">int</span>(<span class="pl-s1">fatal</span>[<span class="pl-c1">1</span>]) <span class="pl-k">if</span> <span class="pl-s1">fatal</span>[<span class="pl-c1">1</span>] <span class="pl-c1">!=</span> <span class="pl-s">''</span> <span class="pl-k">else</span> <span class="pl-c1">None</span>
    <span class="pl-s1">f_total</span> <span class="pl-c1">=</span> <span class="pl-en">int</span>(<span class="pl-s1">fatal</span>[<span class="pl-c1">2</span>]) <span class="pl-k">if</span> <span class="pl-s1">fatal</span>[<span class="pl-c1">2</span>] <span class="pl-c1">!=</span> <span class="pl-s">''</span> <span class="pl-k">else</span> <span class="pl-c1">None</span>
    <span class="pl-k">return</span> (
     <span class="pl-s1">f_deaths</span>, <span class="pl-s1">f_total</span>)


<span class="pl-k">def</span> <span class="pl-en">fatalities_ground</span>(<span class="pl-s1">df</span>):
    <span class="pl-s1">fstr</span> <span class="pl-c1">=</span> <span class="pl-en">extract_field</span>(<span class="pl-s1">field</span><span class="pl-c1">=</span><span class="pl-s">'Ground casualties'</span>, <span class="pl-s1">df</span><span class="pl-c1">=</span><span class="pl-s1">df</span>)
    <span class="pl-k">if</span> <span class="pl-s1">fstr</span> <span class="pl-c1">is</span> <span class="pl-c1">None</span>:
        <span class="pl-k">return</span>
    <span class="pl-s1">fatal</span> <span class="pl-c1">=</span> <span class="pl-s1">re</span>.<span class="pl-en">match</span>(<span class="pl-s">r'.*[Ff]atalities:\s*(\d*)\s*.*'</span>, <span class="pl-s1">fstr</span>)
    <span class="pl-k">if</span> <span class="pl-s1">fatal</span> <span class="pl-c1">is</span> <span class="pl-c1">None</span>:
        <span class="pl-k">return</span>
    <span class="pl-s1">f_deaths</span> <span class="pl-c1">=</span> <span class="pl-en">int</span>(<span class="pl-s1">fatal</span>[<span class="pl-c1">1</span>]) <span class="pl-k">if</span> <span class="pl-s1">fatal</span>[<span class="pl-c1">1</span>] <span class="pl-c1">!=</span> <span class="pl-s">''</span> <span class="pl-k">else</span> <span class="pl-c1">None</span>
    <span class="pl-k">return</span> <span class="pl-s1">f_deaths</span></pre></div>
<h3 dir="auto">Steps to Reproduce</h3>
<ol dir="auto">
<li>Start a blank Scrapy project and copy the code above to the relevant files. Other files are default;</li>
<li>Configure PyCharm to run it in the Python Console, as indicated in my screenshots;</li>
<li>Run the crawler;</li>
<li>Open the generated csv files. They are empty.</li>
<li>Run the crawler through PyCharm's Debugger;</li>
<li>Open the generated csv files again. They are not empty.</li>
<li>Run the crawler through the command line.</li>
<li>Open the generated csv files. They are the same as in step 6.</li>
</ol>
<p dir="auto"><strong>Expected behavior:</strong> Everytime I open the generated csv files, they should be exactly the same regardless of how I run it.</p>
<p dir="auto"><strong>Actual behavior:</strong> They are not the same. When running the above crawler through PyCharm's Python Console, they are empty. Otherwise, they are not empty.</p>
<p dir="auto"><strong>Reproduces how often:</strong> Always happens</p>
<h3 dir="auto">Versions</h3>
<p dir="auto">Python version: <code class="notranslate">Python 3.8.5 (default, Aug  5 2020, 09:44:06) [MSC v.1916 64 bit (AMD64)] on win32</code></p>
<p dir="auto">Anaconda version:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="&gt; conda info

     active environment : asn
    active env location : C:\ProgramData\Anaconda3\envs\asn
            shell level : 2
       user config file : C:\Users\*\.condarc
 populated config files :
          conda version : 4.8.4
    conda-build version : 3.18.11
         python version : 3.7.6.final.0
       virtual packages : __cuda=9.1
       base environment : C:\ProgramData\Anaconda3  (writable)
           channel URLs : https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
          package cache : C:\ProgramData\Anaconda3\pkgs
                          C:\Users\*\.conda\pkgs
                          C:\Users\*\AppData\Local\conda\conda\pkgs
       envs directories : C:\ProgramData\Anaconda3\envs
                          C:\Users\*\.conda\envs
                          C:\Users\*\AppData\Local\conda\conda\envs
               platform : win-64
             user-agent : conda/4.8.4 requests/2.22.0 CPython/3.7.6 Windows/10 Windows/10.0.18362
          administrator : False
             netrc file : None
           offline mode : False"><pre class="notranslate"><code class="notranslate">&gt; conda info

     active environment : asn
    active env location : C:\ProgramData\Anaconda3\envs\asn
            shell level : 2
       user config file : C:\Users\*\.condarc
 populated config files :
          conda version : 4.8.4
    conda-build version : 3.18.11
         python version : 3.7.6.final.0
       virtual packages : __cuda=9.1
       base environment : C:\ProgramData\Anaconda3  (writable)
           channel URLs : https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
          package cache : C:\ProgramData\Anaconda3\pkgs
                          C:\Users\*\.conda\pkgs
                          C:\Users\*\AppData\Local\conda\conda\pkgs
       envs directories : C:\ProgramData\Anaconda3\envs
                          C:\Users\*\.conda\envs
                          C:\Users\*\AppData\Local\conda\conda\envs
               platform : win-64
             user-agent : conda/4.8.4 requests/2.22.0 CPython/3.7.6 Windows/10 Windows/10.0.18362
          administrator : False
             netrc file : None
           offline mode : False
</code></pre></div>
<p dir="auto">I don't know why conda says the Python version is 3.7.6. It is in fact 3.8.5, as indicated above.</p>
<p dir="auto">Scrapy version</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="&gt; scrapy version --verbose
Scrapy       : 1.6.0
lxml         : 4.5.2.0
libxml2      : 2.9.10
cssselect    : 1.1.0
parsel       : 1.5.2
w3lib        : 1.21.0
Twisted      : 20.3.0
Python       : 3.8.5 (default, Aug  5 2020, 09:44:06) [MSC v.1916 64 bit (AMD64)]
pyOpenSSL    : 19.1.0 (OpenSSL 1.1.1g  21 Apr 2020)
cryptography : 2.9.2
Platform     : Windows-10-10.0.18362-SP0"><pre class="notranslate"><code class="notranslate">&gt; scrapy version --verbose
Scrapy       : 1.6.0
lxml         : 4.5.2.0
libxml2      : 2.9.10
cssselect    : 1.1.0
parsel       : 1.5.2
w3lib        : 1.21.0
Twisted      : 20.3.0
Python       : 3.8.5 (default, Aug  5 2020, 09:44:06) [MSC v.1916 64 bit (AMD64)]
pyOpenSSL    : 19.1.0 (OpenSSL 1.1.1g  21 Apr 2020)
cryptography : 2.9.2
Platform     : Windows-10-10.0.18362-SP0
</code></pre></div>
<h3 dir="auto">Additional Information</h3>
<p dir="auto">Posted to <a href="https://stackoverflow.com/questions/63696103/scrapy-doesnt-properly-export-when-run-through-pycharms-python-console" rel="nofollow">Stack Overflow</a> on 2020-09-02, no answers so far.</p>
<p dir="auto">Proxy <code class="notranslate">CONNECT</code> support is required for https urls to work.</p>
<p dir="auto">When proxy support was added to Scrapy, urllib still didn't supported <code class="notranslate">CONNECT</code>, but that has been fixed now:</p>
<p dir="auto"><a href="http://bugs.python.org/issue1424152" rel="nofollow">http://bugs.python.org/issue1424152</a></p>
<p dir="auto">Now we need to add support to Scrapy (which doesn't use urllib at all).</p>
<p dir="auto">This link contains useful tips on how to add <code class="notranslate">CONNECT</code> support with Twisted:</p>
<p dir="auto"><a href="http://twistedmatrix.com/pipermail/twisted-web/2008-August/003878.html" rel="nofollow">http://twistedmatrix.com/pipermail/twisted-web/2008-August/003878.html</a></p>
<p dir="auto">see <a href="http://dev.scrapy.org/ticket/159" rel="nofollow">http://dev.scrapy.org/ticket/159</a> for more info</p>