Where should I keep the authentication settings when using Scrapy?
The "standard" is to use the environment. Depending on what environment you're running in you may not always be guaranteed a filesystem, however env vars are usually present in most application environments.
I have a working Scrapy scraper that does a login before proceeding to parse and retrieve data but I want to open source it with exposing my username/password. Where should I factor these out to?
I was thinking of either storing them in ENV and then accessing them via os.environ, putting them in a settings.local.py file that is imported into settings.py, or just having a configuration file that is read by settings.py.