Scrapy ImportError: No module named project.settings when using subprocess.Popen
I suggest let python focus on the scrape task and use something else for process control. If it were me, I'd write a small bash script to run your program. 
Test that the launcher script works by running it with env -i yourscript.sh because that will make sure it runs without any inherited environment settings. 
Once the bash script works correctly, including setting up virtualenv etc, you could have python run that bash script, not python. You've sidestepped any strange environment issues at that point and got yourself a pretty solid launcher script. 
Even better, given you have the bash script at that point, use a "proper" process controller (daemontools, supervisor...) spin up the process, restart on crash, etc.
I have scrapy crawler scraping thru sites. On some occasions scrapy kills itself due to RAM issues. I rewrote the spider such that it can be split and run for a site.
After the initial run, I use subprocess.Popen to submit the scrapy crawler again with new start item.
But I am getting error 
ImportError: No module named shop.settingsTraceback (most recent call last): 
File "/home/kumar/envs/ishop/bin/scrapy", line 4, in <module> execute() 
File "/home/kumar/envs/ishop/lib/python2.7/site-packages/scrapy/cmdline.py", line 109, in execute settings = get_project_settings() 
File "/home/kumar/envs/ishop/lib/python2.7/site-packages/scrapy/utils/project.py", line 60, in get_project_settings settings.setmodule(settings_module_path, priority='project') 
File "/home/kumar/envs/ishop/lib/python2.7/site-packages/scrapy/settings/__init__.py", line 109, in setmodule module = import_module(module) 
File "/usr/lib64/python2.7/importlib/__init__.py", line 37, in import_module __import__(name)ImportError: No module named shop.settings
The subprocess cmd is 
newp = Popen(comm, stderr=filename, stdout=filename, cwd=fp, shell=True)

comm - 
source /home/kumar/envs/ishop/bin/activate && cd /home/kumar/projects/usg/shop/spiders/../.. && /home/kumar/envs/ishop/bin/scrapy crawl -a category=laptop -a site=newsite -a start=2 -a numpages=10 -a split=1 'allsitespider'
cwd - /home/kumar/projects/usg

I checked sys.path and it is correct ['/home/kumar/envs/ishop/bin', '/home/kumar/envs/ishop/lib64/python27.zip', '/home/kumar/envs/ishop/lib64/python2.7', '/home/kumar/envs/ishop/lib64/python2.7/plat-linux2', '/home/kumar/envs/ishop/lib64/python2.7/lib-tk', '/home/kumar/envs/ishop/lib64/python2.7/lib-old', '/home/kumar/envs/ishop/lib64/python2.7/lib-dynload', '/usr/lib64/python2.7', '/usr/lib/python2.7', '/home/kumar/envs/ishop/lib/python2.7/site-packages']
But looks like the import statement is using "/usr/lib64/python2.7/importlib/__init__.py" instead of my virtual env.
Where am I wrong? Help please?