Stack Exchange
how to prevent failure of subprocess stopping the main process in python
I wrote a python script to run a command called "gtdownload" on a bunch of files with multiprocessing. The function "download" is where I am having trouble with.
#/usr/bin/env python

import os, sys, subprocess
from multiprocessing import Pool

def cghub_dnld_file(file1, file2, file3, np):
    <open files>
    <read in lines>
    p = Pool(int(np))
    map_args = [(line.rstrip(),name_lines[i].rstrip(),bar_lines[i].rstrip()) for i, line in enumerate(id_lines)]
    p.map(download_wrapper,map_args)

def download(id, name, bar):
    <check if file has been downloaded, if not download>
    <.....>
    link = "https://cghub.ucsc.edu/cghub/data/analysis/download/" + id
    dnld_cmd = "gtdownload -c ~/.cghub.key --max-children 4 -vv -d " + link + " > gt.out 2>gt.err"
    subprocess.call(dnld_cmd,shell=True)

def download_wrapper(args):
    return download(*args)

def main():
    <read in arguments>
    <...>
    cghub_dnld_file(file1,file2,file3,threads)

if __name__ == "__main__":
   main()

If this file does not exist in the database, gtdownload would quit which also kills my python job with the following error:
Traceback (most recent call last):
  File "/rsrch1/rists/djiao/bin/cghub_dnld.py", line 102, in <module>
    main()
  File "/rsrch1/rists/djiao/bin/cghub_dnld.py", line 98, in main
    cghub_dnld_file(file1,file2,file3,threads)
  File "/rsrch1/rists/djiao/bin/cghub_dnld.py", line 22, in cghub_dnld_file
    p.map(download_wrapper,map_args)
  File "/rsrch1/rists/apps/x86_64-rhel6/anaconda/lib/python2.7/multiprocessing/pool.py", line 250, in map
    return self.map_async(func, iterable, chunksize).get()
  File "/rsrch1/rists/apps/x86_64-rhel6/anaconda/lib/python2.7/multiprocessing/pool.py", line 554, in get
    raise self._value
OSError: [Errno 2] No such file or directory

The actual error message from gtdownload :
Welcome to gtdownload-3.8.5a.
Ready to download
Communicating with GT Executive ...
Headers received from the client:  'HTTP/1.1 100 Continue

HTTP/1.1 404 Not Found
Date: Tue, 29 Jul 2014 18:49:57 GMT
Server: Apache/2.2.15 (Red Hat and CGHub)
Strict-Transport-Security: max-age=31536000
X-Powered-By: PHP/5.3.3
Content-Length: 669
Connection: close
Content-Type: text/xml

'

Error:  You have requested to download a uuid which either does not exist within the system, or has not yet reached the 'live' state.  The requested action will not be performed.  Please double check the supplied uuid or contact thelpdesk for further assistance.

I would like the script to skip the one that does not exist and start gtdownload on the next one. I tried to output the stderr of subprocess.call to a pipe and see if there is the "error" keyword. But it seems it stops at the exact subprocess.call command. Same thing with os.system.
I made a MCV case without the multiprocessing and subprocess did not kill the main process at all. Looks like multiprocessing messes things up although I had it run with 1 thread just for testing.
#!/usr/bin/env python
import subprocess
#THis is the id that gtdownload had problem with
id = "df1e073f-4485-4d5f-8659-cd8eaac17329"
link = "https://cghub.ucsc.edu/cghub/data/analysis/download/" + id
dlnd_cmd = "gtdownload -c ~/.cghub.key --max-children 4 -vv -d " + link + " > gt.out 2>gt.err"
print dlnd_cmd
subprocess.call(dlnd_cmd,shell=True)
print "done"

Clearly multiprocessing conflicts subprocess.call but it is not clear to me why. 

I don't believe this is your real code. As implied by @dano's answer, this code (unless you're on Windows) will raise a FileNotFoundError (3.3+) or OSError (earlier) from the subprocess.call line, without ever running anything named gtdownload. And yet, you show output from gtdownload running.

Also, the fact that you're doing this on a multiprocessing pool makes it much harder to debug. Please give us a MCVE.

@abarnert I followed your advice and created a MCVE with just these lines: id = "df1e073f-4485-4d5f-8659-cd8eaac17329" link = "https://cghub.ucsc.edu/cghub/data/analysis/download/" + id dlnd_cmd = "gtdownload -c ~/.cghub.key --max-children 4 -vv -d " + link + " > gt.out 2>gt.err"  subprocess.call(dlnd_cmd,shell=True). It seems to be able to go past beyond subprocess.call. So maybe the multiprocessing thing is messing things up. But the code definitely exited at "call" from the original script. What could be wrong?

Read the help on MCVE, it explains what to do. Start with something minimal, and keep adding to it until you can reproduce the problem. Then post that (and do it by editing your question, not in a comment). Showing us code that works, and then telling us that your real code that we haven't seen doesn't work, doesn't help anyone trying to debug your problem.

@abarnert I edited my question by adding the MCVE. I don't think adding more lines would make it any different because clearly multiprocessing is what causes the issue. "Why" is something that is still not clear

Please read the help again: the "V" stands for "Verifiable": "… make sure it reproduces the problem". Code that doesn't reproduce the problem is not an MCVE. If you're sure multiprocessing is what causes the issue, add some trivial multiprocessing to the example so that it does demonstrate the problem, which makes it an MCVE, which means people can debug it for you.

Also, you may want to read How to debug small programs, a blog post that explains how to get as far as you can on your own, and then how to tell us exactly what we need to help you get past that point. (Skip over the parts that are specific to C-like languages, like compiler warnings; the rest is useful to every programmer.)

@abarnert I took your advice and did some debugging. It turned out the line that is really causing it is somewhere else (I was trying to call rename on some file that does not exist and thus the error). I guess the multiprocessing thing makes it trickier to locate the problem. The line the code seems to quit from is not necessarily where it actually fails. Lessen learned. Thanks.


What is the best way to avoid the failure of subprocess killing the main process?

Handle the exception in some appropriate way and move on, of course.
try:
    subprocess.call(dlnd_cmd)
except OSError as e:
    print 'failed to download: {!r}'.format(e)

However, this may not be appropriate here. The kinds of exceptions that subprocess.call raises are usually not transient things that you can just log and work around; if it's not working now, it will continue to not work forever until you fix the underlying problem (a bug in your script, or gtdownload not being installed right, or whatever).
For example, if the code you showed us is your actual code:
dlnd_cmd = "gtdownload -c ~/.cghub.key --max-children 4 -vv -d " + link + " > gt.out 2>gt.err"
subprocess.call(dlnd_cmd)

… then this is guaranteed to raise an OSError for the reason explained in dano's answer: call (without shell=True) will try to take that entire string—spaces, shell-redirection, etc.—as the name of an executable program to find on your $PATH. And there is no such program. So it will raise an OSError(errno.ENOENT). (Which is exactly what you're seeing.) Just logging that doesn't do you any good; it's a good thing that your entire process is exiting, so you can debug that problem.

subprocess.call should not kill the main process. Something else must be wrong with your script or your conclusions about the script's behaviour is wrong. Did you try printing some trace output after the subprocess call?

like I said, the whole thing stops at subprocess.call, it does not execute anything after this line. The script works perfectly fine when all the items exist and downloadable in the online database.

@MrWhite In that case gtdownload must simply not have terminated yet, and subprocess.call is still waiting for it to do so.

Both gtdownload and python were definitely terminated. I just added the python error in my original post. It is a OSError.

@MrWhite I don't see subprocess.call() anywhere in that backtrace. It looks like something else is failing, or is that a backtrace from gtdownload? You should try your three lines of python code in a standalone program.

@flodin It's not in the backtrace because the original backtrace from a worker process is lost when you use multiprocessing.map. You just get the exception string itself, and the backtrace points to the call to map. It's a pain.

@flodin "cghub_dnld_file" calls "download_wrapper" which calls "download" (this is where gtdownload is executed).

This answer isn't really correct; subprocess.call definitely should raise an exception if you try to run a program that doesn't exist (and, as @dano's answer shows, he's doing that), and if you don't handle that exception, that will make the process exit with a traceback (which is exactly what the question shows happening).

@abarnert right, but judging from the way the question is formulated it is finding the program, because he is showing what he claims to be actual output from the subprocess when it is run from the script. He is claiming that the subprocess does start correctly, but on the occasions when it fails to download a file it somehow magically terminates its parent process (Also, the traceback was not in the question when I wrote this answer.)

You have to use shell=True to use subprocess.call with a string for an argument (and with shell redirection):
subprocess.call(dlnd_cmd, shell=True)

Without shell=True, subprocess tries to treat your entire command string like a single executable name, which of course doesn't exist, and leads to the No such file or directory exception.
See this answer for more info on when to use a string vs. when to use a sequence with subprocess.

I don't think the "no such file or directory error" refers to my command name. I tried os.system(dlnd_cmd) before which fails when the entry does not exist that is why I switched to subprocess.call. Anyway, I added "shell=True", same thing.

That's good advice in general (and his code certainly seems like it should be raising on the call for exactly this reason—there's almost certainly no executable file named gtdownload -c ~/.cghub.key --max-children … on the PATH), but the traceback seems to show that somehow he's actually calling gtdownload.

@MrWhite: os.system uses the shell, to parse your command line as a shell command line, which is why it works. subprocess.call (without shell=True) will take the whole string, spaces and redirection commands and other shell special characters and all, to be the name of a program to find on your $PATH and run, so it won't work.

@abarnert os.system never worked, it had the same OSError issue which kills the whole process

@MrWhite: Actually "the same OSError", or just "an OSError with details I didn't look at"? There are plenty of reasons for both system and call to raise an exception, after all.

I actually can't think a reason os.system or call (with shell=True) would raise an OSError. Any error should just end up on stderr, because the shell is handling it.

@dano: Off the top of my head, shouldn't os.environ['SHELL'] = '/no/such/file'; os.system('ls') do it? Or maybe a script running as a dedicated user who isn't allowed to run $SHELL?

@abarnet According to the docs: "On Unix with shell=True, the shell defaults to /bin/sh". So I guess if there's no /bin/sh (perhaps in addition to SHELL not being set) this could happen?

@dano: That seems like an even more obscure possibility; a system can't officially be Unix (or POSIX) without having /bin/sh, and it's hard to imagine a real-life system that wouldn't… although I suppose a badly broken chroot jail could cause that? At any rate, I doubt any of this speculative stuff is the OP's actual problem. I suspect he's just giving us imprecise information about his code and his errors, and if we knew the right details they would be easy to explain.

@abarnert Agreed on all of that.

@abarnert, Ok I see what I did wrong with subprocess.call. Now I added shell=True and again ended up with OSError. The reason I think it is the same error with os.system is because the last error message printed out in gt.err (stderr of gtdownload) was the same for both cases.

@MrWhite: Hold on. Is gtdownload itself a Python script, and maybe os.system successfully runs a Python script that itself fails with an unhandled OSError, which you treated as "the same thing" as subprocess.call raising an exception of the same type in your main program? And, if not, can you please actually give us the information instead of making us guess wildly based on vague descriptions?

@abarnert gtdownload is written in c.
