pipe large amount of data to stdin while using subprocess.Popen
I'm kinda struggling to understand what is the python way of solving this simple problem

my problem is quite simple. If you use the follwing code it will hang. This is well documented in the subprocess module doc.

    import subprocess


    proc = subprocess.Popen(['cat','-'],
                            stdin=subprocess.PIPE,
                            stdout=subprocess.PIPE,
                            )

    for i in range(100000):
        proc.stdin.write('%d\n' % i)

    output = proc.communicate()[0]
    print output

searching for a solution (there is a very insightful thread, but I've lost it now) I found this solution (among others) that uses an explicit fork:

       import os
    import sys
    from subprocess import Popen, PIPE

    def produce(to_sed):
        for i in range(100000):
            to_sed.write("%d\n" % i)
            to_sed.flush()
        #this would happen implicitly, anyway, but is here for the example
        to_sed.close()

    def consume(from_sed):
        while 1:
            res = from_sed.readline()
            if not res:
                sys.exit(0)
                #sys.exit(proc.poll())
            print 'received: ', [res]

    def main():
        proc = Popen(['cat','-'],stdin=PIPE,stdout=PIPE)
        to_sed = proc.stdin
        from_sed = proc.stdout

        pid = os.fork()
        if pid == 0 :
            from_sed.close()
            produce(to_sed)
            return
        else :
            to_sed.close()
            consume(from_sed)

    if __name__ == '__main__':
        main()

while this solution is conceptually very easy to understand, it uses one more process and stuck as too low level compared to the subprocess module (that is there just to hide this kind of things...).

I'm wondering: is there a simple and clean solution using the subprocess module that won't hung or to implement this patter I have to do a step back and implement an old-style select loop  or an explicit fork ?

thanks

pipe large amount of data to stdin while using subprocess.Popen
===

I'm kinda struggling to understand what is the python way of solving this simple problem

my problem is quite simple. If you use the follwing code it will hang. This is well documented in the subprocess module doc.

    import subprocess


    proc = subprocess.Popen(['cat','-'],
                            stdin=subprocess.PIPE,
                            stdout=subprocess.PIPE,
                            )

    for i in range(100000):
        proc.stdin.write('%d\n' % i)

    output = proc.communicate()[0]
    print output

searching for a solution (there is a very insightful thread, but I've lost it now) I found this solution (among others) that uses an explicit fork:

       import os
    import sys
    from subprocess import Popen, PIPE

    def produce(to_sed):
        for i in range(100000):
            to_sed.write("%d\n" % i)
            to_sed.flush()
        #this would happen implicitly, anyway, but is here for the example
        to_sed.close()

    def consume(from_sed):
        while 1:
            res = from_sed.readline()
            if not res:
                sys.exit(0)
                #sys.exit(proc.poll())
            print 'received: ', [res]

    def main():
        proc = Popen(['cat','-'],stdin=PIPE,stdout=PIPE)
        to_sed = proc.stdin
        from_sed = proc.stdout

        pid = os.fork()
        if pid == 0 :
            from_sed.close()
            produce(to_sed)
            return
        else :
            to_sed.close()
            consume(from_sed)

    if __name__ == '__main__':
        main()

while this solution is conceptually very easy to understand, it uses one more process and stuck as too low level compared to the subprocess module (that is there just to hide this kind of things...).

I'm wondering: is there a simple and clean solution using the subprocess module that won't hung or to implement this patter I have to do a step back and implement an old-style select loop  or an explicit fork ?

thanks
