Stack Exchange
Python Multiprocessing: Strange Behaviour Reading Single File in Parallel
I have a file with 100 lines of text. 
paulk@node013:test_parallel$ for i in {1..100}; do echo "trash" >> infile.txt; done

I want several processes to read this file in parallel.
#!/usr/bin/env python
import multiprocessing

def f( in_file, out_file ):
    for row in in_file:
        print >> out_file, row.strip()

#   out_file.seek( 0 ) # interesting line
    return

f1 = open( 'infile.txt' )
f2 = open( 'infile.txt' )

g1 = open( 'outfile1.txt', 'w' )
g2 = open( 'outfile2.txt', 'w' )

p1 = multiprocessing.Process( target=f, args=( f1, g1, ) )
p1.start()

p2 = multiprocessing.Process( target=f, args=( f2, g2, ) )
p2.start()

p1.join()
p2.join()

f1.close()
f2.close()

g1.close()
g2.close()

Without the line marked 'interesting' nothing is written to the output files:
paulk@node013:test_parallel$ wc -l *file*.txt
100 infile.txt
  0 outfile1.txt
  0 outfile2.txt
100 total

Including it works:
paulk@node013:test_parallel$ wc -l *file*.txt
 100 infile.txt
 100 outfile1.txt
 100 outfile2.txt
 300 total

Can someone please provide an explanation on why the seek() method works? Could it be that it implicitly calls the 'right' method to make sure the data is written to the files? I'm confused.
I'm using Python 2.7.
Thanks.
Paul

You are passing an open file object to another process. I don't like this; it doesn't seem very clean. I would prefer to pass the filename to the child process, and the child process would open the file, write to it, and close it. This would be clean.
I guess that when the child process writes to the file object it does some internal caching. Apparently the child process does not close the file, and ends without flushing the cache. The out_file.seek(0) statement has the side effect of flushing the cache. You can probably achieve the same with out_file.flush().
But really, just pass the filename to the child process. Otherwise whatever you achieve would differ between operating systems and Python versions.

You're right. Passing file names does not require the seek and works as expected. Thanks.
