Take the 2-minute tour ×
Stack Overflow is a question and answer site for professional and enthusiast programmers. It's 100% free, no registration required.

I am trying to work out a large program using Map-Reduce framework which needs the entire process be split into three Map-Reduce jobs which should happen sequentially.

I am using mincemeat.py because I read in many places that it is faster than octo.py and other framework implementations in python.

But I am not able to chain the multiple jobs because each client needs to give a password and get connected to server for execution. My idea is that by starting client all the jobs should run in sequence. I am a newbie in python. Appreciate if someone can help me in this regard.

Below is the code that starts a job, for example wordCount here..

s = mincemeat.Server()
s.datasource = datasource
s.mapfn = map_wordCount
s.reducefn = reduce_wordCount
wordCounts = s.run_server(password="password")
print wordCounts

I want another job's map and reduce functions to be called without the requirement for a separate client invocation of it. Anyone with pointers to how this can be done.

Thanks.

share|improve this question

1 Answer 1

Can you just not use map for a Pool of workers that will launch a batch of jobs whose goal is to launch another Pool of workers each running map-reduce jobs? I never heard of mincemeat.py, but I do this with the pathos framework… which provides a Pool with a blocking map, iterative imap, and asynchronous amap (as well as pipes) for backends such as multiprocessing, threading, mpi4py, parallel python (socket-based distributed parallel computing), and ssh-tunneling.

This has the overhead of whatever backend or backends you choose, so for very small tasks you see a good bit of the time as overhead, but for anything larger the nested distributed parallel computing is a win.

You can find pathos (and pyina -- the mpi4py portion of pathos) here: https://github.com/uqfoundation

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.
{"cc_re__cc_re__cc_by_sa_3_0": ["                  <li><a href=\"//area51.stackexchange.com\" title=\"proposing new sites in the Stack Exchange network\">Area 51</a></li>\r\n                        <li><a href=\"//careers.stackoverflow.com\">Stack Overflow Careers</a></li>\r\n                    \r\n                </ol>\r\n            </td>\r\n    </tr>\r\n</table>\r\n                </div>\r\n            </div>\r\n\r\n            <div id=\"copyright\">\r\n                site design / logo &#169; 2015 stack exchange inc; user contributions licensed under <a href=\"http://creativecommons.org/licenses/by-sa/3.0/\" rel=\"license\">cc by-sa 3.0</a> \r\n                with <a href=\"http://blog.stackoverflow.com/2009/06/attribution-required/\" rel=\"license\">attribution required</a>\r\n            </div>\r\n            <div id=\"svnrev\">\r\n                rev 2015.5.22.2603\r\n            </div>\r\n            \r\n        </div>\r\n    </div>\r\n    <noscript>\r\n        <div id=\"noscript-warning\">Stack Overflow works best with JavaScript enabled<img src=\"http://pixel.quantserve.com/pixel/p-c1rF4kxgLUzNc.gif\" alt=\"\" class=\"dno\"></div>\r\n    </nos"], "copyright__copyright__copyright": ["backend or backends you choose, so for very small tasks you see a good bit of the time as overhead, but for anything larger the nested distributed parallel computing is a win.\n\nYou can find pathos (and pyina -- the mpi4py portion of pathos) here: https://github.com/uqfoundation\n\nshare|improve this answer\n\nYour Answer\n\n\u00a0\ndiscard\n\nBy posting your answer, you agree to the privacy policy and terms of service.\n\nNot the answer you're looking for? Browse other questions tagged or ask your own question."]}