Take the 2-minute tour ×
Stack Overflow is a question and answer site for professional and enthusiast programmers. It's 100% free, no registration required.

Map Reduce is a pattern that seems to get a lot of traction lately and I start to see it manifest in one of my projects that is focused on an event processing pipeline (iPhone Accelerometer and GPS data). I needed to built a lot of infrastructure for this project, in fact it overweighs the logic code interacting with it by 2x. Some of the components I built where EventProcessors (with in- and outputbuffers, timing etc.), EventListeners, Aggregators and a staged Pipeline.

This leads me to my question what the "common" required infrastrucutre for map reduce is. Since I am working with .Net a lot I can see map reduce infrastructure built into the Framework and language constructs. Functional languages support this paradigm per se. It seems every language can be used with map reduce. There are even languages built around that concept (e.g. Go).

Apache Hadoop brings Map-Reduce to Java. Google has patented a map-reduce framework. What kind of infrastructure do they provide to enable map reduce? What are the constructs exhibited in functional languages to implement map reduce? What needs/should a map-reduce framework provide?

share|improve this question

3 Answers 3

up vote 2 down vote accepted

Well Hadoop is based on the Google File System. The Hadoop MapReduce implementation is also based on a paper by Google. For both Google and Hadoop the component that allows MapReduce to sucessfully run over massive amounts of data in parallel is the distributed file system.

share|improve this answer

As I understand it, Hadoop is generally based upon the HDFS and/or HBase infrastructure, which acts as the data distribution mechanism for Hadoop itself to operate on.

There's also Amazon Elastic MapReduce, which is a shiny web frontend which uses EC2 and Hadoop to make things easier. The "infrastructure" in this case is EC2 and S3.

P.S. Sorry for the snippy comment :)

share|improve this answer

Since you are used to working with .NET, you may want to look at DryadLINQ. http://research.microsoft.com/en-us/downloads/03960cab-bb92-4c5c-be23-ce51aee0792c/default.aspx

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.
{"cc_re__cc_re__cc_by_sa_3_0": ["                  <li><a href=\"//area51.stackexchange.com\" title=\"proposing new sites in the Stack Exchange network\">Area 51</a></li>\r\n                        <li><a href=\"//careers.stackoverflow.com\">Stack Overflow Careers</a></li>\r\n                    \r\n                </ol>\r\n            </td>\r\n    </tr>\r\n</table>\r\n                </div>\r\n            </div>\r\n\r\n            <div id=\"copyright\">\r\n                site design / logo &#169; 2015 stack exchange inc; user contributions licensed under <a href=\"http://creativecommons.org/licenses/by-sa/3.0/\" rel=\"license\">cc by-sa 3.0</a> \r\n                with <a href=\"http://blog.stackoverflow.com/2009/06/attribution-required/\" rel=\"license\">attribution required</a>\r\n            </div>\r\n            <div id=\"svnrev\">\r\n                rev 2015.3.3.2353\r\n            </div>\r\n            \r\n        </div>\r\n    </div>\r\n    <noscript>\r\n        <div id=\"noscript-warning\">Stack Overflow works best with JavaScript enabled<img src=\"http://pixel.quantserve.com/pixel/p-c1rF4kxgLUzNc.gif\" alt=\"\" class=\"dno\"></div>\r\n    </nosc"], "copyright__copyright__copyright": ["rastructure\" in this case is EC2 and S3.\n\nP.S. Sorry for the snippy comment :)\n\nshare|improve this answer\n\nSince you are used to working with .NET, you may want to look at DryadLINQ. http://research.microsoft.com/en-us/downloads/03960cab-bb92-4c5c-be23-ce51aee0792c/default.aspx\n\nshare|improve this answer\n\nYour Answer\n\n\u00a0\ndiscard\n\nBy posting your answer, you agree to the privacy policy and terms of service.\n\nNot the answer you're looking for? Browse other questions tagged or ask your own question."]}