baseline performance measure for the various algorithms that are used to evaluate the different parallel programming runtimes. Table 1 summarizes the different characteristics of Hadoop, Dryad, CGL-MapReduce, and MPI. Comparison of features supported by different parallel programming runtimes. Feature Hadoop DryadLINQ MapReduce++ MPI Programming MapReduce DAG based execution MapReduce with a Variety of Model flows Combine phase topologies constructed the rich set of parallel constructs Data Handling HDFS Shared directories/ Shared file system / Shared file Local disks Local disks systems Intermediate HDFS/ Files/TCP pipes/ Content Distribution Low latency Data via Shared memory FIFO Network communication Communication HTTP (NaradaBrokering channels (Pallickara and Fox 2003) ) Scheduling Data locality/ Data locality/ Data locality Available Rack aware Network processing topology based run capabilities time graph optimizations Failure Persistence via HDFS Re-execution of Currently not Program level Handling Re-execution of map vertices implemented Check pointing and reduce tasks (Re-executing map OpenMPI , tasks, redundant FT MPI reduce tasks) Monitoring Monitoring support Monitoring support Programming interface Minimal of HDFS, Monitoring for execution graphs to monitor the progress support for task MapReduce of jobs level computations monitoring Language Implemented Programmable via C# Implemented using C, C++, Fortran, Support Java. Other DryadLINQ provides Java Java, C# languages are LINQ programming languages are supported via API for Dryad supported via Java Hadoop Streaming wrappers
baseline performance measure for the various algorithms that are used to evaluate the different parallel programming runtimes. Table 1 summarizes the different characteristics of Hadoop, Dryad, CGL-MapReduce, and MPI. Comparison of features supported by different parallel programming runtimes. Feature Hadoop DryadLINQ MapReduce++ MPI Programming Model MapReduce DAG based execution flows MapReduce with a Combine phase Variety of topologies constructed the rich set of parallel constructs Data Handling HDFS Shared directories/ Local disks Shared file system / Local disks Shared file systems Intermediate Data Communication HDFS/ via HTTP Files/TCP pipes/ Shared memory FIFO Content Distribution Network (NaradaBrokering (Pallickara and Fox 2003) ) Low latency communication channels Scheduling Data locality/ Rack aware Data locality/ Network topology based run time graph optimizations Data locality Available processing capabilities Failure Handling Persistence via HDFS Re-execution of map and reduce tasks Re-execution of vertices Currently not implemented (Re-executing map tasks, redundant reduce tasks) Program level Check pointing OpenMPI , FT MPI Monitoring Monitoring support of HDFS, Monitoring MapReduce computations Monitoring support for execution graphs Programming interface to monitor the progress of jobs Minimal support for task level monitoring Language Support Implemented Java. Other languages are supported via Hadoop Streaming Programmable via C# DryadLINQ provides LINQ programming API for Dryad Implemented using Java languages are supported via Java wrappers C, C++, Fortran, Java, C#
DryadReport_draft.docx