In Memory Parallel Processing of Massive Remotely Sensed Data Using an Apache Spark on Hadoop YARN Model
MapReduce has been widely used in Hadoop for parallel processing larger scale data for the last decade. However, remote sensing (RS) algorithms based on the programming model are trapped in dense disk I/O operations and unconstrained network communication, and thus inappropriate for timely processing and analyzing massive, heterogeneous RS data. In this paper, a novel in memory computing framework called Apache Spark (Spark) is introduced. Through its merits of transferring transformation to in memory datasets of Spark, the shortages are eliminated. To facilitate implementation and assure high performance of Spark based algorithms in a complex cloud computing environment, a strip oriented parallel programming model is proposed. By incorporating strips of RS data with resilient distributed datasets (RDDs) of Spark, all level parallel RS algorithms can be easily expressed with coarse grained transformation primitives and BitTorrent enabled broadcast variables. Additionally, a generic image partition method for Spark based RS algorithms to efficiently generate differentiable key/value strips from a Hadoop distributed file system (HDFS) is implemented for concealing the heterogeneousness of RS data. Data intensive multitasking algorithms and iteration intensive algorithms were evaluated on a Hadoop yet another resource negotiator (YARN) platform. Experiments indicated that our Spark based parallel algorithms are of great efficiency, a multitasking algorithm took less than 4 h to process more than half a terabyte of RS data on a small YARN cluster, and 9*9 convolution operations against a 909 MB image took less than 260 s. Further, the efficiency of iteration intensive algorithms is insensitive to image size.
IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing