Question: I'm reading an article on Apache Spark and I came across the following sentence: 

"Hadoop as a big data processing technology has been around for 10 years and has proven to be the solution of choice for processing large data sets. MapReduce is a great solution for one-pass computations, but not very efficient for use cases that require multi-pass computations and algorithms." ([Full article](https://www.infoq.com/articles/apache-spark-introduction/))

Searching the web yields results about the difference between one-pass and multi-pass compilers (For instance, see [This SO question](https://stackoverflow.com/questions/35673818/difference-between-one-pass-and-multi-pass-compilers))

However, I'm not really sure if the answer also applies for data processing. Can somebody explain me what one-pass computation and multi-pass computation is, and why the latter is better, and thus is used in Spark?

Answer: 
Map Reduce
==========

[![enter image description here](https://i.stack.imgur.com/LEqzi.png)](https://i.stack.imgur.com/LEqzi.png)

Source : <https://www.guru99.com/introduction-to-mapreduce.html> 

Here you can see, the input file is processed as follows.

1. first split
2. goes into mapping phase
3. Shuffling
4. Reducer

In Map-reduce paradigm, after every stage the intermediate result is written to disk. Also, Mapper and Reducer are two different process. That is, first the mapper job runs, spits out the mapping files, then the reducer job starts. At every stage the job requires resource allocation. Therefore, a single map-reduce job required multiple iterations. If you have multiple map phases, after every map the data needs to spit out to disk before other map task starts. This is the **multi-step** process.

> 
> Each step in the data processing workflow has one Map phase and one Reduce phase and you'll need to convert any use case into MapReduce pattern to leverage this solution.
> 
> 
> 

Spark
=====

On the other hand, spark does the resource negotiation only once. Once the negotiation is completed, it spawns all the executors and that stays throughout the tenure of the job.
During the execution, spark doesn't write the intermediate output of the Map phases to the disk, rather keeps in memory. Therefore, all the map operations can happen back to back without writing to disk or spawning new executors. This is the single step process.

> 
> Spark allows programmers to develop complex, multi-step data pipelines using directed acyclic graph (DAG) pattern. It also supports in-memory data sharing across DAGs, so that different jobs can work with the same data.
> 
> 
>
One pass computations is when you are reading the dataset once whereas multipass computations is when a dataset is read once from the disk and multiple computations or operation are done on the same dataset. Apache Spark processing framework allows you to read data once which is then cached into memory and then we can perform multi pass computations on the data. These computations can be done on the dataset very quickly because the data is present into memory of the machine and apache spark does not need to read the data again from the disk which helps us to save lot of input output operations time. As per the definition of apache spark it is an in memory processing framework which means the data and transformation on which the computation is done is present in memory itself.