Question: I'm working on an optimization problem that involves minimizing an expensive map operation over a collection of objects.

The naive solution would be something like

```
rdd.map(expensive).min()

```

However, the map function returns values that guaranteed to be >= 0. So, if any single result is 0, I can take that as the answer and do not need to compute the rest of the map operations. 

Is there an idiomatic way to do this using Spark?

Answer: 
> 
> Is there an idiomatic way to do this using Spark?
> 
> 
> 

No. If you're concerned with low level optimizations like this one, then Spark is not the best option. It doesn't mean it is completely impossible.

If you can for example try something like this:

```
rdd.cache()
(min_value, ) =  rdd.filter(lambda x: x == 0).take(1) or [rdd.min()]
rdd.unpersist()

```

short circuit partitions:

```
def min_part(xs):
    min_ = None
    for x in xs:
        min_ = min(x, min_) if min_ is not None else x
        if x == 0:
            return [0]
    return [min_] in min_ is not None else []

rdd.mapPartitions(min_part).min()

```

Both will usually execute more than required, each giving slightly different performance profile, but can skip evaluating some records. With rare zeros the first one might be better.

You can even listen to accumulator updates and use `sc.cancelJobGroup` once 0 is seen. Here is one example of similar approach [Is there a way to stream results to driver without waiting for all partitions to complete execution?](https://stackoverflow.com/q/41810032/8371915)
If "expensive" is **really** expensive, maybe you can write the result of "expensive" to, say, SQL (Or any other storage available to all the workers).
Then in the beginning of "expensive" check the number currently stored, if it is zero return zero from "expensive" without performing the expensive part.

You can also do this localy for each worker which will save you a lot of time but won't be as "global".