Question: I am trying to make a function to help me debug. When I do the following:

```
s = traceback.format_stack()[-2]
print(s)

```

I get a console output something like:

```
File "/home/path/to/dir/filename.py", line 888, in function_name

```

How can I extract `filename.py`, `888` and `function_name` into separate strings? Using a regex?

Answer: 
Here is a working example of what you want to accomplish.

```
val rddMap = sc.parallelize(Map("A" -> ("v", "v"), "B" -> ("d","d")).toSeq)
// Collects all the data in the RDD and converts the data to a Map
val myMap = rddMap.collect().toMap
myMap.foreach(println)

```

Output: 

```
(A,(v,v))  
(B,(d,d))

```

Here is similar code to what you've posted

```
rddMap.map { t=> 
  println("t" + t)
  newHashMap.put(t._1, t._2)
  println(newHashMap.toString) 
}.collect

```

Here is the output to the above code from the Spark shell

```
t(A,(v,v))  
Map(A -> (v,v), test1 -> (10,20))  
t(B,(d,d))  
Map(test1 -> (10,20), B -> (d,d))

```

To me it looks like Spark copies your HashMap and does add the element to the **copied** map.
What you are trying to do is not really supported in Spark today. 

Note that every user defined function (e.g., what you add inside a `map()`) is a closure that gets serialized and pushed to each executioner. 

Therefore everything you have inside this `map()` gets serialized and gets transferred around:

```
.map{ t =>
  println(" t " + t)
  myHashMap.put(t._1, t._2)
}

```

Essentially, your `myHashMap` will be copied to each executioner and each executioner will be updating it's own version of that `HashMap`. This is why at the end of the execution the `myHashMap` you have in your driver will never get changed. (Driver is the JVM that manages/orchestrates your Spark jobs. It's the place where you define your SparkContext.) 

In order to push structures defined in the driver to all executioners you need to `broadcast` them (see link [here](https://spark.apache.org/docs/latest/programming-guide.html#broadcast-variables)). Note that broadcasted variables are read-only, so again, using broadcasts will not help you here.

Another way is to use `Accumulators` but I feel that these are more tune towards summarizing numeric values, like doing sum, max, min, etc. Maybe you can take a look at creating a custom accumulator that extends `AccumulatorParam`. See link [here](https://spark.apache.org/docs/latest/programming-guide.html#accumulators-a-nameaccumlinka). 

Coming back to the original question, if you want to collect values to your driver, currently the best way to do this is to transform your RDDs until they become a small and manageable collection of elements and then you `collect()` this final/small RDD.