Created 09-06-2016 03:57 PM
Hi All,
Can some one please clarify how to create broadcast variable for a Dataframe.
Also help me with an example which would be great.
Thanks in advance.
Regards,
Vijay
Hi @Vijay Kumar J,
You can't create broadcast variable for a DataFrame. If you would like to do broadcast joins, however, if you are using Spark 1.5 or newer, you can still do that like following:
from pyspark.sql import SQLContext from pyspark.sql.functions import broadcast sqlContext = SQLContext(sc) df_tiny = sqlContext.sql('select * from tiny_table') df_large = sqlContext.sql('select * from massive_table') df3 = df_large.join(broadcast(df_tiny), df_large.some_sort_of_key == df_tiny.key)
By using broadcast function on your DataFrame, you will mark it as small enough for use in broadcast joins.)