Apache Spark, Data, Ab Testing, Hbase, Memsql.
data increases, we wanted to reduce our storage footprint. We use parquet format (which is native to Spark) to save costs and space. Results With our new pipeline, we now see a huge time-savings in job execution. On an average, we gain at least three hours (as shown in the figure below), so