Pyspark, Mongodb, AWS, Machine Learning.
dataframes train_full_df = train_year_rdd.map(lambda d: getYearFeatures(d)).\ map(lambda d: getDFeatures(d)).toDF() test_df = test_year_rdd.map(lambda d: getYearFeatures(d)).\ map(lambda d: getDFeatures(d, True)).toDF() Create a training and validation set: # split to train and validation