Sports prediction using Keras NN stuck at ~0.5 accuracy
I'm currently in the process of getting into data science, ML, and especially neural networks (coming from a "pure" software engineering background/degree). I did some models on classic datasets like MNIST and MNIST fashion that worked great and wanted to tackle a more difficult kind of problem next as a challenge: sports/soccer prediction. I've found some papers where this was already tried using deep neural networks with results of up to ~0.65 to 0.7 accuracy. This is a level I'd like to reach as well. However, I cannot get my model above around 0.507 accuracy. Here's what I did:
I have datasets of several thousand fixtures in leagues around the world (Premier League, Bundesliga, etc.) from the past 10 seasons. 458
As input data, I'm using the following: - Weekday of the game - Season (year) - Round no - League id (one hot encoded) - Night match (yes/no) - Venue id (one hot encoded) - Home & Away team id (one hot encoded) - Home & Away team season points
As you can see, some of my data is one hot encoded. All the data points (including the data that is not one hot encoded) are placed into np arrays of the same size. So all the data is padded with zeroes until they are of the same size to guarantee the same input shape for every dataset of (10, 480). As I add more datasets, the last number grows bigger as there are more team ids, venue ids etc, so the needed array size/padding grows. I'm not entirely sure if this is the best solution, but as far as I understood, neural networks do depend on the input being the same shape, so this was the best solution in my eyes.
The model is then built like this: ```python def build_model(input_shape, dropout=0.2, hidden_layers=1, hidden_sizes=[64], learning_rate=0.01, optimizer="adam", last_layer_activation="softmax"): if optimizer == 'adam': optimizer = tf.keras.optimizers.Adam(learning_rate=learning_rate) elif optimizer == 'adagrad': optimizer = tf.keras.optimizers.Adagrad(learning_rate=learning_rate) elif optimizer == 'sgd': optimizer = tf.keras.optimizers.SGD(learning_rate=learning_rate) elif optimizer == 'rmsprop': optimizer = tf.keras.optimizers.RMSprop(learning_rate=learning_rate) elif optimizer == 'adadelta': optimizer = tf.keras.optimizers.Adadelta(learning_rate=learning_rate) elif optimizer == 'adamax': optimizer = tf.keras.optimizers.Adamax(learning_rate=learning_rate) else: raise ValueError(f"Unknown optimizer {optimizer}")
model = tf.keras.models.Sequential() model.add(tf.keras.layers.Flatten(input_shape=input_shape)) model.add(tf.keras.layers.Dropout(dropout)) for i in range(hidden_layers): model.add(tf.keras.layers.Dense(hidden_sizes[i], activation='relu')) model.add(tf.keras.layers.Dense(3, activation=last_layer_activation))
model.compile(optimizer=optimizer, loss='categorical_crossentropy', metrics=['accuracy']) return model ```
The data is then split up intro training & validation data with a 80:20 split and trained using model.fit(x_train, y_train, epochs=100, callbacks=callbacks, batch_size=64, validation_data=(x_val, y_val)) with an EarlyStop. I have also tried bigger batch sizes.
The model then outputs an one hot encoded array: [Home Win, Draw, Away Win].
I have tried tuning the model's hyperparameters (seen in the function parameters) with KerasTuner to no success. I cannot get above accuracy of about 0.507 max.
Also, my validation accuracy, as well as loss, is very noisy as you can see from this plot (where I've tried training for a little longer than 100 epochs):
I would expect some noise (around the level of the training data) - due to the randomness factor of sports - but not this much noise. 458