@KFrank, thank you for all your comments. I certainly learnt a lot from the discussion. I have been able to make it work, but the performance could be improved upon. I think this would come in the form of hyperparameter tuning.
I really appreciate your effort in all this. It was not just a mere reply, rather a detailed and wonderful explanation of what my problem should be looked as. I am sure others who run into such problems would find your replies helpful. That said, I am able to do what I set out to do, so I am going to mark this thread as solved.
Thank you so much!