I found the solution. It was an indexing issue with my labels.
My labels were starting from 1 to 8, I changed them to 0…7 and that fixed the issue for me.
Credit to this answer 1 on Stackoverflow.
Hope this will help someone in the future.