The imagelabels.mat is  an array of labels.
    label_list = [77 77 77 ... 62 62 62]

    <class 'numpy.ndarray'>

The sorted version of my file path is as follows
    image_path_list = sorted([os.path.join('data/images/', filename) for filename in os.listdir('data/images/')])

This gives
[‘data/images/image_00001.jpg’, ‘data/images/image_00002.jpg’, ‘data/images/image_00003.jpg’, ‘data/images/image_00004.jpg’…and so on]
The corresponding class indeed is 77. And it maps perfectly to Passion flower. A few other randomly tested map too.
Me being the very embodiment of stupidity did not sort my labels. And the order they were in since I not sorted them was totally arbitrary as os.listdir() doesn’t read files in the order they appear in the directory.
So, I ended up subtracting one from label_list with 77 becoming 76 and 73 72 and so on. Create a
new class_to_label map that starts from 0.
    {
"0": "pink primrose",
"1": "hard-leaved pocket orchid",
"2": "canterbury bells",
"3": "sweet pea",
"4": "english marigold",
"5": "tiger lily",
"6": "moon orchid",
"7": "bird of paradise",
"8": "monkshood",
"9": "globe thistle",
"10": "snapdragon",
"11": "colt's foot",
"12": "king protea",
 and so on...
 "76": "passion flower"

}
I now think I am good to go. Thanks a ton.