Based on your description it seems you have multiple json files, where each file might contain the same class indices as keys with the corresponding image samples.
If that’s the case, indexing into these file would work using the class label, which would have the drawback of a possibly complicated calculation of the sample index.
You could iterate your json files once and store each image in a subfolder corresponding to the class index. Using this approach you could simply get all the image paths and index them directly to lazily load the data inside your Dataset.