As per the source code for ConcatDataSet 1, your datasets object has an attribute called cumulative_sizes that contains information which can help you figure this out. Try querying cumulative_sizes and see if it gives you something useful.