Hi guys!
Please consider this idea 368 I just came up with.
The ideas is:

have multiple heads in your CNN and some if-logic in forward statement
have a cluster_index for each different type of image
feed only consistent batches, but randomly (batch 1 - size 1, batch 2 - size 2, batch 3 - size 1, etc)