interleave the reading of files

You mean run the two reads in parallel?
I think even in your initial implementation TensorFlow is supposed to notice that the image and text branch are independent, and execute them in parallel.