Hey Mac,
I have check the link you sent to me. However, it seems this problem is not addressed in that post. People still complain about the efficiency.
Having both TF and Pytorch in the same script is a little bit weird, and I am afraid that there might be some efficiency problem if we load data by TF and then convert it to Pytorch Tensor for Dataloader. So I think converting tfrecord to arvo and read avro to the dataloader is really a good idea.
Could you please show me a sketch about how to convert tf data to arvo and load them to pytorch tensor? Thanks very much! I really appreciate it.
Regards,
Yin