Thanks! Setting keep_in_memory=True did indeed speed things up drastically. This change alone made my training loop 16% faster and my eval loop 25% faster, so that’s really cool to see.
I noticed too that load_metric has a keyword num_process - can I just set this number higher to easily do multiprocessing?