The above question is not important to me. The following one is what suspect as an IO bottleneck in the torch.load when loading pickle file
 if f_should_read_directly and f.tell() == 0:
        # legacy_load requires that f has fileno()
        # only if offset is zero we can attempt the legacy tar file loader
        try:
            return legacy_load(f)
        except tarfile.TarError:
            # if not a tarfile, reset file offset and proceed
            f.seek(0)

The legacy_load is to open the file with taropen, but in case of not a tar file, this path seems costly, and some non-necessary IO operations are triggered.
adding a condition check seems helpful:
 if f_should_read_directly and f.tell() == 0 and tarfile.is_tarfile(fn): 

but using is_tarfile requires _load to accept a new parameter, which filename: fn.
Check my benchmark: https://github.com/NERSC/pyprob/blob/distributed/torch_load_bench.py 8