Cache datasets pre-processing
bodokaiser
apaszke
jekbradbury
Is there maybe something like a cache proxy which just wraps a custom python class and caches each calls?
bodokaiser
apaszke
Yep. That’s what I was looking for!
bodokaiser
EKami
joblib does an amazing job in caching to disk. This removes a lot of hassle and works great for me: <SCODE>from joblib import Memory
cachedir = '/data/cache/'
memory = Memory(cachedir, verbose=0, compress=True)

@memory.cache
def preprocess_file(file, ..params ):
    ... load data and do time consuming pre=processing
    sample['data'] = data
    sample['labels'] = labels
    return sample


class MyData(Dataset):
    def __init__(self, data_path, ... parameters):
        self.files  = list of files

    def __getitem__(self, index):
        sample = preprocess_file(self.files[index],  parameters....)     
        return sample
<ECODE>
gonzales2010
Amirhj