Thank you very much in advance for your help.
I want to train an image classifying NN but am running into the following memory Error:
RuntimeError: [enforce fail at CPUAllocator.cpp:64] . DefaultCPUAllocator: can't allocate memory: you tried to allocate 123681636352 bytes. Error code 12 (Cannot allocate memory)
I found this error strange because I am running on an ml.t2.xlarge SageMaker instance (lots of memory). Here’s a bit of context on the code:
I wrote a python script to:
i) load a file list (csv file) into the dataloader
ii) convert input images from TIFF to Numpy to Torch
iii) permute channels (H, W, C vs C, H, W)
iv) crop picture into smaller picture to avoid memory problems (dataset has 380GB of images, each .TIFF images is about ~25MB)
Here is the python script:
class load_csv(Dataset):
    def __init__(self, csv_file, root_dir, transform=None):
        self.annotations = pd.read_csv(csv_file)
        self.root_dir = root_dir
        self.transform = transform
    
    def __len__(self):
        return len(self.annotations)
    
    def __getitem__(self, index):
        img_path = os.path.join(self.root_dir, self.annotations.iloc[index, 0])
        image = torch.from_numpy(tiff.imread(img_path)).permute(2,0,1).float()
        #Image.MAX_IMAGE_PIXELS = None
                
       image.transform = transforms.RandomResizedCrop(224)
        
        y_label = torch.tensor(int(self.annotations.iloc[index, 1]))
        
        #if self.transform:
        #    image = self.transform(image)
        
        return (image, y_label)

Now, I’m running into a memory-related problem that I don’t know how to resolve. I am running a ml.t2.xlarge SageMaker notebook with plenty of memory and images being loaded into the dataloader have supposedly undergone a “transforms.RandomResizedCrop(224)” transformation and should be smaller.
Why am I running into this problem?
Please find project repo with CNN.ipynb and python script csv_loader.py where getitem class is defined for dataloader (here).
Thank you very much again,
Edwin