your problem could be in your file fer.py, in these lines:
 self.data = h5py.File('./data/data.h5', 'r', driver='core')
 if self.split == 'Training':
            self.train_data = self.data['Training_pixel']
            self.train_labels = self.data['Training_label']

self.train_data points into the .h5 file. Try wrapping it with a list constructor, or whatever data type is appropriate, to create a deep copy instead of a shallow reference:
 self.data = h5py.File('./data/data.h5', 'r', driver='core')
 if self.split == 'Training':
            self.train_data = list(self.data['Training_pixel'])
            self.train_labels = list(self.data['Training_label'])

Apply the same for your other datasets.