Are you loading images? Both storing and loading of images in uncompressed form (all values for all pixels are encoded) are very inefficient…
You can preprocess your images and save them in .jpg or .png format, then load them using PIL.Image.open(filename).
Note that some other parts of the training can bottleneck the execution, but as you use raw numpy arrays, I would start there and see if training is still slow after the modifications.