Answers.Body: <p>I am not allowed to comment, so I am adding it here. Not sure I understand, are you asking if the training data must be ordered?</p>
  
  <p>If so, training data must be random, and not AaBbCc or ABCabc.
  Here the input to the NN will be the source image, and a category 90 degrees or 270 degrees as too separate inputs. i.e. inputs [0...k][k+1][k+2], where 0 - k are normalised pixel data, k+1 is 1 for 90 degree rotation else 0, and k+2 is 1 for 270 degree rotation else 0. Alternatively for more rotation options lets make a = input [k+1] and b = input [k+2]. If a = 0 and b = 0 then 0 degrees, if a = 0 and b = 1 then 90 degrees, if a = 1 and b = 0 then 180 degrees if a = 1 and b = 1 then 270 degrees.</p>
  
  <p>From your image set, draw 3 samples. Sample 1 [training set] must be say 50% of the images, sample 2 [validation set] say 30% of the images and sample 3 [out of sample set] 20% of the images.</p>
  
  <p>Use the training images from sample 1 to update the weights. Read in images from sample 1 at random, the rotation must be random, use your image rotator to rotate the image, and compare it to the output of the neural network, calculate the sum of the mean error and perform gradient descent to update the weights. Repeat the process. After every x weight updates randomly select a few images from the validation set and calculate an error value. If the average error value of the validation images are above some threshold then stop, or if the error begins to get worse. Do not use the validation set to update the weights</p>
  
  <p>At the end use the out-of-sample set of images to verify the performance of the neural network. </p>
  
  <p>The process above does not include hyper-parameter optimisation, or any other optimisation techniques, but does describe the basic way of training a NN using supervised learning.</p>
  
  <p><strong>Edit:</strong> I assumed that you wanted to NN to learn to rotate the image. After your updates, I assume you want rotated images to increase generalisation of your NN. In which case randomising or shuffling the input batch is preferred. i.e. send your images to Keras to rotate them, get the batch back any order is fine ABab or AaBb and then shuffle them. Pass the shuffled training and validation set to the NN and compare it to the known label for weight updates.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-08T11:03:53.530Z
Answers.Text: Actually my question is about how should the image generator return the images. A, B are my images and I rotate them 90 and 270 degrees. My question is should the generator return a batch as [90, 90, 90, 270, 270, 270] deg or [90, 270, 90, 270, 90, 270] deg. I added the related part of the code. Thank you for your help.
Answers.UserDisplayName: jonathan eslava
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-09T13:16:14.360Z
Answers.Text: I understand. Passing in A, B into the generator can rotate the images in any way, thats not really that important. However to get the best results the batch of images both original and rotated should be shuffled. If you pass the images to the NN ordered during training ie AaBb or ABab it will effect learning.
Answers.UserDisplayName: Jason
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-07T20:00:46.963Z
Answers.LastActivityDate: 2018-12-09T13:21:19.303Z
Answers.LastEditDate: 2018-12-09T13:21:19.303Z
Answers.UserDisplayName: Jason
Body: <p>I am trying to generate 90 and 270 degrees rotated versions of my sample images on the fly during training. I found an example and modifying it. But I am confused about what should be the order? For instance in one batch I have 32 images and my image generator should return total 64 images. Let's say upper case letters are 90 degree and lower case letters are 270 degree rotated images. Should the order be AaBbCc or ABCabc? I apply the same to the validation set. Here is the related code fragment:</p>
  
  <p><strong>Edit:</strong> Code fragment added.</p>
  
  <pre><code>    def _get_batches_of_transformed_samples(self, index_array):
      # create list to hold the images
      batch_x = []
      # create list to hold the labels      
      batch_y = []
      # rotation angles
      target_angles = [0, 90, 180, 270]
      angle_categories = list(range(0, len(target_angles)))
      self.classes = target_angles
      self.class_indices = angle_categories
      # generate rotated images and corresponding labels
      for i, j in enumerate(index_array):           
          is_color = int(self.color_mode == 'rgb')
          image = cv2.imread(self.filenames[j], is_color)
          if is_color:
              image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)                               
  
          for rotation_angle, cat_angle in zip(target_angles, angle_categories):
              rotated_im = rotate(image, rotation_angle, self.target_size[:2])
              if self.preprocess_func: rotated_im = self.preprocess_func(rotated_im)
              # add dimension to account for the channels if the image is greyscale
              if rotated_im.ndim == 2: rotated_im = np.expand_dims(rotated_im, axis=2)                                
              batch_x.append(rotated_im)
              batch_y.append(cat_angle)
  
      # convert lists to numpy arrays
      batch_x=np.asarray(batch_x)
      batch_y=np.asarray(batch_y)
  
      batch_y = to_categorical(batch_y, len(target_angles))            
      return batch_x, batch_y
  </code></pre>
  
  <p>I actually rotate them as 0, 90, 180 and 270 degrees. As seen in the code, for each batch I return all the rotated versions of all the images in the batch. But is this correct or should I return first 0 degree rotated versions, second 90 degree rotated versions so on?</p>
  
  <p><strong>Edit2:</strong> I checked my previous work which I use the built in Keras <code>ImageDataGenerator</code>. <code>generator.classes</code> returns [zeros(100,1); ones(100,1)]. In that study I only have two classes. I understand that Keras indexes the images as [class1, class2, ...]. I think I have to do the same.</p>
  
Tags: <python><keras>
Title: Label arrangement for custom Keras image generator
UserDisplayName: jonathan eslava
ThreadText: # Label arrangement for custom Keras image generator
  
  (Asked by: username_0 on 2018-12-06)
  
  I am trying to generate 90 and 270 degrees rotated versions of my sample images on the fly during training. I found an example and modifying it. But I am confused about what should be the order? For instance in one batch I have 32 images and my image generator should return total 64 images. Let's say upper case letters are 90 degree and lower case letters are 270 degree rotated images. Should the order be AaBbCc or ABCabc? I apply the same to the validation set. Here is the related code fragment:
  
  **Edit:** Code fragment added.
  
  ```
      def _get_batches_of_transformed_samples(self, index_array):
      # create list to hold the images
      batch_x = []
      # create list to hold the labels      
      batch_y = []
      # rotation angles
      target_angles = [0, 90, 180, 270]
      angle_categories = list(range(0, len(target_angles)))
      self.classes = target_angles
      self.class_indices = angle_categories
      # generate rotated images and corresponding labels
      for i, j in enumerate(index_array):           
          is_color = int(self.color_mode == 'rgb')
          image = cv2.imread(self.filenames[j], is_color)
          if is_color:
              image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)                               
  
          for rotation_angle, cat_angle in zip(target_angles, angle_categories):
              rotated_im = rotate(image, rotation_angle, self.target_size[:2])
              if self.preprocess_func: rotated_im = self.preprocess_func(rotated_im)
              # add dimension to account for the channels if the image is greyscale
              if rotated_im.ndim == 2: rotated_im = np.expand_dims(rotated_im, axis=2)                                
              batch_x.append(rotated_im)
              batch_y.append(cat_angle)
  
      # convert lists to numpy arrays
      batch_x=np.asarray(batch_x)
      batch_y=np.asarray(batch_y)
  
      batch_y = to_categorical(batch_y, len(target_angles))            
      return batch_x, batch_y
  
  ```
  
  I actually rotate them as 0, 90, 180 and 270 degrees. As seen in the code, for each batch I return all the rotated versions of all the images in the batch. But is this correct or should I return first 0 degree rotated versions, second 90 degree rotated versions so on?
  
  **Edit2:** I checked my previous work which I use the built in Keras `ImageDataGenerator`. `generator.classes` returns [zeros(100,1); ones(100,1)]. In that study I only have two classes. I understand that Keras indexes the images as [class1, class2, ...]. I think I have to do the same.
  
  **Tags:** python, keras
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-12-07. Score: 0 | Accepted answer
  
  I am not allowed to comment, so I am adding it here. Not sure I understand, are you asking if the training data must be ordered?
  
  If so, training data must be random, and not AaBbCc or ABCabc.
  Here the input to the NN will be the source image, and a category 90 degrees or 270 degrees as too separate inputs. i.e. inputs [0...k][k+1][k+2], where 0 - k are normalised pixel data, k+1 is 1 for 90 degree rotation else 0, and k+2 is 1 for 270 degree rotation else 0. Alternatively for more rotation options lets make a = input [k+1] and b = input [k+2]. If a = 0 and b = 0 then 0 degrees, if a = 0 and b = 1 then 90 degrees, if a = 1 and b = 0 then 180 degrees if a = 1 and b = 1 then 270 degrees.
  
  From your image set, draw 3 samples. Sample 1 [training set] must be say 50% of the images, sample 2 [validation set] say 30% of the images and sample 3 [out of sample set] 20% of the images.
  
  Use the training images from sample 1 to update the weights. Read in images from sample 1 at random, the rotation must be random, use your image rotator to rotate the image, and compare it to the output of the neural network, calculate the sum of the mean error and perform gradient descent to update the weights. Repeat the process. After every x weight updates randomly select a few images from the validation set and calculate an error value. If the average error value of the validation images are above some threshold then stop, or if the error begins to get worse. Do not use the validation set to update the weights
  
  At the end use the out-of-sample set of images to verify the performance of the neural network.
  
  The process above does not include hyper-parameter optimisation, or any other optimisation techniques, but does describe the basic way of training a NN using supervised learning.
  
  **Edit:** I assumed that you wanted to NN to learn to rotate the image. After your updates, I assume you want rotated images to increase generalisation of your NN. In which case randomising or shuffling the input batch is preferred. i.e. send your images to Keras to rotate them, get the batch back any order is fine ABab or AaBb and then shuffle them. Pass the shuffled training and validation set to the NN and compare it to the known label for weight updates.
  
  > **Comments:**
  > **username_0**: Actually my question is about how should the image generator return the images. A, B are my images and I rotate them 90 and 270 degrees. My question is should the generator return a batch as [90, 90, 90, 270, 270, 270] deg or [90, 270, 90, 270, 90, 270] deg. I added the related part of the code. Thank you for your help.  
  > **username_1**: I understand. Passing in A, B into the generator can rotate the images in any way, thats not really that important. However to get the best results the batch of images both original and rotated should be shuffled. If you pass the images to the NN ordered during training ie AaBb or ABab it will effect learning.  
  
  ---
  
  