title: Training data dimensions for semantic segmentation using CNN
body: <p>I encountered many hardships when trying to fit a CNN (U-Net) to my tif training images in Python.</p>
  <p>I have the following structure to my data:</p>
  <ul>
  <li>X</li>
  <li>
  <ul>
  <li>0</li>
  </ul>
  </li>
  <li>
  <ul>
  <li>
  <ul>
  <li>[Images] (tif, 3-band, 128x128, values ∈ [0, 255])</li>
  </ul>
  </li>
  </ul>
  </li>
  <li>X_val</li>
  <li>
  <ul>
  <li>0</li>
  </ul>
  </li>
  <li>
  <ul>
  <li>
  <ul>
  <li>[Images] (tif, 3-band, 128x128, values ∈ [0, 255])</li>
  </ul>
  </li>
  </ul>
  </li>
  <li>y</li>
  <li>
  <ul>
  <li>0</li>
  </ul>
  </li>
  <li>
  <ul>
  <li>
  <ul>
  <li>[Images] (tif, 1-band, 128x128, values ∈ [0, 255])</li>
  </ul>
  </li>
  </ul>
  </li>
  <li>y_val</li>
  <li>
  <ul>
  <li>0</li>
  </ul>
  </li>
  <li>
  <ul>
  <li>
  <ul>
  <li>[Images] (tif, 1-band, 128x128, values ∈ [0, 255])</li>
  </ul>
  </li>
  </ul>
  </li>
  </ul>
  <p>Starting with this data, I defined ImageDataGenerators:</p>
  <pre><code>import tensorflow as tf
  from tensorflow import keras as ks
  
      from matplotlib import pyplot as plt
      import numpy as np
      
      bs = 10 # batch size
      args_col = {&quot;data_format&quot; : &quot;channels_last&quot;,
                  &quot;brightness_range&quot; : [0.5, 1.5]
                  }
      args_aug = {&quot;rotation_range&quot; : 365,
                  &quot;width_shift_range&quot; : 0.05,
                  &quot;height_shift_range&quot; : 0.05,
                  &quot;horizontal_flip&quot; : True,
                  &quot;vertical_flip&quot; : True,
                  &quot;fill_mode&quot; : &quot;constant&quot;,
                  &quot;featurewise_std_normalization&quot; : False,
                  &quot;featurewise_center&quot; : False
                  }
      args_flow = {&quot;color_mode&quot; : &quot;rgb&quot;,
                   &quot;class_mode&quot; : &quot;sparse&quot;,
                   &quot;batch_size&quot; : bs,
                   &quot;target_size&quot; : (128, 128),
                   &quot;seed&quot; : 42
                   }
      # train generator
      X_generator = ks.preprocessing.image.ImageDataGenerator(rescale = 1.0/255.0,
                                                              **args_aug,
                                                              **args_col)
      X_gen = X_generator.flow_from_directory(directory = &quot;my/directory/X&quot;,
                               **args_flow)
      
      y_generator = ks.preprocessing.image.ImageDataGenerator(**args_aug,
                                                              cval = NoDataValue)
      y_gen = y_generator.flow_from_directory(directory = &quot;my/directory/y&quot;,
                               **args_flow, color_mode = &quot;grayscale&quot;)
      
      train_generator = zip(X_gen, y_gen)
      # val generator
      X_val_generator = ks.preprocessing.image.ImageDataGenerator(rescale = 1.0/255.0)
      X_val_gen = X_generator.flow_from_directory(directory = &quot;my/directory/X_val&quot;),
                                   **args_flow)
      
      y_val_generator = ks.preprocessing.image.ImageDataGenerator()
      y_val_gen = y_generator.flow_from_directory(directory = &quot;my/directory/y_val&quot;),
                                   **args_flow, color_mode = &quot;grayscale&quot;)
      
      val_generator = zip(X_val_gen, y_val_gen)
  
  </code></pre>
  <p>Using this generator, I can create pairs of training images and corresponding masks and visualize them like this:</p>
  <pre><code>    X, y = next(train_generator)
      X_test = X[0][0]
      y_test = y[0][0]
      
      plt.subplot(1, 2, 1)
      plt.imshow(np.array(X_test))
      plt.subplot(1, 2, 2)
      plt.imshow(np.array(y_test))
  
  </code></pre>
  <p>Resulting in:</p>
  <p><a href="https://i.stack.imgur.com/lq38V.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/lq38V.png" alt="Visualization of ImageDataGenerator output" /></a></p>
  <p>However, I cannot train a U-Net, as I intended:</p>
  <p>When I define a U-Net based on an <a href="https://github.com/bnsreenu/python_for_microscopists/blob/master/074-Defining%20U-net%20in%20Python%20using%20Keras.py" rel="nofollow noreferrer">example</a> from the internet (or basically any other example of a U-Net I've found) as <code>model</code> and then do the following:</p>
  <pre><code>model.compile(optimizer = &quot;adam&quot;, loss = &quot;sparse_categorical_crossentropy&quot;, metrics = [&quot;accuracy&quot;])
  model.fit(train_generator, epochs = 5, steps_per_epoch = 10, validation_data = val_generator)
  </code></pre>
  <p>it will fail with the error:</p>
  <pre><code>ValueError: Layer model expects 1 input(s), but it received 2 input tensors. Inputs received: [&lt;tf.Tensor 'IteratorGetNext:0' shape=(None, None, None, None) dtype=float32&gt;, &lt;tf.Tensor 'ExpandDims:0' shape=(None, 1) dtype=float32&gt;]
  </code></pre>
  <p>I tried other loss functions and other <code>class_mode</code> arguments, but it always failed with some error related to the dimensions of the input data or the data passed between layers. An other example (when setting <code>class_mode = None</code>:</p>
  <pre><code>InvalidArgumentError:  logits and labels must have the same first dimension, got logits shape [16384,1] and labels shape [49152]
  </code></pre>
  <p>I just started getting into CNNs and Python, so I have no clue what to try further or how to resolve those errors. I was pretty sure I use the correct loss function, which seems to be often the problem when similar errors occur (I have multiple classes, hence the <code>&quot;sparse_categorical_crossentropy&quot;</code>).</p>
  <p>Any ideas how to solve this and make the data fit the expected CNN input (or the other way round, depending on what the problem is)?</p>
  <p><strong>Note:</strong>
  My <code>ImageDataGenerator</code> outputs a pair of images (X and y) with the following format (I noticed I had to set <code>color_mode</code> to &quot;grayscale&quot; for the masks (y)):</p>
  <p><a href="https://i.stack.imgur.com/Apoxf.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Apoxf.png" alt="Screenshot of training data format" /></a></p>
  <p>I used <code>keras.layers.Input(shape = (128, 128, 3))</code> in the example U-Net, since the <a href="https://www.tensorflow.org/api_docs/python/tf/keras/Input" rel="nofollow noreferrer">keras documentation</a> states shape = &quot;A shape tuple (integers), not including the batch size&quot;.</p>