title: Best method to add noise on tf.dataset images "on the fly"
body: <p>After training a model (image classification) I would like to see how it performs differently when I evaluate a proper image and various noised versions of it.</p>
  
  <p>The type of noise I'm thinking is a random change in pixels value, I tried with this approach:</p>
  
  <pre class="lang-py prettyprint-override"><code># --Inside the generator function that I provide to model.predict_generator--
  # dataset is a numpy array with denoised images path
  dt = tf.data.Dataset.from_generator(lambda: image_generator(dataset), output_types=(tf.float32))
  </code></pre>
  
  <pre class="lang-py prettyprint-override"><code>def image_generator_(image_paths):
      for path in image_paths:
          # im is keras.preprocessing image
          img = im.load_img(path,
                            color_mode='rgb',
                            target_size=(224,224))
  
          img_to_numpy = np.array(img)
  
          for _ in range (0, 5):
              tmp_numpy_image = img_to_numpy.copy()
              for i in range(tmp_numpy_image.shape[0]): 
                  for j in range(tmp_numpy_image.shape[1]):
                      # add noise
                      tmp_numpy_image.shape[i][j] = ...
              yield tmp_numpy_image
  </code></pre>
  
  <p>This process works fine but it is very slow. I also use dataset.batch and dataset.prefetch on <strong>dt</strong> and I didn't found a combination for their values that reduces the algorithm time</p>
  
  <p>Is there a smarter way to do it? I tried by yielding not noised images and to add the noise later inside dataset.map. The problem is that inside map I have to manipulate tensors and I didn't found a way to change each pixel value</p>
  
  <hr>
  
  <p><strong>SOLUTION</strong></p>
  
  <p>I used @Marat approach and it worked like a charm, the whole process went from 20-30 hours to minutes. My noise was a simple +-1 but I didn't want to go in overflow (255+1 = 0 in uint8) and therefore I only had to use numpy masks</p>
  
  <pre><code>...
  tmp_numpy_image = img_to_numpy.copy()
  
  noise = np.random.randint(-1, 1, img_to_numpy.shape)
  
  # tmp_numpy_imag will become of type int32
  tmp_numpy_image = tmp_numpy_image + noise
  
  np.putmask(tmp_numpy_image, tmp_numpy_image &lt; 0, 0)
  np.putmask(tmp_numpy_image, tmp_numpy_image &gt; 255, 255)
  
  tmp_numpy_image = tmp_numpy_image.astype('uint8') 
  
  yield tmp_numpy_image
  </code></pre>