title: Generator function in LSTM Keras for outputting mini batches of one files
body: <p>I have a generator function which works fine. I have a large list of .txt files in which each file is also quite long. The task would now be to write a generator function which Takes:</p>
  
  <ol>
  <li>a batch of Files</li>
  <li>and then a batch of size 128 out of one file</li>
  </ol>
  
  <p>my code now:</p>
  
  <pre><code>data_files_generator &lt;- function(train_set) {
  
    files &lt;- train_set
    next_file &lt;- 0
  
    function() {
  
      # move to the next file (note the &lt;&lt;- assignment operator)
      next_file &lt;&lt;- next_file + 1
  
      # if we've exhausted all of the files then start again at the
      # beginning of the list (keras generators need to yield
      # data infinitely -- termination is controlled by the epochs
      # and steps_per_epoch arguments to fit_generator())
      if (next_file &gt; length(files))
      {next_file &lt;&lt;- 1}
  
      # determine the file name
      file &lt;- files[[next_file]]
  
      text &lt;- read_lines(paste(data_dir, file, sep = "" )) %&gt;%
        str_to_lower() %&gt;%
        str_c(collapse = "\n") %&gt;%
        removeNumbers() %&gt;%
        tokenize_characters(strip_non_alphanum = FALSE, simplify = TRUE)
  
      text &lt;- text[text %in% chars]
  
      dataset &lt;- map(
        seq(1, length(text) - maxlen - 1, by = 3), 
        ~list(sentece = text[.x:(.x + maxlen - 1)], next_char = text[.x + maxlen])
      )
  
      dataset &lt;- transpose(dataset)
  
      # Vectorization
      x &lt;- array(0, dim = c(length(dataset$sentece), maxlen, length(chars)))
      y &lt;- array(0, dim = c(length(dataset$sentece), length(chars)))
  
      for(i in 1:length(dataset$sentece)){
  
        x[i,,] &lt;- sapply(chars, function(x){
          as.integer(x == dataset$sentece[[i]])
        })
  
        y[i,] &lt;- as.integer(chars == dataset$next_char[[i]])
  
      }
      rounded_dim &lt;- floor(dim(x)[1]/mini_batch_size)
      match_size_to_batch &lt;- 128 * rounded_dim
  
      x &lt;- x[1:match_size_to_batch, 1:maxlen, 1:length(chars)]
      y &lt;- y_val[1:match_size_to_batch, 1:length(chars)]
  
      return(list(x, y))
  
    }
  }
  </code></pre>
  
  <p>So what is coming is coming in is a Text file which is transformed into smaller pieces of text (of length <code>maxlen</code>) and is then one hot encoded into 0 and 1 matrices. </p>
  
  <p>The problem is that from my code the output is one Data Cube of size <code>maxlen x lenght(chars) x samples</code> where the number of samples is very big and that why I would like my generator function to output always a cube of size <code>maxlen x lenght(chars) x samples(128)</code> and then output the next batch of size <code>maxlen x lenght(chars) x samples</code> until the whole text file is read in and then go to the next text file...</p>
  
  <p>The output for now is an error:</p>
  
  <pre><code> Error in py_call_impl(callable, dots$args, dots$keywords) : 
    ValueError: Cannot feed value of shape (112512, 40, 43) for Tensor 'lstm_layer_input_1:0', which has shape '(128, 40, 43)' 
  </code></pre>
  
  <p>hope I have explained it good enough to understand. I think I have to input some kind of for loop to iterate over the sample length but I have no Idea how to include this into the gen. function.</p>