<p dir="auto"><code class="notranslate">fit_generator()</code> seems to keep consuming memory throughout an epoch if objects defined inside the generator class are not converted into a numpy array. RAM keeps on getting filled until an epoch ends. For large enough datasets, this leads to RAM completely filling up before a single epoch is completed.</p>
<p dir="auto">Is it possible that the objects keep getting duplicated?</p>
<p dir="auto">Python = 3.6.6<br>
Keras = 2.2.4<br>
Tensorflow = 1.12.0</p>
<p dir="auto"><a href="https://gist.github.com/Punyaslok/5a3dd01bd004ba84ae37329799d47430">Here is a sample script to reproduce the issue.</a></p>
<p dir="auto">A major use-case for the scikit-learn API is automating hyperparameter opt. With the current API we need to either:</p>
<ol dir="auto">
<li>Search through <code class="notranslate">nb_epochs</code>, duplicating most of the computation (essentially every <code class="notranslate">nb_epoch</code> except the largest is redundant).</li>
<li>Implement getting the best epoch ourselves, possibly using <code class="notranslate">EarlyStopping</code> or the like.</li>
</ol>
<p dir="auto">Would it be possible to just have <code class="notranslate">KerasClassifier.fit()</code> return the best epoch?</p>