<p>Small set for multiple epochs causes overfit, so expose entire data in one epoch, then augment (change a littel bit), and train again. Use small amount of dropout also.</p>

<pre><code>I have a big dataset (300000 of images after augumentation) and I've splitted it into 10 parts, because I can't convert the images into a numpy and save it, the file would be too large.
</code></pre>

<p>Do not store augmented data, just do augment with function right after next epoch and overwrite the dataset array</p>

CC BY-SA 4.0
2019-07-16T12:30:35.320Z
2019-07-16T12:30:35.320Z
user8426627
<p>I don't know if I worded the title correctly.</p>

<p>I have a big dataset (300000 of images after augumentation) and I've splitted it into 10 parts, because I can't convert the images into a numpy and save it, the file would be too large.</p>

<p>Now, I have a neural network (Using keras with tf). My question is, is it better to train each file individually for X epochs (File 1 for 5 epochs, then File 2 for 5 epochs, etc.), or should I do an epoch for each, repeatedly (File 1 for an epoch, File 2 for an epoch, etc., and repeat for 5 times).</p>

<p>I've used the first and I get an accuracy of about 88%. Whould I get an improvement by doing the latter?</p>

<neural-networks><image-recognition><python>
Better to learn the same small set for multiple epochs then go to the next or learn from each one time repeatedly for multiple times?
alo98
# Better to learn the same small set for multiple epochs then go to the next or learn from each one time repeatedly for multiple times?

(Asked by: username_0 on 2019-07-16)

I don't know if I worded the title correctly.

I have a big dataset (300000 of images after augumentation) and I've splitted it into 10 parts, because I can't convert the images into a numpy and save it, the file would be too large.

Now, I have a neural network (Using keras with tf). My question is, is it better to train each file individually for X epochs (File 1 for 5 epochs, then File 2 for 5 epochs, etc.), or should I do an epoch for each, repeatedly (File 1 for an epoch, File 2 for an epoch, etc., and repeat for 5 times).

I've used the first and I get an accuracy of about 88%. Whould I get an improvement by doing the latter?

**Tags:** neural-networks, image-recognition, python


---

## 1 answers

### Answer by username_1 on 2019-07-16. Score: 1 | Accepted answer

Small set for multiple epochs causes overfit, so expose entire data in one epoch, then augment (change a littel bit), and train again. Use small amount of dropout also.
I have a big dataset (300000 of images after augumentation) and I've splitted it into 10 parts, because I can't convert the images into a numpy and save it, the file would be too large.

Do not store augmented data, just do augment with function right after next epoch and overwrite the dataset array


---

