<p dir="auto">How do I prevent the untar from the above get_file() call from creating duplicate "._*" files?</p>
<p dir="auto">I get double the images in the dataset when I run the following, and the duplicated images have a prefix of ._</p>
<p dir="auto">import pathlib<br>
import matplotlib.pyplot as plt<br>
import numpy as np<br>
import PIL<br>
import tensorflow as tf</p>
<p dir="auto">from tensorflow import keras<br>
from tensorflow.keras import layers<br>
from tensorflow.keras.models import Sequential</p>
<h1 dir="auto">custom data attempt</h1>
<p dir="auto">weeds_dataset_url = "<a href="https://storage.googleapis.com/aquarium-public/datasets/deepweeds/full_image_set/weed_images.tar" rel="nofollow">https://storage.googleapis.com/aquarium-public/datasets/deepweeds/full_image_set/weed_images.tar</a>"<br>
weeds_data_dir = tf.keras.utils.get_file('weed_images', origin=weeds_dataset_url, untar=True)<br>
weeds_data_dir = pathlib.Path(weeds_data_dir)</p>
<p dir="auto">snake_weeds = list(weeds_data_dir.glob('Snake weed/*'))<br>
print(snake_weeds[0]) # gives me a file starting with ._</p>
<p dir="auto">I'm using a data generator to feed the <code class="notranslate">fit_generator</code>. My generator have as output the tuple <code class="notranslate">(x_val, y_val, val_sample_weights)</code> so showing sample weights. This is like:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import numpy as np
import keras
import librosa
from time import time
import random
from config import *

class DataGenerator(keras.utils.Sequence):

    'Generates data for Keras'

    def __init__(self, dataframe, batch_size=None, dim=None, labels_dim=None,
                 n_classes=None, shuffle=True, samples=None, duration=None, sample_weights=None):
        'Initialization'
        self.dim = dim
        self.batch_size = batch_size

        self.dataframe = dataframe
        self.dataframe = self.dataframe.sample(n=len(self.dataframe))
        self.samples = samples
        self.on_epoch_end()
        self.shuffle = shuffle
        self.sample_weights = sample_weights

    def __len__(self):
        'Denotes the number of batches per epoch'
        return int(np.floor(len(self.dataframe) / self.batch_size))

    def __getitem__(self, index):
        'Generate one batch of data'
        random_pd = self.dataframe.iloc[self.batch_size*index : (index+1)*self.batch_size]
        # Generate data
        X, y = self.__data_generation(random_pd)
        return X, y
    def __data_generation(self, random_pd):
        'Generates data containing batch_size samples' # X : (n_samples, *dim, n_channels)
        # Initialization
        X = np.empty((self.batch_size, 1, self.samples))
        y = np.empty((self.batch_size, self.n_classes))
        i = 0
        while i &lt; self.batch_size:
            for index, row in random_pd.iterrows():
                 # generate 
                y[i,] = label
                X[i,] = ...
                i += 1
        return X, y, self.sample_weights"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">numpy</span> <span class="pl-k">as</span> <span class="pl-s1">np</span>
<span class="pl-k">import</span> <span class="pl-s1">keras</span>
<span class="pl-k">import</span> <span class="pl-s1">librosa</span>
<span class="pl-k">from</span> <span class="pl-s1">time</span> <span class="pl-k">import</span> <span class="pl-s1">time</span>
<span class="pl-k">import</span> <span class="pl-s1">random</span>
<span class="pl-k">from</span> <span class="pl-s1">config</span> <span class="pl-k">import</span> <span class="pl-c1">*</span>

<span class="pl-k">class</span> <span class="pl-v">DataGenerator</span>(<span class="pl-s1">keras</span>.<span class="pl-s1">utils</span>.<span class="pl-v">Sequence</span>):

    <span class="pl-s">'Generates data for Keras'</span>

    <span class="pl-k">def</span> <span class="pl-en">__init__</span>(<span class="pl-s1">self</span>, <span class="pl-s1">dataframe</span>, <span class="pl-s1">batch_size</span><span class="pl-c1">=</span><span class="pl-c1">None</span>, <span class="pl-s1">dim</span><span class="pl-c1">=</span><span class="pl-c1">None</span>, <span class="pl-s1">labels_dim</span><span class="pl-c1">=</span><span class="pl-c1">None</span>,
                 <span class="pl-s1">n_classes</span><span class="pl-c1">=</span><span class="pl-c1">None</span>, <span class="pl-s1">shuffle</span><span class="pl-c1">=</span><span class="pl-c1">True</span>, <span class="pl-s1">samples</span><span class="pl-c1">=</span><span class="pl-c1">None</span>, <span class="pl-s1">duration</span><span class="pl-c1">=</span><span class="pl-c1">None</span>, <span class="pl-s1">sample_weights</span><span class="pl-c1">=</span><span class="pl-c1">None</span>):
        <span class="pl-s">'Initialization'</span>
        <span class="pl-s1">self</span>.<span class="pl-s1">dim</span> <span class="pl-c1">=</span> <span class="pl-s1">dim</span>
        <span class="pl-s1">self</span>.<span class="pl-s1">batch_size</span> <span class="pl-c1">=</span> <span class="pl-s1">batch_size</span>

        <span class="pl-s1">self</span>.<span class="pl-s1">dataframe</span> <span class="pl-c1">=</span> <span class="pl-s1">dataframe</span>
        <span class="pl-s1">self</span>.<span class="pl-s1">dataframe</span> <span class="pl-c1">=</span> <span class="pl-s1">self</span>.<span class="pl-s1">dataframe</span>.<span class="pl-en">sample</span>(<span class="pl-s1">n</span><span class="pl-c1">=</span><span class="pl-en">len</span>(<span class="pl-s1">self</span>.<span class="pl-s1">dataframe</span>))
        <span class="pl-s1">self</span>.<span class="pl-s1">samples</span> <span class="pl-c1">=</span> <span class="pl-s1">samples</span>
        <span class="pl-s1">self</span>.<span class="pl-en">on_epoch_end</span>()
        <span class="pl-s1">self</span>.<span class="pl-s1">shuffle</span> <span class="pl-c1">=</span> <span class="pl-s1">shuffle</span>
        <span class="pl-s1">self</span>.<span class="pl-s1">sample_weights</span> <span class="pl-c1">=</span> <span class="pl-s1">sample_weights</span>

    <span class="pl-k">def</span> <span class="pl-en">__len__</span>(<span class="pl-s1">self</span>):
        <span class="pl-s">'Denotes the number of batches per epoch'</span>
        <span class="pl-k">return</span> <span class="pl-en">int</span>(<span class="pl-s1">np</span>.<span class="pl-en">floor</span>(<span class="pl-en">len</span>(<span class="pl-s1">self</span>.<span class="pl-s1">dataframe</span>) <span class="pl-c1">/</span> <span class="pl-s1">self</span>.<span class="pl-s1">batch_size</span>))

    <span class="pl-k">def</span> <span class="pl-en">__getitem__</span>(<span class="pl-s1">self</span>, <span class="pl-s1">index</span>):
        <span class="pl-s">'Generate one batch of data'</span>
        <span class="pl-s1">random_pd</span> <span class="pl-c1">=</span> <span class="pl-s1">self</span>.<span class="pl-s1">dataframe</span>.<span class="pl-s1">iloc</span>[<span class="pl-s1">self</span>.<span class="pl-s1">batch_size</span><span class="pl-c1">*</span><span class="pl-s1">index</span> : (<span class="pl-s1">index</span><span class="pl-c1">+</span><span class="pl-c1">1</span>)<span class="pl-c1">*</span><span class="pl-s1">self</span>.<span class="pl-s1">batch_size</span>]
        <span class="pl-c"># Generate data</span>
        <span class="pl-v">X</span>, <span class="pl-s1">y</span> <span class="pl-c1">=</span> <span class="pl-s1">self</span>.<span class="pl-en">__data_generation</span>(<span class="pl-s1">random_pd</span>)
        <span class="pl-k">return</span> <span class="pl-v">X</span>, <span class="pl-s1">y</span>
    <span class="pl-k">def</span> <span class="pl-en">__data_generation</span>(<span class="pl-s1">self</span>, <span class="pl-s1">random_pd</span>):
        <span class="pl-s">'Generates data containing batch_size samples'</span> <span class="pl-c"># X : (n_samples, *dim, n_channels)</span>
        <span class="pl-c"># Initialization</span>
        <span class="pl-v">X</span> <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-en">empty</span>((<span class="pl-s1">self</span>.<span class="pl-s1">batch_size</span>, <span class="pl-c1">1</span>, <span class="pl-s1">self</span>.<span class="pl-s1">samples</span>))
        <span class="pl-s1">y</span> <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-en">empty</span>((<span class="pl-s1">self</span>.<span class="pl-s1">batch_size</span>, <span class="pl-s1">self</span>.<span class="pl-s1">n_classes</span>))
        <span class="pl-s1">i</span> <span class="pl-c1">=</span> <span class="pl-c1">0</span>
        <span class="pl-k">while</span> <span class="pl-s1">i</span> <span class="pl-c1">&lt;</span> <span class="pl-s1">self</span>.<span class="pl-s1">batch_size</span>:
            <span class="pl-k">for</span> <span class="pl-s1">index</span>, <span class="pl-s1">row</span> <span class="pl-c1">in</span> <span class="pl-s1">random_pd</span>.<span class="pl-en">iterrows</span>():
                 <span class="pl-c"># generate </span>
                <span class="pl-s1">y</span>[<span class="pl-s1">i</span>,] <span class="pl-c1">=</span> <span class="pl-s1">label</span>
                <span class="pl-v">X</span>[<span class="pl-s1">i</span>,] <span class="pl-c1">=</span> ...
                <span class="pl-s1">i</span> <span class="pl-c1">+=</span> <span class="pl-c1">1</span>
        <span class="pl-k">return</span> <span class="pl-v">X</span>, <span class="pl-s1">y</span>, <span class="pl-s1">self</span>.<span class="pl-s1">sample_weights</span></pre></div>
<p dir="auto">so it will return <code class="notranslate">X, y, self.sample_weights</code>.</p>
<p dir="auto">The problem is that we will get a <code class="notranslate">StopIteration: too many values to unpack </code> as it would expect 2 but I'm giving 3 values - as actually I'm doing.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Traceback (most recent call last):
File &quot;train.py&quot;, line 438, in &lt;module&gt;
train()
File &quot;train.py&quot;, line 422, in train
callbacks=callbacks
File &quot;/usr/local/lib/python2.7/dist-packages/keras/legacy/interfaces.py&quot;, line 91, in wrapper
return func(*args, **kwargs)
File &quot;/usr/local/lib/python2.7/dist-packages/keras/models.py&quot;, line 1315, in fit_generator
initial_epoch=initial_epoch)
File &quot;/usr/local/lib/python2.7/dist-packages/keras/legacy/interfaces.py&quot;, line 91, in wrapper
return func(*args, **kwargs)
File &quot;/usr/local/lib/python2.7/dist-packages/keras/engine/training.py&quot;, line 2250, in fit_generator
max_queue_size=max_queue_size)
File &quot;/usr/local/lib/python2.7/dist-packages/keras/legacy/interfaces.py&quot;, line 91, in wrapper
return func(*args, **kwargs)
File &quot;/usr/local/lib/python2.7/dist-packages/keras/engine/training.py&quot;, line 2383, in evaluate_generator
generator_output = next(output_generator)
File &quot;/usr/local/lib/python2.7/dist-packages/keras/utils/data_utils.py&quot;, line 584, in get
six.raise_from(StopIteration(e), e)
File &quot;/usr/local/lib/python2.7/dist-packages/six.py&quot;, line 737, in raise_from
raise value
StopIteration: too many values to unpack "><pre class="notranslate"><code class="notranslate">Traceback (most recent call last):
File "train.py", line 438, in &lt;module&gt;
train()
File "train.py", line 422, in train
callbacks=callbacks
File "/usr/local/lib/python2.7/dist-packages/keras/legacy/interfaces.py", line 91, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/keras/models.py", line 1315, in fit_generator
initial_epoch=initial_epoch)
File "/usr/local/lib/python2.7/dist-packages/keras/legacy/interfaces.py", line 91, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/keras/engine/training.py", line 2250, in fit_generator
max_queue_size=max_queue_size)
File "/usr/local/lib/python2.7/dist-packages/keras/legacy/interfaces.py", line 91, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/keras/engine/training.py", line 2383, in evaluate_generator
generator_output = next(output_generator)
File "/usr/local/lib/python2.7/dist-packages/keras/utils/data_utils.py", line 584, in get
six.raise_from(StopIteration(e), e)
File "/usr/local/lib/python2.7/dist-packages/six.py", line 737, in raise_from
raise value
StopIteration: too many values to unpack 
</code></pre></div>
<p dir="auto">I call the fit_generator as usual passing my <code class="notranslate">training_generator</code> then</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="history = model.fit_generator(generator=training_generator,
                                class_weight=class_weights,
                                verbose=1,
                                use_multiprocessing=True,
                                workers=24, 
                                steps_per_epoch=training_steps_per_epoch,
                                epochs=epochs,
                                validation_data=validation_generator,
                                validation_steps = validation_steps_per_epoch,
                                callbacks=callbacks
                                )"><pre class="notranslate"><span class="pl-s1">history</span> <span class="pl-c1">=</span> <span class="pl-s1">model</span>.<span class="pl-en">fit_generator</span>(<span class="pl-s1">generator</span><span class="pl-c1">=</span><span class="pl-s1">training_generator</span>,
                                <span class="pl-s1">class_weight</span><span class="pl-c1">=</span><span class="pl-s1">class_weights</span>,
                                <span class="pl-s1">verbose</span><span class="pl-c1">=</span><span class="pl-c1">1</span>,
                                <span class="pl-s1">use_multiprocessing</span><span class="pl-c1">=</span><span class="pl-c1">True</span>,
                                <span class="pl-s1">workers</span><span class="pl-c1">=</span><span class="pl-c1">24</span>, 
                                <span class="pl-s1">steps_per_epoch</span><span class="pl-c1">=</span><span class="pl-s1">training_steps_per_epoch</span>,
                                <span class="pl-s1">epochs</span><span class="pl-c1">=</span><span class="pl-s1">epochs</span>,
                                <span class="pl-s1">validation_data</span><span class="pl-c1">=</span><span class="pl-s1">validation_generator</span>,
                                <span class="pl-s1">validation_steps</span> <span class="pl-c1">=</span> <span class="pl-s1">validation_steps_per_epoch</span>,
                                <span class="pl-s1">callbacks</span><span class="pl-c1">=</span><span class="pl-s1">callbacks</span>
                                )</pre></div>
<p dir="auto">I do this because when using the <code class="notranslate">fit_generator</code> it is not possibile to pass the <code class="notranslate">sample_weight</code>, since the method signature only supports the <code class="notranslate">class_weight</code> - see here <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="387720418" data-permission-text="Title is private" data-url="https://github.com/keras-team/keras/issues/11800" data-hovercard-type="issue" data-hovercard-url="/keras-team/keras/issues/11800/hovercard" href="https://github.com/keras-team/keras/issues/11800">#11800</a></p>