<p dir="auto"><strong>System information</strong></p>
<ul dir="auto">
<li>Have I written custom code (as opposed to using a stock example script provided in TensorFlow):</li>
</ul>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Yes"><pre class="notranslate"><code class="notranslate">Yes
</code></pre></div>
<ul dir="auto">
<li>OS Platform and Distribution (e.g., Linux Ubuntu 16.04):</li>
</ul>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Container derived from nvidia/cuda:10.0-cudnn7-runtime-ubuntu18.04 running on Ubuntu 18.04 via Kubernetes 1.13"><pre class="notranslate"><code class="notranslate">Container derived from nvidia/cuda:10.0-cudnn7-runtime-ubuntu18.04 running on Ubuntu 18.04 via Kubernetes 1.13
</code></pre></div>
<ul dir="auto">
<li>TensorFlow installed from (source or binary):</li>
</ul>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip3 install tensorflow-gpu==2.0.0-beta1"><pre class="notranslate"><code class="notranslate">pip3 install tensorflow-gpu==2.0.0-beta1
</code></pre></div>
<ul dir="auto">
<li>TensorFlow version (use command below):</li>
</ul>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="v1.12.1-6250-g37eafe0e74 2.0.0-dev20190715"><pre class="notranslate"><code class="notranslate">v1.12.1-6250-g37eafe0e74 2.0.0-dev20190715
</code></pre></div>
<ul dir="auto">
<li>Python version:</li>
</ul>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Python 3.6.8"><pre class="notranslate"><code class="notranslate">Python 3.6.8
</code></pre></div>
<ul dir="auto">
<li>CUDA/cuDNN version:</li>
</ul>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="NVIDIA-SMI 410.78       Driver Version: 410.78       CUDA Version: 10.0"><pre class="notranslate"><code class="notranslate">NVIDIA-SMI 410.78       Driver Version: 410.78       CUDA Version: 10.0
</code></pre></div>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="        libcudnn.so.7 -&gt; libcudnn.so.7.6.0
libcudnn is installed"><pre class="notranslate"><code class="notranslate">        libcudnn.so.7 -&gt; libcudnn.so.7.6.0
libcudnn is installed
</code></pre></div>
<ul dir="auto">
<li>GPU model and memory:</li>
</ul>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="name, pci.bus_id, vbios_version
Tesla V100-SXM2-32GB, 00000000:1B:00.0, 88.00.43.00.03"><pre class="notranslate"><code class="notranslate">name, pci.bus_id, vbios_version
Tesla V100-SXM2-32GB, 00000000:1B:00.0, 88.00.43.00.03
</code></pre></div>
<p dir="auto"><strong>Describe the current behavior</strong><br>
Doing a training with tf.keras results in out of memory after some time when including the <code class="notranslate">class_weight</code> parameter. Also there is a long delay between the start of each epoch. If I omit the <code class="notranslate">class_weight</code> parameter, training proceeds normally with constant memory.</p>
<p dir="auto"><strong>Describe the expected behavior</strong><br>
Training proceed normally when the <code class="notranslate">class_weight</code> parameter is included without running out of memory.</p>
<p dir="auto"><strong>Code to reproduce the issue</strong></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import tensorflow as tf

################ Data
def _parse_fn2(fn, label):
    img = tf.random.uniform([224, 224, 3])
    return img, label

train_data2 = tf.data.Dataset.from_tensor_slices(
  (tf.random.uniform([100]), tf.random.uniform([100], maxval=9, dtype=tf.dtypes.int32))
)

val_data2 = tf.data.Dataset.from_tensor_slices(
  (tf.random.uniform([100]), tf.random.uniform([100], maxval=9, dtype=tf.dtypes.int32))
)
train_data2 = (train_data2.map(_parse_fn2)).batch(32)
val_data2 = (val_data2.map(_parse_fn2)).batch(32)

############### Model
IMG_SHAPE = (224, 224, 3)

base_model = tf.keras.applications.ResNet50(input_shape=IMG_SHAPE,include_top=False, weights=None)
base_model.trainable = True
maxpool_layer = tf.keras.layers.GlobalMaxPooling2D()
prediction_layer = tf.keras.layers.Dense(9, activation='softmax')

model = tf.keras.Sequential([
    base_model,
    maxpool_layer,
    tf.keras.layers.Dropout(0.4),
    prediction_layer
])

model.compile(optimizer=tf.keras.optimizers.Adam(lr=0.01), loss='categorical_crossentropy', metrics=['accuracy'])

model.summary()
history = model.fit(train_data2.repeat(),
                epochs=100,
                steps_per_epoch = 50,
                validation_data=val_data2.repeat(),
                validation_steps=10,
                class_weight={0:1,1:1,2:1,3:1,4:1,5:1,6:1,7:1,8:1,9:1},
                callbacks = [])"><pre class="notranslate"><code class="notranslate">import tensorflow as tf

################ Data
def _parse_fn2(fn, label):
    img = tf.random.uniform([224, 224, 3])
    return img, label

train_data2 = tf.data.Dataset.from_tensor_slices(
  (tf.random.uniform([100]), tf.random.uniform([100], maxval=9, dtype=tf.dtypes.int32))
)

val_data2 = tf.data.Dataset.from_tensor_slices(
  (tf.random.uniform([100]), tf.random.uniform([100], maxval=9, dtype=tf.dtypes.int32))
)
train_data2 = (train_data2.map(_parse_fn2)).batch(32)
val_data2 = (val_data2.map(_parse_fn2)).batch(32)

############### Model
IMG_SHAPE = (224, 224, 3)

base_model = tf.keras.applications.ResNet50(input_shape=IMG_SHAPE,include_top=False, weights=None)
base_model.trainable = True
maxpool_layer = tf.keras.layers.GlobalMaxPooling2D()
prediction_layer = tf.keras.layers.Dense(9, activation='softmax')

model = tf.keras.Sequential([
    base_model,
    maxpool_layer,
    tf.keras.layers.Dropout(0.4),
    prediction_layer
])

model.compile(optimizer=tf.keras.optimizers.Adam(lr=0.01), loss='categorical_crossentropy', metrics=['accuracy'])

model.summary()
history = model.fit(train_data2.repeat(),
                epochs=100,
                steps_per_epoch = 50,
                validation_data=val_data2.repeat(),
                validation_steps=10,
                class_weight={0:1,1:1,2:1,3:1,4:1,5:1,6:1,7:1,8:1,9:1},
                callbacks = [])
</code></pre></div>
<p dir="auto"><strong>Other info / logs</strong><br>
Memory Usage at Each Epoch:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="              total        used        free      shared  buff/cache   available
E1:           502G         11G        461G        140M         30G        487G
E2:           502G         16G        456G        140M         30G        483G
E3:           502G         22G        449G        140M         30G        476G
E4:           502G         30G        441G        141M         30G        468G
E5:           502G         40G        431G        141M         30G        458G
E6:           502G         52G        419G        141M         30G        446G"><pre class="notranslate"><code class="notranslate">              total        used        free      shared  buff/cache   available
E1:           502G         11G        461G        140M         30G        487G
E2:           502G         16G        456G        140M         30G        483G
E3:           502G         22G        449G        140M         30G        476G
E4:           502G         30G        441G        141M         30G        468G
E5:           502G         40G        431G        141M         30G        458G
E6:           502G         52G        419G        141M         30G        446G
</code></pre></div>
<p dir="auto">vmstat output:<br>
<a href="https://github.com/tensorflow/tensorflow/files/3407988/vmstat_output.txt">vmstat_output.txt</a></p>
<p dir="auto">When creating a Reader op inside core/user_ops that inherits from ReaderBase, Bazel responds with</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="tensorflow/core/BUILD:325:1: undeclared inclusion(s) in rule '//tensorflow/core:user_ops_op_lib':
this rule is missing dependency declarations for the following files included by 'tensorflow/core/user_ops/user_reader_op.cc':
  'bazel-out/local_darwin-opt/genfiles/tensorflow/core/kernels/reader_base.pb.h'."><pre class="notranslate"><code class="notranslate">tensorflow/core/BUILD:325:1: undeclared inclusion(s) in rule '//tensorflow/core:user_ops_op_lib':
this rule is missing dependency declarations for the following files included by 'tensorflow/core/user_ops/user_reader_op.cc':
  'bazel-out/local_darwin-opt/genfiles/tensorflow/core/kernels/reader_base.pb.h'.
</code></pre></div>
<p dir="auto">Is this because the user_ops_op_lib depends only on framework? user_reader_op.cc includes reader_base.h which includes reader_base.pb.h. What dependency should I add to make this work? I'm still figuring out Bazel atm.</p>
<p dir="auto">This is with latest source code.</p>