<p dir="auto"><strong>System information</strong></p>
<ul dir="auto">
<li>Have I written custom code (as opposed to using a stock example script provided in TensorFlow): no (<a href="https://www.tensorflow.org/alpha/tutorials/keras/feature_columns" rel="nofollow">https://www.tensorflow.org/alpha/tutorials/keras/feature_columns</a> + model.save + load_model)</li>
<li>OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 10</li>
<li>Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: no</li>
<li>TensorFlow installed from (source or binary): binary</li>
<li>TensorFlow version (use command below): 2.0.0-alpha0</li>
<li>Python version: 3.6.8</li>
<li>CUDA/cuDNN version: 10.0.130/7.3.1</li>
<li>GPU model and memory: GTX 1060 6GB</li>
</ul>
<p dir="auto"><strong>Describe the current behavior</strong><br>
loading a saved .h5 model which includes a DenseFeatures Layer fails:<br>
<code class="notranslate">ValueError: Unknown layer: DenseFeatures</code></p>
<p dir="auto"><strong>Describe the expected behavior</strong><br>
model is loading</p>
<p dir="auto"><strong>Code to reproduce the issue</strong></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import pandas as pd

# pip install -q tensorflow==2.0.0-alpha0
import tensorflow as tf
from tensorflow import feature_column
from tensorflow.keras import layers
from sklearn.model_selection import train_test_split

URL = 'https://storage.googleapis.com/applied-dl/heart.csv'
dataframe = pd.read_csv(URL)
train, test = train_test_split(dataframe, test_size=0.2)
train, val = train_test_split(train, test_size=0.2)

# A utility method to create a tf.data dataset from a Pandas Dataframe
def df_to_dataset(dataframe, shuffle=True, batch_size=32):
  dataframe = dataframe.copy()
  labels = dataframe.pop('target')
  ds = tf.data.Dataset.from_tensor_slices((dict(dataframe), labels))
  if shuffle:
    ds = ds.shuffle(buffer_size=len(dataframe))
  ds = ds.batch(batch_size)
  return ds

feature_columns = []

# numeric cols
for header in ['age', 'trestbps', 'chol', 'thalach', 'oldpeak', 'slope', 'ca']:
  feature_columns.append(feature_column.numeric_column(header))

feature_layer = tf.keras.layers.DenseFeatures(feature_columns)

batch_size = 32
train_ds = df_to_dataset(train, batch_size=batch_size)
val_ds = df_to_dataset(val, shuffle=False, batch_size=batch_size)
test_ds = df_to_dataset(test, shuffle=False, batch_size=batch_size)

model = tf.keras.Sequential([
  feature_layer,
  layers.Dense(128, activation='relu'),
  layers.Dense(128, activation='relu'),
  layers.Dense(1, activation='sigmoid')
])

model.compile(optimizer='adam',
              loss='binary_crossentropy')

model.fit(train_ds, 
          validation_data=val_ds, 
          epochs=5)


model.save('my_model.h5')
from tensorflow import keras
new_model = keras.models.load_model('my_model.h5')"><pre class="notranslate"><code class="notranslate">import pandas as pd

# pip install -q tensorflow==2.0.0-alpha0
import tensorflow as tf
from tensorflow import feature_column
from tensorflow.keras import layers
from sklearn.model_selection import train_test_split

URL = 'https://storage.googleapis.com/applied-dl/heart.csv'
dataframe = pd.read_csv(URL)
train, test = train_test_split(dataframe, test_size=0.2)
train, val = train_test_split(train, test_size=0.2)

# A utility method to create a tf.data dataset from a Pandas Dataframe
def df_to_dataset(dataframe, shuffle=True, batch_size=32):
  dataframe = dataframe.copy()
  labels = dataframe.pop('target')
  ds = tf.data.Dataset.from_tensor_slices((dict(dataframe), labels))
  if shuffle:
    ds = ds.shuffle(buffer_size=len(dataframe))
  ds = ds.batch(batch_size)
  return ds

feature_columns = []

# numeric cols
for header in ['age', 'trestbps', 'chol', 'thalach', 'oldpeak', 'slope', 'ca']:
  feature_columns.append(feature_column.numeric_column(header))

feature_layer = tf.keras.layers.DenseFeatures(feature_columns)

batch_size = 32
train_ds = df_to_dataset(train, batch_size=batch_size)
val_ds = df_to_dataset(val, shuffle=False, batch_size=batch_size)
test_ds = df_to_dataset(test, shuffle=False, batch_size=batch_size)

model = tf.keras.Sequential([
  feature_layer,
  layers.Dense(128, activation='relu'),
  layers.Dense(128, activation='relu'),
  layers.Dense(1, activation='sigmoid')
])

model.compile(optimizer='adam',
              loss='binary_crossentropy')

model.fit(train_ds, 
          validation_data=val_ds, 
          epochs=5)


model.save('my_model.h5')
from tensorflow import keras
new_model = keras.models.load_model('my_model.h5')
</code></pre></div>
<p dir="auto"><strong>Other info / logs</strong></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Traceback (most recent call last):
  File &quot;error.py&quot;, line 67, in &lt;module&gt;
    new_model = keras.models.load_model('my_model.h5')
  File &quot;C:\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\keras\saving\hdf5_format.py&quot;, line 215, in load_model
    custom_objects=custom_objects)
  File &quot;C:\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\keras\saving\model_config.py&quot;, line 55, in model_from_config
    return deserialize(config, custom_objects=custom_objects)
  File &quot;C:\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\keras\layers\serialization.py&quot;, line 95, in deserialize
    printable_module_name='layer')
  File &quot;C:\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\keras\utils\generic_utils.py&quot;, line 192, in deserialize_keras_object
    list(custom_objects.items())))
  File &quot;C:\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\keras\engine\sequential.py&quot;, line 351, in from_config
    custom_objects=custom_objects)
  File &quot;C:\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\keras\layers\serialization.py&quot;, line 95, in deserialize
    printable_module_name='layer')
  File &quot;C:\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\keras\utils\generic_utils.py&quot;, line 181, in deserialize_keras_object
    config, module_objects, custom_objects, printable_module_name)
  File &quot;C:\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\keras\utils\generic_utils.py&quot;, line 166, in class_and_config_for_serialized_keras_object
    raise ValueError('Unknown ' + printable_module_name + ': ' + class_name)
ValueError: Unknown layer: DenseFeatures"><pre class="notranslate"><code class="notranslate">Traceback (most recent call last):
  File "error.py", line 67, in &lt;module&gt;
    new_model = keras.models.load_model('my_model.h5')
  File "C:\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\keras\saving\hdf5_format.py", line 215, in load_model
    custom_objects=custom_objects)
  File "C:\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\keras\saving\model_config.py", line 55, in model_from_config
    return deserialize(config, custom_objects=custom_objects)
  File "C:\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\keras\layers\serialization.py", line 95, in deserialize
    printable_module_name='layer')
  File "C:\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\keras\utils\generic_utils.py", line 192, in deserialize_keras_object
    list(custom_objects.items())))
  File "C:\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\keras\engine\sequential.py", line 351, in from_config
    custom_objects=custom_objects)
  File "C:\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\keras\layers\serialization.py", line 95, in deserialize
    printable_module_name='layer')
  File "C:\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\keras\utils\generic_utils.py", line 181, in deserialize_keras_object
    config, module_objects, custom_objects, printable_module_name)
  File "C:\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\keras\utils\generic_utils.py", line 166, in class_and_config_for_serialized_keras_object
    raise ValueError('Unknown ' + printable_module_name + ': ' + class_name)
ValueError: Unknown layer: DenseFeatures
</code></pre></div>
<p dir="auto">Do we have any options to control the number of threads in TF-Slim both in training and evaluation processes?</p>
<p dir="auto">Specifically, I use <a href="https://github.com/pudae/tensorflow-densenet">this network</a> for my classification problem. I changed the evaluation part in a way that runs train and evaluation in parallel like <a href="https://github.com/mnuke/tf-slim-mnist">this code</a>. I can run it on my own CPU without any problem. But I can't execute them on a supercomputer. It seems that it is related to the very large number of threads which are being created by Tensorflow. If the number of threads exceeds the maximum number of threads pre-set in SLURM (= 28) then the job will fail. Since it's unable to create new threads it will end up with error "resource temporarily unavailable".</p>
<p dir="auto">This error provided when the code tries to restore parameters from checkpoints. If there is no limitation on the number of threads (like on my pc) it works fine:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="INFO:tensorflow:Restoring parameters from ./model.ckpt-0
INFO:tensorflow:Starting evaluation at
I tensorflow/core/kernels/logging_ops.cc:79] eval/Accuracy[0]
I tensorflow/core/kernels/logging_ops.cc:79] eval/Recall_5[0]
INFO:tensorflow:Evaluation [1/60]"><pre class="notranslate"><code class="notranslate">INFO:tensorflow:Restoring parameters from ./model.ckpt-0
INFO:tensorflow:Starting evaluation at
I tensorflow/core/kernels/logging_ops.cc:79] eval/Accuracy[0]
I tensorflow/core/kernels/logging_ops.cc:79] eval/Recall_5[0]
INFO:tensorflow:Evaluation [1/60]
</code></pre></div>
<p dir="auto">However, when there is a limitation on the number of threads (like SLURM job submission on supercomputers) we get:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="INFO:tensorflow:Restoring parameters from ./model.ckpt-0
terminate called after throwing an instance of 'std::system_error'
what():  Resource temporarily unavailable"><pre class="notranslate"><code class="notranslate">INFO:tensorflow:Restoring parameters from ./model.ckpt-0
terminate called after throwing an instance of 'std::system_error'
what():  Resource temporarily unavailable
</code></pre></div>
<p dir="auto">I tried to limit the number of CPU threads used by Tensorflow to 1 by creating config like:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="  FLAGS.num_preprocessing_threads=1

  config = tf.ConfigProto()
  config.intra_op_parallelism_threads = FLAGS.num_preprocessing_threads
  config.inter_op_parallelism_threads = FLAGS.num_preprocessing_threads

    slim.evaluation.evaluation_loop(
        master=FLAGS.master,
        checkpoint_path=each_ckpt,
        logdir=FLAGS.eval_dir,
        num_evals=num_batches,
        eval_op=list(names_to_updates.values()) + print_ops,
        variables_to_restore=variables_to_restore,
        session_config=config)"><pre class="notranslate"><code class="notranslate">  FLAGS.num_preprocessing_threads=1

  config = tf.ConfigProto()
  config.intra_op_parallelism_threads = FLAGS.num_preprocessing_threads
  config.inter_op_parallelism_threads = FLAGS.num_preprocessing_threads

    slim.evaluation.evaluation_loop(
        master=FLAGS.master,
        checkpoint_path=each_ckpt,
        logdir=FLAGS.eval_dir,
        num_evals=num_batches,
        eval_op=list(names_to_updates.values()) + print_ops,
        variables_to_restore=variables_to_restore,
        session_config=config)
</code></pre></div>
<p dir="auto">But unfortunately, that didn't help. In my opinion, the main problem we are having here is the fact that we are not able to control the number of threads here. Although we set it to 1 with various TF options you can actually see that this job is creating many more threads on the node:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="slurm_script─┬─python───128*[{python}]
             └─python───8*[{python}]"><pre class="notranslate"><code class="notranslate">slurm_script─┬─python───128*[{python}]
             └─python───8*[{python}]
</code></pre></div>
<p dir="auto">Training script is creating 128 threads and evaluation script is creating 8 (both numbers vary over time).</p>
<p dir="auto">P.S. I'm using Python 2.7.13 and Tensorflow 1.3.0.</p>