<p dir="auto">(Using <code class="notranslate">tf 1.0.0</code>)<br>
I tried to follow the following example (<a href="https://www.tensorflow.org/get_started/tflearn" rel="nofollow">https://www.tensorflow.org/get_started/tflearn</a>) from the TF website and I get the following error:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="INFO:tensorflow:Using default config.
INFO:tensorflow:Using config: {'_cluster_spec': &lt;tensorflow.python.training.server_lib.ClusterSpec object at 0x7f9cfb283e80&gt;, '_task_id': 0, '_keep_checkpoint_every_n_hours': 10000, '_keep_checkpoint_max': 5, '_save_checkpoints_steps': None, '_task_type': None, '_environment': 'local', '_tf_config': gpu_options {
  per_process_gpu_memory_fraction: 1.0
}
, '_master': '', '_save_checkpoints_secs': 600, '_is_chief': True, '_tf_random_seed': None, '_num_ps_replicas': 0, '_save_summary_steps': 100, '_evaluation_master': ''}
WARNING:tensorflow:From &lt;ipython-input-131-b49d002a31c2&gt;:14: calling BaseEstimator.fit (from tensorflow.contrib.learn.python.learn.estimators.estimator) with x is deprecated and will be removed after 2016-12-01.
Instructions for updating:
Estimator is decoupled from Scikit Learn interface by moving into
separate class SKCompat. Arguments x, y and batch_size are only
available in the SKCompat class, Estimator will only accept input_fn.
Example conversion:
  est = Estimator(...) -&gt; est = SKCompat(Estimator(...))
WARNING:tensorflow:From &lt;ipython-input-131-b49d002a31c2&gt;:14: calling BaseEstimator.fit (from tensorflow.contrib.learn.python.learn.estimators.estimator) with y is deprecated and will be removed after 2016-12-01.
Instructions for updating:
Estimator is decoupled from Scikit Learn interface by moving into
separate class SKCompat. Arguments x, y and batch_size are only
available in the SKCompat class, Estimator will only accept input_fn.
Example conversion:
  est = Estimator(...) -&gt; est = SKCompat(Estimator(...))
WARNING:tensorflow:From /opt/conda/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/estimators/head.py:1362: scalar_summary (from tensorflow.python.ops.logging_ops) is deprecated and will be removed after 2016-11-30.
Instructions for updating:
Please switch to tf.summary.scalar. Note that tf.summary.scalar uses the node name instead of the tag. This means that TensorFlow will automatically de-duplicate summary names based on the scope they are created in. Also, passing a tensor or list of tags to a scalar summary op is no longer supported.
/opt/conda/lib/python3.5/site-packages/tensorflow/python/util/deprecation.py:247: FutureWarning: comparison to `None` will result in an elementwise object comparison in the future.
  equality = a == b
INFO:tensorflow:Create CheckpointSaverHook.
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
TypeError: expected bytes, tuple found

During handling of the above exception, another exception occurred:

SystemError                               Traceback (most recent call last)
&lt;ipython-input-131-b49d002a31c2&gt; in &lt;module&gt;()
     12 classifier.fit(x=results,
     13                y=labels,
---&gt; 14                steps=2000)
     15 
     16 # Evaluate accuracy.

/opt/conda/lib/python3.5/site-packages/tensorflow/python/util/deprecation.py in new_func(*args, **kwargs)
    278             _call_location(), decorator_utils.get_qualified_name(func),
    279             func.__module__, arg_name, date, instructions)
--&gt; 280       return func(*args, **kwargs)
    281     new_func.__doc__ = _add_deprecated_arg_notice_to_docstring(
    282         func.__doc__, date, instructions)

/opt/conda/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py in fit(self, x, y, input_fn, steps, batch_size, monitors, max_steps)
    408     _verify_input_args(x, y, input_fn, None, batch_size)
    409     if x is not None:
--&gt; 410       SKCompat(self).fit(x, y, batch_size, steps, max_steps, monitors)
    411       return self
    412 

/opt/conda/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py in fit(self, x, y, batch_size, steps, max_steps, monitors)
   1351                         steps=steps,
   1352                         max_steps=max_steps,
-&gt; 1353                         monitors=all_monitors)
   1354     return self
   1355 

/opt/conda/lib/python3.5/site-packages/tensorflow/python/util/deprecation.py in new_func(*args, **kwargs)
    278             _call_location(), decorator_utils.get_qualified_name(func),
    279             func.__module__, arg_name, date, instructions)
--&gt; 280       return func(*args, **kwargs)
    281     new_func.__doc__ = _add_deprecated_arg_notice_to_docstring(
    282         func.__doc__, date, instructions)

/opt/conda/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py in fit(self, x, y, input_fn, steps, batch_size, monitors, max_steps)
    424       hooks.append(basic_session_run_hooks.StopAtStepHook(steps, max_steps))
    425 
--&gt; 426     loss = self._train_model(input_fn=input_fn, hooks=hooks)
    427     logging.info('Loss for final step: %s.', loss)
    428     return self

/opt/conda/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py in _train_model(self, input_fn, hooks)
    982         loss = None
    983         while not mon_sess.should_stop():
--&gt; 984           _, loss = mon_sess.run([model_fn_ops.train_op, model_fn_ops.loss])
    985       summary_io.SummaryWriterCache.clear()
    986       return loss

/opt/conda/lib/python3.5/site-packages/tensorflow/python/training/monitored_session.py in run(self, fetches, feed_dict, options, run_metadata)
    460                           feed_dict=feed_dict,
    461                           options=options,
--&gt; 462                           run_metadata=run_metadata)
    463 
    464   def should_stop(self):

/opt/conda/lib/python3.5/site-packages/tensorflow/python/training/monitored_session.py in run(self, fetches, feed_dict, options, run_metadata)
    784                               feed_dict=feed_dict,
    785                               options=options,
--&gt; 786                               run_metadata=run_metadata)
    787       except errors.AbortedError:
    788         logging.info('An AbortedError was raised. Closing the current session. '

/opt/conda/lib/python3.5/site-packages/tensorflow/python/training/monitored_session.py in run(self, *args, **kwargs)
    742 
    743   def run(self, *args, **kwargs):
--&gt; 744     return self._sess.run(*args, **kwargs)
    745 
    746 

/opt/conda/lib/python3.5/site-packages/tensorflow/python/training/monitored_session.py in run(self, fetches, feed_dict, options, run_metadata)
    889                                   feed_dict=feed_dict,
    890                                   options=options,
--&gt; 891                                   run_metadata=run_metadata)
    892 
    893     for hook in self._hooks:

/opt/conda/lib/python3.5/site-packages/tensorflow/python/training/monitored_session.py in run(self, *args, **kwargs)
    742 
    743   def run(self, *args, **kwargs):
--&gt; 744     return self._sess.run(*args, **kwargs)
    745 
    746 

/opt/conda/lib/python3.5/site-packages/tensorflow/python/client/session.py in run(self, fetches, feed_dict, options, run_metadata)
    765     try:
    766       result = self._run(None, fetches, feed_dict, options_ptr,
--&gt; 767                          run_metadata_ptr)
    768       if run_metadata:
    769         proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

/opt/conda/lib/python3.5/site-packages/tensorflow/python/client/session.py in _run(self, handle, fetches, feed_dict, options, run_metadata)
    963     if final_fetches or final_targets:
    964       results = self._do_run(handle, final_targets, final_fetches,
--&gt; 965                              feed_dict_string, options, run_metadata)
    966     else:
    967       results = []

/opt/conda/lib/python3.5/site-packages/tensorflow/python/client/session.py in _do_run(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)
   1013     if handle is None:
   1014       return self._do_call(_run_fn, self._session, feed_dict, fetch_list,
-&gt; 1015                            target_list, options, run_metadata)
   1016     else:
   1017       return self._do_call(_prun_fn, self._session, handle, feed_dict,

/opt/conda/lib/python3.5/site-packages/tensorflow/python/client/session.py in _do_call(self, fn, *args)
   1020   def _do_call(self, fn, *args):
   1021     try:
-&gt; 1022       return fn(*args)
   1023     except errors.OpError as e:
   1024       message = compat.as_text(e.message)

/opt/conda/lib/python3.5/site-packages/tensorflow/python/client/session.py in _run_fn(session, feed_dict, fetch_list, target_list, options, run_metadata)
   1002         return tf_session.TF_Run(session, options,
   1003                                  feed_dict, fetch_list, target_list,
-&gt; 1004                                  status, run_metadata)
   1005 
   1006     def _prun_fn(session, handle, feed_dict, fetch_list):

SystemError: &lt;built-in function TF_Run&gt; returned a result with an error set"><pre class="notranslate"><code class="notranslate">INFO:tensorflow:Using default config.
INFO:tensorflow:Using config: {'_cluster_spec': &lt;tensorflow.python.training.server_lib.ClusterSpec object at 0x7f9cfb283e80&gt;, '_task_id': 0, '_keep_checkpoint_every_n_hours': 10000, '_keep_checkpoint_max': 5, '_save_checkpoints_steps': None, '_task_type': None, '_environment': 'local', '_tf_config': gpu_options {
  per_process_gpu_memory_fraction: 1.0
}
, '_master': '', '_save_checkpoints_secs': 600, '_is_chief': True, '_tf_random_seed': None, '_num_ps_replicas': 0, '_save_summary_steps': 100, '_evaluation_master': ''}
WARNING:tensorflow:From &lt;ipython-input-131-b49d002a31c2&gt;:14: calling BaseEstimator.fit (from tensorflow.contrib.learn.python.learn.estimators.estimator) with x is deprecated and will be removed after 2016-12-01.
Instructions for updating:
Estimator is decoupled from Scikit Learn interface by moving into
separate class SKCompat. Arguments x, y and batch_size are only
available in the SKCompat class, Estimator will only accept input_fn.
Example conversion:
  est = Estimator(...) -&gt; est = SKCompat(Estimator(...))
WARNING:tensorflow:From &lt;ipython-input-131-b49d002a31c2&gt;:14: calling BaseEstimator.fit (from tensorflow.contrib.learn.python.learn.estimators.estimator) with y is deprecated and will be removed after 2016-12-01.
Instructions for updating:
Estimator is decoupled from Scikit Learn interface by moving into
separate class SKCompat. Arguments x, y and batch_size are only
available in the SKCompat class, Estimator will only accept input_fn.
Example conversion:
  est = Estimator(...) -&gt; est = SKCompat(Estimator(...))
WARNING:tensorflow:From /opt/conda/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/estimators/head.py:1362: scalar_summary (from tensorflow.python.ops.logging_ops) is deprecated and will be removed after 2016-11-30.
Instructions for updating:
Please switch to tf.summary.scalar. Note that tf.summary.scalar uses the node name instead of the tag. This means that TensorFlow will automatically de-duplicate summary names based on the scope they are created in. Also, passing a tensor or list of tags to a scalar summary op is no longer supported.
/opt/conda/lib/python3.5/site-packages/tensorflow/python/util/deprecation.py:247: FutureWarning: comparison to `None` will result in an elementwise object comparison in the future.
  equality = a == b
INFO:tensorflow:Create CheckpointSaverHook.
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
TypeError: expected bytes, tuple found

During handling of the above exception, another exception occurred:

SystemError                               Traceback (most recent call last)
&lt;ipython-input-131-b49d002a31c2&gt; in &lt;module&gt;()
     12 classifier.fit(x=results,
     13                y=labels,
---&gt; 14                steps=2000)
     15 
     16 # Evaluate accuracy.

/opt/conda/lib/python3.5/site-packages/tensorflow/python/util/deprecation.py in new_func(*args, **kwargs)
    278             _call_location(), decorator_utils.get_qualified_name(func),
    279             func.__module__, arg_name, date, instructions)
--&gt; 280       return func(*args, **kwargs)
    281     new_func.__doc__ = _add_deprecated_arg_notice_to_docstring(
    282         func.__doc__, date, instructions)

/opt/conda/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py in fit(self, x, y, input_fn, steps, batch_size, monitors, max_steps)
    408     _verify_input_args(x, y, input_fn, None, batch_size)
    409     if x is not None:
--&gt; 410       SKCompat(self).fit(x, y, batch_size, steps, max_steps, monitors)
    411       return self
    412 

/opt/conda/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py in fit(self, x, y, batch_size, steps, max_steps, monitors)
   1351                         steps=steps,
   1352                         max_steps=max_steps,
-&gt; 1353                         monitors=all_monitors)
   1354     return self
   1355 

/opt/conda/lib/python3.5/site-packages/tensorflow/python/util/deprecation.py in new_func(*args, **kwargs)
    278             _call_location(), decorator_utils.get_qualified_name(func),
    279             func.__module__, arg_name, date, instructions)
--&gt; 280       return func(*args, **kwargs)
    281     new_func.__doc__ = _add_deprecated_arg_notice_to_docstring(
    282         func.__doc__, date, instructions)

/opt/conda/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py in fit(self, x, y, input_fn, steps, batch_size, monitors, max_steps)
    424       hooks.append(basic_session_run_hooks.StopAtStepHook(steps, max_steps))
    425 
--&gt; 426     loss = self._train_model(input_fn=input_fn, hooks=hooks)
    427     logging.info('Loss for final step: %s.', loss)
    428     return self

/opt/conda/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py in _train_model(self, input_fn, hooks)
    982         loss = None
    983         while not mon_sess.should_stop():
--&gt; 984           _, loss = mon_sess.run([model_fn_ops.train_op, model_fn_ops.loss])
    985       summary_io.SummaryWriterCache.clear()
    986       return loss

/opt/conda/lib/python3.5/site-packages/tensorflow/python/training/monitored_session.py in run(self, fetches, feed_dict, options, run_metadata)
    460                           feed_dict=feed_dict,
    461                           options=options,
--&gt; 462                           run_metadata=run_metadata)
    463 
    464   def should_stop(self):

/opt/conda/lib/python3.5/site-packages/tensorflow/python/training/monitored_session.py in run(self, fetches, feed_dict, options, run_metadata)
    784                               feed_dict=feed_dict,
    785                               options=options,
--&gt; 786                               run_metadata=run_metadata)
    787       except errors.AbortedError:
    788         logging.info('An AbortedError was raised. Closing the current session. '

/opt/conda/lib/python3.5/site-packages/tensorflow/python/training/monitored_session.py in run(self, *args, **kwargs)
    742 
    743   def run(self, *args, **kwargs):
--&gt; 744     return self._sess.run(*args, **kwargs)
    745 
    746 

/opt/conda/lib/python3.5/site-packages/tensorflow/python/training/monitored_session.py in run(self, fetches, feed_dict, options, run_metadata)
    889                                   feed_dict=feed_dict,
    890                                   options=options,
--&gt; 891                                   run_metadata=run_metadata)
    892 
    893     for hook in self._hooks:

/opt/conda/lib/python3.5/site-packages/tensorflow/python/training/monitored_session.py in run(self, *args, **kwargs)
    742 
    743   def run(self, *args, **kwargs):
--&gt; 744     return self._sess.run(*args, **kwargs)
    745 
    746 

/opt/conda/lib/python3.5/site-packages/tensorflow/python/client/session.py in run(self, fetches, feed_dict, options, run_metadata)
    765     try:
    766       result = self._run(None, fetches, feed_dict, options_ptr,
--&gt; 767                          run_metadata_ptr)
    768       if run_metadata:
    769         proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

/opt/conda/lib/python3.5/site-packages/tensorflow/python/client/session.py in _run(self, handle, fetches, feed_dict, options, run_metadata)
    963     if final_fetches or final_targets:
    964       results = self._do_run(handle, final_targets, final_fetches,
--&gt; 965                              feed_dict_string, options, run_metadata)
    966     else:
    967       results = []

/opt/conda/lib/python3.5/site-packages/tensorflow/python/client/session.py in _do_run(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)
   1013     if handle is None:
   1014       return self._do_call(_run_fn, self._session, feed_dict, fetch_list,
-&gt; 1015                            target_list, options, run_metadata)
   1016     else:
   1017       return self._do_call(_prun_fn, self._session, handle, feed_dict,

/opt/conda/lib/python3.5/site-packages/tensorflow/python/client/session.py in _do_call(self, fn, *args)
   1020   def _do_call(self, fn, *args):
   1021     try:
-&gt; 1022       return fn(*args)
   1023     except errors.OpError as e:
   1024       message = compat.as_text(e.message)

/opt/conda/lib/python3.5/site-packages/tensorflow/python/client/session.py in _run_fn(session, feed_dict, fetch_list, target_list, options, run_metadata)
   1002         return tf_session.TF_Run(session, options,
   1003                                  feed_dict, fetch_list, target_list,
-&gt; 1004                                  status, run_metadata)
   1005 
   1006     def _prun_fn(session, handle, feed_dict, fetch_list):

SystemError: &lt;built-in function TF_Run&gt; returned a result with an error set
</code></pre></div>
<p dir="auto">To replicate:<br>
(For better accuracy use Docker image <code class="notranslate">jupyter/datascience-notebook</code> and install tf)</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
import tensorflow as tf

results = np.ndarray(10,2632)
labels = [0,1] * 5

feature_columns = [tf.contrib.layers.real_valued_column(&quot;&quot;, dimension=4)]


classifier = tf.contrib.learn.DNNClassifier(feature_columns=feature_columns,
                                            hidden_units=[2623,2],
                                            n_classes=2,
                                            model_dir='./model')

# Fit model.
classifier.fit(x=results,
               y=labels,
               steps=2000)

# Evaluate accuracy.
accuracy_score = classifier.evaluate(x=test_set.data,
                                     y=test_set.target)[&quot;accuracy&quot;]
print('Accuracy: {0:f}'.format(accuracy_score))"><pre class="notranslate"><code class="notranslate">from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
import tensorflow as tf

results = np.ndarray(10,2632)
labels = [0,1] * 5

feature_columns = [tf.contrib.layers.real_valued_column("", dimension=4)]


classifier = tf.contrib.learn.DNNClassifier(feature_columns=feature_columns,
                                            hidden_units=[2623,2],
                                            n_classes=2,
                                            model_dir='./model')

# Fit model.
classifier.fit(x=results,
               y=labels,
               steps=2000)

# Evaluate accuracy.
accuracy_score = classifier.evaluate(x=test_set.data,
                                     y=test_set.target)["accuracy"]
print('Accuracy: {0:f}'.format(accuracy_score))
</code></pre></div>
<p dir="auto">I think this might be a problem with tensorflow badly interacting with the underlying file system? Not sure.</p>
<p dir="auto">Per the comment on <a href="https://github.com/tensorflow/tensorflow/blob/master/tensorflow/g3doc/how_tos/distributed/index.md">this introduction</a>, i.e.</p>
<blockquote>
<p dir="auto">N.B. Manually specifying these cluster specifications can be tedious, especially for large clusters. We are working on tools for launching tasks programmatically, e.g. using a cluster manager like Kubernetes. If there are particular cluster managers for which you'd like to see support, please raise a GitHub issue.</p>
</blockquote>
<p dir="auto">Is there is any possibility of supporting <a href="http://slurm.schedmd.com/" rel="nofollow">Slurm</a>? Forgive my ignorance but I've really only played around with TensorFlow and I've only used Slurm for fairly simple MPI projects, but I recently got access to a cluster with some GPU nodes and I'd like to incorporate TF in my research project. It would be great if I was able to use all the resources I could to speed things along.</p>
<p dir="auto">If it helps, specific info about the setup can be found <a href="https://wiki.auckland.ac.nz/display/CER/Centre+for+eResearch+User+Documentation+Start" rel="nofollow">here</a>.</p>