<p dir="auto"><em>Please make sure that this is a bug. As per our <a href="https://github.com/tensorflow/tensorflow/blob/master/ISSUES.md">GitHub Policy</a>, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:bug_template</em></p>
<p dir="auto"><strong>System information</strong></p>
<ul dir="auto">
<li>Have I written custom code (as opposed to using a stock example script provided in TensorFlow):</li>
<li>OS Platform and Distribution (e.g., Linux Ubuntu 16.04):</li>
<li>Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:</li>
<li>TensorFlow installed from (source or binary): from conda</li>
<li>TensorFlow version (use command below): 1.10 gpu from conda</li>
<li>Python version:  3.7 from conda</li>
<li>Bazel version (if compiling from source): n/a</li>
<li>GCC/Compiler version (if compiling from source): n/a</li>
<li>CUDA/cuDNN version: 10</li>
<li>GPU model and memory: GX 1070 maxq</li>
</ul>
<p dir="auto">You can collect some of this information using our environment capture <a href="https://github.com/tensorflow/tensorflow/tree/master/tools/tf_env_collect.sh">script</a><br>
You can also obtain the TensorFlow version with<br>
python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"</p>
<p dir="auto"><strong>Describe the current behavior</strong></p>
<p dir="auto">Suddenly running update row operations became laggy and never progressed...</p>
<p dir="auto"><strong>Describe the expected behavior</strong></p>
<p dir="auto">update row operations should not run indefinetly</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="

from mymodule  import ex
import gc, resource, multiprocessing

def run_ex(print_usage=False, *args, **kwargs):
    ex.run(config_updates=kwargs)
    if print_usage:
        usage = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
        print('resources used: {}'.format(usage))
    gc.collect()

def pool_run(*args, processes=1, print_usage=False, **kwargs):
    with multiprocessing.Pool(processes=processes) as pool:
        results = pool.apply(run_ex, (print_usage, *args), kwargs)
    gc.collect()
    return results

pool_run(processes=1, print_usage=True, **opts)


# ex is a wrapper that calls

def train_fn(
    model,                                   # &lt;--- WALSModel
    input_tensor,                         # tf.SparseTensor
    iterations:int,
    sp_sparse=None,                 # sp.sparse.coo_matrix
    sacred_run=None,              
    print_progress:bool=True,
    logger=None
):
    os.environ['KMP_DUPLICATE_LIB_OK']='True'
    # extract row and column factors
    row_factor = model.row_factors[0]
    col_factor = model.col_factors[0]

    # update operations
    row_update_op = model.update_row_factors(sp_input=input_tensor)#[1]
    col_update_op = model.update_col_factors(sp_input=input_tensor)#[1]



    with tf.Session() as sess:
        # init model
        sess.run(model.initialize_op)
        sess.run(model.worker_init)

        if print_progress:
            status = Sil(total=iterations)
        for i in range(iterations):
            if logger is not None: logger.debug('Training iteration {}/{}'.format(str(i), str(iterations)))
            # update rows
            if logger is not None: logger.debug('Updating rows')   # &lt;--- I see this in logs
            sess.run(model.row_update_prep_gramian_op)
            sess.run(model.initialize_row_update_op)
            _, _, loss, reg, _ = sess.run(row_update_op)
            if sacred_run is not None: sacred_run.log_scalar(&quot;loss.row&quot;, loss, i)   

            # update cols
            if logger is not None: logger.debug('Updating columns')   # &lt;--- never makes it here
            sess.run(model.col_update_prep_gramian_op)
            sess.run(model.initialize_col_update_op)
            _, _, loss, reg, _ = sess.run(col_update_op)
            if sacred_run is not None: sacred_run.log_scalar(&quot;loss.col&quot;, loss, i)

            # update status
            if print_progress:
                status.tick(prefix='iteration')

            if sacred_run is not None and sp_sparse is not None:
                if logger is not None: logger.debug('Calculating rmse.')
                rf = row_factor.eval(session=sess)
                cf = col_factor.eval(session=sess)
                it_rmse = rmse(sp_sparse, rf, cf)
                sacred_run.log_scalar(&quot;rmse&quot;, it_rmse, i)


        # eval row / col factors
        output_row = row_factor.eval(session=sess)
        output_col = col_factor.eval(session=sess)
        sess.close()
    os.environ['KMP_DUPLICATE_LIB_OK']='False'
    return output_row, output_col
"><pre class="notranslate"><span class="pl-k">from</span> <span class="pl-s1">mymodule</span>  <span class="pl-k">import</span> <span class="pl-s1">ex</span>
<span class="pl-k">import</span> <span class="pl-s1">gc</span>, <span class="pl-s1">resource</span>, <span class="pl-s1">multiprocessing</span>

<span class="pl-k">def</span> <span class="pl-en">run_ex</span>(<span class="pl-s1">print_usage</span><span class="pl-c1">=</span><span class="pl-c1">False</span>, <span class="pl-c1">*</span><span class="pl-s1">args</span>, <span class="pl-c1">**</span><span class="pl-s1">kwargs</span>):
    <span class="pl-s1">ex</span>.<span class="pl-en">run</span>(<span class="pl-s1">config_updates</span><span class="pl-c1">=</span><span class="pl-s1">kwargs</span>)
    <span class="pl-k">if</span> <span class="pl-s1">print_usage</span>:
        <span class="pl-s1">usage</span> <span class="pl-c1">=</span> <span class="pl-s1">resource</span>.<span class="pl-en">getrusage</span>(<span class="pl-s1">resource</span>.<span class="pl-v">RUSAGE_SELF</span>).<span class="pl-s1">ru_maxrss</span>
        <span class="pl-en">print</span>(<span class="pl-s">'resources used: {}'</span>.<span class="pl-en">format</span>(<span class="pl-s1">usage</span>))
    <span class="pl-s1">gc</span>.<span class="pl-en">collect</span>()

<span class="pl-k">def</span> <span class="pl-en">pool_run</span>(<span class="pl-c1">*</span><span class="pl-s1">args</span>, <span class="pl-s1">processes</span><span class="pl-c1">=</span><span class="pl-c1">1</span>, <span class="pl-s1">print_usage</span><span class="pl-c1">=</span><span class="pl-c1">False</span>, <span class="pl-c1">**</span><span class="pl-s1">kwargs</span>):
    <span class="pl-k">with</span> <span class="pl-s1">multiprocessing</span>.<span class="pl-v">Pool</span>(<span class="pl-s1">processes</span><span class="pl-c1">=</span><span class="pl-s1">processes</span>) <span class="pl-k">as</span> <span class="pl-s1">pool</span>:
        <span class="pl-s1">results</span> <span class="pl-c1">=</span> <span class="pl-s1">pool</span>.<span class="pl-en">apply</span>(<span class="pl-s1">run_ex</span>, (<span class="pl-s1">print_usage</span>, <span class="pl-c1">*</span><span class="pl-s1">args</span>), <span class="pl-s1">kwargs</span>)
    <span class="pl-s1">gc</span>.<span class="pl-en">collect</span>()
    <span class="pl-k">return</span> <span class="pl-s1">results</span>

<span class="pl-en">pool_run</span>(<span class="pl-s1">processes</span><span class="pl-c1">=</span><span class="pl-c1">1</span>, <span class="pl-s1">print_usage</span><span class="pl-c1">=</span><span class="pl-c1">True</span>, <span class="pl-c1">**</span><span class="pl-s1">opts</span>)


<span class="pl-c"># ex is a wrapper that calls</span>

<span class="pl-k">def</span> <span class="pl-en">train_fn</span>(
    <span class="pl-s1">model</span>,                                   <span class="pl-c"># &lt;--- WALSModel</span>
    <span class="pl-s1">input_tensor</span>,                         <span class="pl-c"># tf.SparseTensor</span>
    <span class="pl-s1">iterations</span>:<span class="pl-s1">int</span>,
    <span class="pl-s1">sp_sparse</span><span class="pl-c1">=</span><span class="pl-c1">None</span>,                 <span class="pl-c"># sp.sparse.coo_matrix</span>
    <span class="pl-s1">sacred_run</span><span class="pl-c1">=</span><span class="pl-c1">None</span>,              
    <span class="pl-s1">print_progress</span>:<span class="pl-s1">bool</span><span class="pl-c1">=</span><span class="pl-c1">True</span>,
    <span class="pl-s1">logger</span><span class="pl-c1">=</span><span class="pl-c1">None</span>
):
    <span class="pl-s1">os</span>.<span class="pl-s1">environ</span>[<span class="pl-s">'KMP_DUPLICATE_LIB_OK'</span>]<span class="pl-c1">=</span><span class="pl-s">'True'</span>
    <span class="pl-c"># extract row and column factors</span>
    <span class="pl-s1">row_factor</span> <span class="pl-c1">=</span> <span class="pl-s1">model</span>.<span class="pl-s1">row_factors</span>[<span class="pl-c1">0</span>]
    <span class="pl-s1">col_factor</span> <span class="pl-c1">=</span> <span class="pl-s1">model</span>.<span class="pl-s1">col_factors</span>[<span class="pl-c1">0</span>]

    <span class="pl-c"># update operations</span>
    <span class="pl-s1">row_update_op</span> <span class="pl-c1">=</span> <span class="pl-s1">model</span>.<span class="pl-en">update_row_factors</span>(<span class="pl-s1">sp_input</span><span class="pl-c1">=</span><span class="pl-s1">input_tensor</span>)<span class="pl-c">#[1]</span>
    <span class="pl-s1">col_update_op</span> <span class="pl-c1">=</span> <span class="pl-s1">model</span>.<span class="pl-en">update_col_factors</span>(<span class="pl-s1">sp_input</span><span class="pl-c1">=</span><span class="pl-s1">input_tensor</span>)<span class="pl-c">#[1]</span>



    <span class="pl-k">with</span> <span class="pl-s1">tf</span>.<span class="pl-v">Session</span>() <span class="pl-k">as</span> <span class="pl-s1">sess</span>:
        <span class="pl-c"># init model</span>
        <span class="pl-s1">sess</span>.<span class="pl-en">run</span>(<span class="pl-s1">model</span>.<span class="pl-s1">initialize_op</span>)
        <span class="pl-s1">sess</span>.<span class="pl-en">run</span>(<span class="pl-s1">model</span>.<span class="pl-s1">worker_init</span>)

        <span class="pl-k">if</span> <span class="pl-s1">print_progress</span>:
            <span class="pl-s1">status</span> <span class="pl-c1">=</span> <span class="pl-v">Sil</span>(<span class="pl-s1">total</span><span class="pl-c1">=</span><span class="pl-s1">iterations</span>)
        <span class="pl-k">for</span> <span class="pl-s1">i</span> <span class="pl-c1">in</span> <span class="pl-en">range</span>(<span class="pl-s1">iterations</span>):
            <span class="pl-k">if</span> <span class="pl-s1">logger</span> <span class="pl-c1">is</span> <span class="pl-c1">not</span> <span class="pl-c1">None</span>: <span class="pl-s1">logger</span>.<span class="pl-en">debug</span>(<span class="pl-s">'Training iteration {}/{}'</span>.<span class="pl-en">format</span>(<span class="pl-en">str</span>(<span class="pl-s1">i</span>), <span class="pl-en">str</span>(<span class="pl-s1">iterations</span>)))
            <span class="pl-c"># update rows</span>
            <span class="pl-k">if</span> <span class="pl-s1">logger</span> <span class="pl-c1">is</span> <span class="pl-c1">not</span> <span class="pl-c1">None</span>: <span class="pl-s1">logger</span>.<span class="pl-en">debug</span>(<span class="pl-s">'Updating rows'</span>)   <span class="pl-c"># &lt;--- I see this in logs</span>
            <span class="pl-s1">sess</span>.<span class="pl-en">run</span>(<span class="pl-s1">model</span>.<span class="pl-s1">row_update_prep_gramian_op</span>)
            <span class="pl-s1">sess</span>.<span class="pl-en">run</span>(<span class="pl-s1">model</span>.<span class="pl-s1">initialize_row_update_op</span>)
            <span class="pl-s1">_</span>, <span class="pl-s1">_</span>, <span class="pl-s1">loss</span>, <span class="pl-s1">reg</span>, <span class="pl-s1">_</span> <span class="pl-c1">=</span> <span class="pl-s1">sess</span>.<span class="pl-en">run</span>(<span class="pl-s1">row_update_op</span>)
            <span class="pl-k">if</span> <span class="pl-s1">sacred_run</span> <span class="pl-c1">is</span> <span class="pl-c1">not</span> <span class="pl-c1">None</span>: <span class="pl-s1">sacred_run</span>.<span class="pl-en">log_scalar</span>(<span class="pl-s">"loss.row"</span>, <span class="pl-s1">loss</span>, <span class="pl-s1">i</span>)   

            <span class="pl-c"># update cols</span>
            <span class="pl-k">if</span> <span class="pl-s1">logger</span> <span class="pl-c1">is</span> <span class="pl-c1">not</span> <span class="pl-c1">None</span>: <span class="pl-s1">logger</span>.<span class="pl-en">debug</span>(<span class="pl-s">'Updating columns'</span>)   <span class="pl-c"># &lt;--- never makes it here</span>
            <span class="pl-s1">sess</span>.<span class="pl-en">run</span>(<span class="pl-s1">model</span>.<span class="pl-s1">col_update_prep_gramian_op</span>)
            <span class="pl-s1">sess</span>.<span class="pl-en">run</span>(<span class="pl-s1">model</span>.<span class="pl-s1">initialize_col_update_op</span>)
            <span class="pl-s1">_</span>, <span class="pl-s1">_</span>, <span class="pl-s1">loss</span>, <span class="pl-s1">reg</span>, <span class="pl-s1">_</span> <span class="pl-c1">=</span> <span class="pl-s1">sess</span>.<span class="pl-en">run</span>(<span class="pl-s1">col_update_op</span>)
            <span class="pl-k">if</span> <span class="pl-s1">sacred_run</span> <span class="pl-c1">is</span> <span class="pl-c1">not</span> <span class="pl-c1">None</span>: <span class="pl-s1">sacred_run</span>.<span class="pl-en">log_scalar</span>(<span class="pl-s">"loss.col"</span>, <span class="pl-s1">loss</span>, <span class="pl-s1">i</span>)

            <span class="pl-c"># update status</span>
            <span class="pl-k">if</span> <span class="pl-s1">print_progress</span>:
                <span class="pl-s1">status</span>.<span class="pl-en">tick</span>(<span class="pl-s1">prefix</span><span class="pl-c1">=</span><span class="pl-s">'iteration'</span>)

            <span class="pl-k">if</span> <span class="pl-s1">sacred_run</span> <span class="pl-c1">is</span> <span class="pl-c1">not</span> <span class="pl-c1">None</span> <span class="pl-c1">and</span> <span class="pl-s1">sp_sparse</span> <span class="pl-c1">is</span> <span class="pl-c1">not</span> <span class="pl-c1">None</span>:
                <span class="pl-k">if</span> <span class="pl-s1">logger</span> <span class="pl-c1">is</span> <span class="pl-c1">not</span> <span class="pl-c1">None</span>: <span class="pl-s1">logger</span>.<span class="pl-en">debug</span>(<span class="pl-s">'Calculating rmse.'</span>)
                <span class="pl-s1">rf</span> <span class="pl-c1">=</span> <span class="pl-s1">row_factor</span>.<span class="pl-en">eval</span>(<span class="pl-s1">session</span><span class="pl-c1">=</span><span class="pl-s1">sess</span>)
                <span class="pl-s1">cf</span> <span class="pl-c1">=</span> <span class="pl-s1">col_factor</span>.<span class="pl-en">eval</span>(<span class="pl-s1">session</span><span class="pl-c1">=</span><span class="pl-s1">sess</span>)
                <span class="pl-s1">it_rmse</span> <span class="pl-c1">=</span> <span class="pl-en">rmse</span>(<span class="pl-s1">sp_sparse</span>, <span class="pl-s1">rf</span>, <span class="pl-s1">cf</span>)
                <span class="pl-s1">sacred_run</span>.<span class="pl-en">log_scalar</span>(<span class="pl-s">"rmse"</span>, <span class="pl-s1">it_rmse</span>, <span class="pl-s1">i</span>)


        <span class="pl-c"># eval row / col factors</span>
        <span class="pl-s1">output_row</span> <span class="pl-c1">=</span> <span class="pl-s1">row_factor</span>.<span class="pl-en">eval</span>(<span class="pl-s1">session</span><span class="pl-c1">=</span><span class="pl-s1">sess</span>)
        <span class="pl-s1">output_col</span> <span class="pl-c1">=</span> <span class="pl-s1">col_factor</span>.<span class="pl-en">eval</span>(<span class="pl-s1">session</span><span class="pl-c1">=</span><span class="pl-s1">sess</span>)
        <span class="pl-s1">sess</span>.<span class="pl-en">close</span>()
    <span class="pl-s1">os</span>.<span class="pl-s1">environ</span>[<span class="pl-s">'KMP_DUPLICATE_LIB_OK'</span>]<span class="pl-c1">=</span><span class="pl-s">'False'</span>
    <span class="pl-k">return</span> <span class="pl-s1">output_row</span>, <span class="pl-s1">output_col</span></pre></div>
<p dir="auto"><strong>Code to reproduce the issue</strong><br>
Provide a reproducible test case that is the bare minimum necessary to generate the problem.</p>
<p dir="auto"><strong>Other info / logs</strong><br>
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.</p>
<h3 dir="auto">If possible, provide a minimal reproducible example (We usually don't have time to read hundreds of lines of your code)</h3>
<p dir="auto">x_nparray = $VERY_LARGE_NP_ARRAY(bigger than 2G)</p>
<p dir="auto">x = tf.Variable(init_value=x_nparray)</p>
<p dir="auto">saver=tf.Saver()<br>
with tf.Session() as sess:<br>
saver.save(sess, 'path/to/save')</p>
<p dir="auto">that would generate very large metagraph binary file, when i try to restore the model, it would failed because of the meta graph file size exceed  the protobuf limit size.</p>
<p dir="auto">I cannot reduce the size of x_nparray, because it's generated by other training system like caffe.</p>
<h3 dir="auto">What other attempted solutions have you tried?</h3>
<p dir="auto">set bigger limit byte of protobuf, however, the x_nparray is bigger than 3G, while the largest limit i can set is 2G.</p>