<p dir="auto"><strong>System information</strong></p>
<ul dir="auto">
<li>TensorFlow version (you are using): 1.12</li>
<li>Are you willing to contribute it (Yes/No): Yes</li>
</ul>
<p dir="auto"><strong>Describe the feature and the current behavior/state.</strong><br>
Gradient Accumulation(GA) is a workaround to enable big batches on limited memory GPUs which has been supported in Caffe and PyTorch. Instead of back-propagating for every batch feed-forward; gradients across multiple batches are accumulated. After multiple feed forwards, the accumulated gradient is back-propagated through the network layer.  It boosts performance with a couple of percentages on our several workload (e.g., XLNet, Transformer) with Distribution Strategy. Unfortunately, I am not aware of any official documentation to use such feature in Tensorflow. <code class="notranslate">tf.contrib.opt.AGNOptimizer</code> has a similar GA implementation but is not common for general distribution job on DistributionStrategy. Besides it will introduces OOM for large embeddings(<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="481025609" data-permission-text="Title is private" data-url="https://github.com/tensorflow/tensorflow/issues/31637" data-hovercard-type="issue" data-hovercard-url="/tensorflow/tensorflow/issues/31637/hovercard" href="https://github.com/tensorflow/tensorflow/issues/31637">#31637</a>).<br>
<strong>Will this change the current api? How?</strong><br>
Yes. Just need to add one new parameter <code class="notranslate">iter_size</code> to current strategy API, which  for example:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content=" # For MirroredStrategy, gradient accumulation is supported using iter_size parameter.
 distribution = tf.distribute.MirroredStrategy(num_gpus=2, iter_size=4)

 # For MultiWorkerMirroredStrategy:
 tf.distribute.experimental.MultiWorkerMirroredStrategy(..., iter_size=4)"><pre class="notranslate"><code class="notranslate"> # For MirroredStrategy, gradient accumulation is supported using iter_size parameter.
 distribution = tf.distribute.MirroredStrategy(num_gpus=2, iter_size=4)

 # For MultiWorkerMirroredStrategy:
 tf.distribute.experimental.MultiWorkerMirroredStrategy(..., iter_size=4)
</code></pre></div>
<p dir="auto"><strong>Who will benefit with this feature?</strong><br>
Anyone who use distribution strategy and want to enable big batches on limited memory GPUs<br>
<strong>Any Other info.</strong><br>
None.</p>
<p dir="auto">We already have a concise implementation of GA on DistributionStrategy and are willing to contribute. Thanks in advance for any feedback.</p>
<p dir="auto"><strong>System information</strong></p>
<ul dir="auto">
<li>CentOS Linux release 7.4.1708</li>
<li>Source, hash: <a class="commit-link" data-hovercard-type="commit" data-hovercard-url="https://github.com/tensorflow/tensorflow/commit/bf4767c6bafb077fc591107691199a6981c29304/hovercard" href="https://github.com/tensorflow/tensorflow/commit/bf4767c6bafb077fc591107691199a6981c29304"><tt>bf4767c</tt></a></li>
<li>TensorFlow version: 1.12</li>
<li>Python version: Python 3.4.9</li>
<li>Installed using virtualenv? pip? conda?:</li>
<li>Bazel version (if compiling from source): 0.19.2 (bazel from centos repos)</li>
<li>GCC/Compiler version (if compiling from source): 4.8.5 (gcc from centos repos)</li>
<li>CUDA/cuDNN version: -</li>
<li>GPU model and memory: -</li>
</ul>
<p dir="auto"><strong>Describe the problem</strong></p>
<p dir="auto">Cannot  import tensorflow due to  import 'mock' library from tensorflow.python.platform.googletest  failling.</p>
<p dir="auto"><strong>Provide the exact sequence of commands / steps that you executed before running into the problem</strong></p>
<p dir="auto">Successfully built and installed TensorFlow with the following commands:</p>
<p dir="auto"><code class="notranslate">bazel build --incompatible_remove_native_http_archive=false --incompatible_package_name_is_a_function=false --config=opt //tensorflow/tools/pip_package:build_pip_package</code></p>
<p dir="auto"><code class="notranslate">./bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg</code></p>
<p dir="auto"><code class="notranslate">pip3.4 install --user /tmp/tensorflow_pkg/tensorflow-1.12.0-cp34-cp34m-linux_x86_64.whl</code></p>
<blockquote>
<p dir="auto">See pip log below for found dependencies</p>
</blockquote>
<p dir="auto">Attempting to import tensorflow i get this:</p>
<p dir="auto"><code class="notranslate">Python 3.4.9 (default, Aug 14 2018, 21:28:57) </code><br>
<code class="notranslate">[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux</code><br>
<code class="notranslate">Type "help", "copyright", "credits" or "license" for more information.</code><br>
<code class="notranslate">&gt;&gt;&gt; import tensorflow</code><br>
<code class="notranslate">/usr/lib64/python3.4/importlib/_bootstrap.py:321: FutureWarning: Conversion of the second argument of issubdtype from </code>float<code class="notranslate">to</code>np.floating<code class="notranslate"> is deprecated. In future, it will be treated as 'np.float64 == np.dtype(float).type'.</code><br>
<code class="notranslate">  return f(*args, **kwds)</code><br>
<code class="notranslate">Traceback (most recent call last):</code><br>
<code class="notranslate">  File "&lt;stdin&gt;", line 1, in &lt;module&gt;</code><br>
<code class="notranslate">  File "/home1/private/mavridis/.local/lib/python3.4/site-packages/tensorflow/__init__.py", line 34, in &lt;module&gt;</code><br>
<code class="notranslate">    from tensorflow._api.v1 import compat</code><br>
<code class="notranslate">  File "/home1/private/mavridis/.local/lib/python3.4/site-packages/tensorflow/_api/v1/compat/__init__.py", line 21, in &lt;module&gt;</code><br>
<code class="notranslate">    from tensorflow._api.v1.compat import v1</code><br>
<code class="notranslate">  File "/home1/private/mavridis/.local/lib/python3.4/site-packages/tensorflow/_api/v1/compat/v1/__init__.py", line 69, in &lt;module&gt;</code><br>
<code class="notranslate">    from tensorflow._api.v1.compat.v1 import test</code><br>
<code class="notranslate">  File "/home1/private/mavridis/.local/lib/python3.4/site-packages/tensorflow/_api/v1/compat/v1/test/__init__.py", line 24, in &lt;module&gt;</code><br>
<code class="notranslate">    from tensorflow.python.platform.googletest import mock</code><br>
<code class="notranslate">ImportError: cannot import name 'mock'</code></p>
<p dir="auto"><strong>Any other info / logs</strong></p>
<p dir="auto">pip install output:</p>
<p dir="auto"><code class="notranslate">Processing /tmp/tensorflow_pkg/tensorflow-1.12.0-cp34-cp34m-linux_x86_64.whl</code><br>
<code class="notranslate">Requirement already satisfied: grpcio&gt;=1.8.6 in /usr/lib64/python3.4/site-packages (from tensorflow==1.12.0) (1.10.0)</code><br>
<code class="notranslate">Requirement already satisfied: keras-preprocessing&gt;=1.0.5 in /home1/private/mavridis/.local/lib/python3.4/site-packages (from tensorflow==1.12.0) (1.0.5)</code><br>
<code class="notranslate">Requirement already satisfied: six&gt;=1.10.0 in /home1/private/mavridis/.local/lib/python3.4/site-packages (from tensorflow==1.12.0) (1.12.0)</code><br>
<code class="notranslate">Requirement already satisfied: absl-py&gt;=0.1.6 in /usr/lib/python3.4/site-packages (from tensorflow==1.12.0) (0.1.11)</code><br>
<code class="notranslate">Requirement already satisfied: google-pasta&gt;=0.1.1 in /home1/private/mavridis/.local/lib/python3.4/site-packages (from tensorflow==1.12.0) (0.1.1)</code><br>
<code class="notranslate">Requirement already satisfied: numpy&gt;=1.13.3 in /usr/lib64/python3.4/site-packages (from tensorflow==1.12.0) (1.14.1)</code><br>
<code class="notranslate">Requirement already satisfied: protobuf&gt;=3.6.1 in /home1/private/mavridis/.local/lib/python3.4/site-packages (from tensorflow==1.12.0) (3.6.1)</code><br>
<code class="notranslate">Requirement already satisfied: wheel&gt;=0.26 in /home1/private/mavridis/.local/lib/python3.4/site-packages (from tensorflow==1.12.0) (0.32.3)</code><br>
<code class="notranslate">Requirement already satisfied: tensorboard&lt;1.13.0,&gt;=1.12.0 in /home1/private/mavridis/.local/lib/python3.4/site-packages (from tensorflow==1.12.0) (1.12.2)</code><br>
<code class="notranslate">Requirement already satisfied: astor&gt;=0.6.0 in /usr/lib/python3.4/site-packages (from tensorflow==1.12.0) (0.6.2)</code><br>
<code class="notranslate">Requirement already satisfied: keras-applications&gt;=1.0.6 in /home1/private/mavridis/.local/lib/python3.4/site-packages (from tensorflow==1.12.0) (1.0.6)</code><br>
<code class="notranslate">Requirement already satisfied: tensorflow-estimator&lt;1.14.0rc0,&gt;=1.13.0rc0 in /home1/private/mavridis/.local/lib/python3.4/site-packages (from tensorflow==1.12.0) (1.13.0rc0)</code><br>
<code class="notranslate">Requirement already satisfied: gast&gt;=0.2.0 in /usr/lib/python3.4/site-packages (from tensorflow==1.12.0) (0.2.0)</code><br>
<code class="notranslate">Requirement already satisfied: termcolor&gt;=1.1.0 in /usr/lib/python3.4/site-packages (from tensorflow==1.12.0) (1.1.0)</code><br>
<code class="notranslate">Requirement already satisfied: setuptools in /home1/private/mavridis/.local/lib/python3.4/site-packages (from protobuf&gt;=3.6.1-&gt;tensorflow==1.12.0) (40.6.3)</code><br>
<code class="notranslate">Requirement already satisfied: werkzeug&gt;=0.11.10 in /usr/lib64/python3.4/site-packages (from tensorboard&lt;1.13.0,&gt;=1.12.0-&gt;tensorflow==1.12.0) (0.14.1)</code><br>
<code class="notranslate">Requirement already satisfied: markdown&gt;=2.6.8 in /usr/lib64/python3.4/site-packages (from tensorboard&lt;1.13.0,&gt;=1.12.0-&gt;tensorflow==1.12.0) (2.6.11)</code><br>
<code class="notranslate">Requirement already satisfied: h5py in /usr/lib64/python3.4/site-packages (from keras-applications&gt;=1.0.6-&gt;tensorflow==1.12.0) (2.7.1)</code><br>
<code class="notranslate">Requirement already satisfied: mock&gt;=2.0.0 in /home1/private/mavridis/.local/lib/python3.4/site-packages (from tensorflow-estimator&lt;1.14.0rc0,&gt;=1.13.0rc0-&gt;tensorflow==1.12.0) (2.0.0)</code><br>
<code class="notranslate">Requirement already satisfied: pbr&gt;=0.11 in /home1/private/mavridis/.local/lib/python3.4/site-packages (from mock&gt;=2.0.0-&gt;tensorflow-estimator&lt;1.14.0rc0,&gt;=1.13.0rc0-&gt;tensorflow==1.12.0) (5.1.1)</code><br>
<code class="notranslate">Installing collected packages: tensorflow</code><br>
<code class="notranslate">Successfully installed tensorflow-1.12.0</code></p>