<p dir="auto"><em>Please make sure that this is a feature request. As per our <a href="https://github.com/tensorflow/tensorflow/blob/master/ISSUES.md">GitHub Policy</a>, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:feature_template</em></p>
<p dir="auto"><strong>System information</strong></p>
<ul dir="auto">
<li>TensorFlow version (you are using): 1.12.0</li>
<li>Are you willing to contribute it (Yes/No): Yes</li>
</ul>
<p dir="auto"><strong>Describe the feature and the current behavior/state.</strong></p>
<p dir="auto">The STFT Tensorflow function takes into consideration the Hermitian symmetry of the generated output i.e. it only considers the positive frequency elements. Is there a way to recover the entire pre-processed tensor that contains all the calculated elements?</p>
<p dir="auto">In other words, is there a way to calculate the STFT of an input tensor and generate a spectrogram that has dimensions <code class="notranslate">fft_length * frame_length</code>?</p>
<p dir="auto"><strong>Will this change the current api? How?</strong></p>
<p dir="auto">This will just add an additional feature to the existing function.</p>
<p dir="auto"><strong>Who will benefit with this feature?</strong></p>
<p dir="auto">Anyone who is doing audio processing using Tensorflow.</p>
<p dir="auto"><strong>Any Other info.</strong></p>
<h3 dir="auto">System information</h3>
<ul dir="auto">
<li><strong>Have I written custom code (as opposed to using a stock example script provided in TensorFlow)</strong>: No, running tf_cnn_benchmarks.py from benchmarks repo</li>
<li><strong>OS Platform and Distribution (e.g., Linux Ubuntu 16.04)</strong>: Ubuntu 16.04.2 LTS</li>
<li><strong>TensorFlow installed from (source or binary)</strong>: Unmodified source with RDMA Verbs enabled</li>
<li><strong>TensorFlow version (use command below)</strong>: 1.3.0-rc0</li>
<li><strong>Python version</strong>: 2.7.12</li>
<li><strong>Bazel version (if compiling from source)</strong>: 0.5.1</li>
<li><strong>CUDA/cuDNN version</strong>: 8.0/6</li>
<li><strong>GPU model and memory</strong>: NVIDIA Tesla P100 PCIe 16GB (8 per node)</li>
<li><strong>Exact command to reproduce</strong>:</li>
</ul>
<p dir="auto">PS: CUDA_VISIBLE_DEVICES='' python tf_cnn_benchmarks.py --ps_hosts 12.12.12.43:20000 --worker_hosts 12.12.12.44:20000,12.12.12.41:20000 --batch_size=64 --model=inception3 --variable_update=parameter_server --local_parameter_device=cpu --job_name=ps --task_index=0 --server_protocol grpc+verbs</p>
<p dir="auto">Worker0: CUDA_VISIBLE_DEVICES='0,1,2,3,4,5,6,7' python tf_cnn_benchmarks.py --ps_hosts 12.12.12.43:20000 --worker_hosts 12.12.12.44:20000,12.12.12.41:20000 --batch_size=64 --model=inception3 --variable_update=parameter_server --local_parameter_device=cpu --job_name=worker --task_index=0 --num_gpus=8 --data_dir=/data/imagenet_data/ --train_dir=/data/imagenet_train/ --server_protocol grpc+verbs</p>
<p dir="auto">Worker1: CUDA_VISIBLE_DEVICES='0,1,2,3,4,5,6,7' python tf_cnn_benchmarks.py --ps_hosts 12.12.12.43:20000 --worker_hosts 12.12.12.44:20000,12.12.12.41:20000 --batch_size=64 --model=inception3 --variable_update=parameter_server --local_parameter_device=cpu --job_name=worker --task_index=1 --num_gpus=8 --data_dir=/data/imagenet_data/ --train_dir=/data/imagenet_train/ --server_protocol grpc+verbs</p>
<ul dir="auto">
<li><strong>RDMA driver version</strong>: MLNX_OFED_LINUX-4.1-1.0.2.0</li>
</ul>
<h3 dir="auto">Describe the problem</h3>
<p dir="auto">When running the above commands (Inception V3 synchronized data parallelism training with 2 workers and 1 external ps), the tf_cnn_benchmarks application hangs forever after some iterations (usually in warm up).</p>
<p dir="auto">It happens only when real data is involved (ImageNet), and with &gt;4 GPUs. (More GPUs, less iterations before it hangs). Doesn't happen with grpc protocol, or when running with "synthetic" data.</p>
<p dir="auto">The master_service in the workers is stuck <a href="https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/distributed_runtime/master_session.cc#L608">here</a>, which I guess means some operations in the computation have not been completed.</p>
<p dir="auto">The RDMA protocol looks valid and clean, all messages corresponds to the protocol (see below logs).<br>
There some tensors requested by the workers which they don't receive, but they are passed by the RDMA Verbs transport to the BaseRendezvoudMgr with RecvLocalAsync in a valid way, and for some reason the higher level worker service doesn't trigger the Send kernel on those tensors.</p>
<p dir="auto">Any help is much appreciated!<br>
If there are some debug mechanisms I can use to understand which tensors/operations have not been completed it can greatly help. I was mostly debugging this from the RDMA Verbs layer till now, without much success, and I feel I don't have enough information there to understand what's missing.<br>
Also I feel we don't have enough knowledge on how the step_id acts (diving into this in the code now, but there's some higher level documentation it can greatly help).</p>
<p dir="auto">My initial guess was an occurrence of a racy condition when loading the data, since it creates a gap in execution time (worker0 starts the first training step 30-60 seconds after worker1, since it does the preprocessing of the data twice for a reason I couldn't understand yet), but after the first iteration (which usually passes successfully) the time is synchronized between workers.</p>
<h3 dir="auto">Source code / logs</h3>
<p dir="auto">Those are the logs of the runtime after moving the logging in <a href="https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/verbs/rdma.cc">rdma.cc</a> to VLOG(0) (also adding Tensor name and step id for all cases, in some cases the step_id doesn't mean anything like BUFFER_REQUEST/RESPONSE for example), and also some VLOG in <a href="https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/distributed_runtime/master_session.cc">master_session.cc</a></p>
<p dir="auto"><a href="https://gist.github.com/shamoya/15a42f421e088473b8f02bf00c16d0fc">worker0</a><br>
<a href="https://gist.github.com/shamoya/dd3126c02c73990a6e28b534d9a9ddf6">worker1</a><br>
<a href="https://gist.github.com/shamoya/0c856365802ae4d42b38baf988149574">ps</a></p>
<p dir="auto">Unfortunately they are fairly large, but it's better then to cut the log files IMO.<br>
Example for analysis I did in the verbs layer, comparing the Sent Tensor requests to the actual received tensors writes in both workers:</p>
<p dir="auto">worker 0:</p>
<ul dir="auto">
<li>/job:ps/replica:0/task:0/cpu:0;f3c10d28b54074c0;/job:worker/replica:0/task:0/gpu:0;edge_116943_group_deps_2/NoOp_1;0:0 80661058974090965</li>
<li>/job:worker/replica:0/task:1/cpu:0;1a50d5c51cd9c5d1;/job:worker/replica:0/task:0/gpu:0;edge_116947_group_deps_3/NoOp_1;0:0 80661058974090965</li>
<li>/job:worker/replica:0/task:1/gpu:2;7f00fadabfe781f5;/job:worker/replica:0/task:0/gpu:0;edge_111078_group_deps_1/NoOp_2;0:0 80661058974090965</li>
<li>/job:worker/replica:0/task:1/gpu:4;b07185dd19f62088;/job:worker/replica:0/task:0/gpu:0;edge_111080_group_deps_1/NoOp_4;0:0 80661058974090965</li>
</ul>
<p dir="auto">worker 1:</p>
<ul dir="auto">
<li>/job:ps/replica:0/task:0/cpu:0;f3c10d28b54074c0;/job:worker/replica:0/task:1/cpu:0;edge_155113_AssignAdd;0:0 80661058974090965</li>
<li>/job:worker/replica:0/task:0/gpu:0;f3df8abf03739fe8;/job:worker/replica:0/task:1/cpu:0;edge_116948_group_deps_3;0:0 80661058974090965</li>
</ul>
<p dir="auto">The tensors requests received well by the other side and passed to RecvLocalAsync, but are not called later.</p>
<p dir="auto">Thanks a lot.</p>