Why Tensorflow does NOT quit when CUDA_ERROR_OUT_OF_MEMORY
It was my understanding that when trying to create a tensorflow session, if the amount of available GPU memory is not enough for what the program asks for, the program should raise an error and exit. But it seems to be not this case, and this behavior is very annoying and hard to detect especially when using Jupyter Notebook.
Code
I was trying to run the following program (supposedly debug.py)
import tensorflow as tf
a1 = tf.constant([0.0,0.0])
b1 = tf.constant([2.0,2.0])
mse1, update1 = tf.metrics.mean_squared_error(a1,b1)
mse2 = tf.losses.mean_squared_error(a1,b1)
with tf.Session() as sess:
sess.run(tf.local_variables_initializer())
update1.eval()
print(mse1.eval())
print(mse2.eval())
This program simply calculate the mean_squared_error (MSE) between the two vectors a1 and b1, and the correct output (from the two print() lines) should be 4.0 and 4.0. If no other python programs are using my GPU, this is indeed the output.
Unexpected behavior
If another program is using the GPU (say, another jupyter notebook running something with tensorflow without limiting its GPU usage by gpu_options.per_process_gpu_memory_fraction), then the above code would output something different. 695
The first print line (mse1) would output some random number, e.g. "0.0" (most common), "3e-45", "nan", etc, which is obviously wrong. The second print line (mse2) always print the correct answer: 4.0.
Running this code by python debug.py SOMETIMES generate an error message
E tensorflow/stream_executor/cuda/cuda_driver.cc:924] failed to allocate 176.44M (185008128 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
But the program does NOT exit, and still prints the results, although incorrect.
In other cases, running this code by python debug.py generate another error message
E tensorflow/core/common_runtime/direct_session.cc:170] Internal: failed initializing StreamExecutor for CUDA device ordinal 0: Internal: failed call to cuDevicePrimaryCtxRetain: CUDA_ERROR_OUT_OF_MEMORY; total memory reported: <PHONE_NUMBER>
and the program quits without printing any output.
The most annoying thing is, running the same code in Jupyte Notebook ALWAYS generates no error or warning messages, though the output is incorrect. That's why I always thought I misunderstood about how to call tf.metrics.mean_squared_error without realizing there could be a memory error.
System
I verified the problem with my two computers running Tensorflow 1.3 and 1.4 respectively, both with CUDA 8, Ubuntu 16.04 and Nvidia GPUs (1080 and 770M).
My Questions
(0) Is there anything wrong with my code?
(1) If you also use Tensorflow with Jupyter Notebook, do you ever get the wrong output (printing anything other than 4.0, 4.0) when running the above code? If so, does Jupyter alerts you about any error?
(2) What is the nature of this problem of returning the wrong mse1? Is it a GPU memory leak?
(3) Why could there be such inconsistent behavior when the problem occurs? To be specific, why the debug.py sometimes quits, sometimes not? Why Jupyter Notebook never quits or raises an error?
(4) Why the mse2 never goes wrong when mse1 goes wrong? My understanding is that tf.metrics.mean_squared_error is a streaming operation and therefore need to store some variable in a different place. 695