Question: Tensorflow tends to preallocate the entire available memory on it's GPUs. For debugging, is there a way of telling how much of that memory is actually in use?

Answer: 
Here's a practical solution that worked well for me: 

Disable GPU memory pre-allocation using TF session configuration: 

```
config = tf.ConfigProto()  
config.gpu_options.allow_growth=True  
sess = tf.Session(config=config)  

```

run **nvidia-smi -l** (or some other utility) to monitor GPU memory consumption.

Step through your code with the debugger until you see the unexpected GPU memory consumption.
The TensorFlow profiler has improved memory timeline that is based on real gpu memory allocator information
<https://github.com/tensorflow/tensorflow/tree/master/tensorflow/core/profiler#visualize-time-and-memory>