Question: Tensorflow tends to preallocate the entire available memory on it's GPUs. For debugging, is there a way of telling how much of that memory is actually in use?

Answer: 
Here's a practical solution that worked well for me: 

Disable GPU memory pre-allocation using TF session configuration: 

```
config = tf.ConfigProto()  
config.gpu_options.allow_growth=True  
sess = tf.Session(config=config)  

```

run **nvidia-smi -l** (or some other utility) to monitor GPU memory consumption.

Step through your code with the debugger until you see the unexpected GPU memory consumption.
as @V.M previously mentioned, a solution that works well is using: `tf.config.experimental.get_memory_info('DEVICE_NAME')`

This function returns a dictionary with two keys:

* **'current'**: The current memory used by the device, in bytes
* **'peak'**: The peak memory used by the device across the run of the program, in bytes.
The value of these keys is the ACTUAL memory used not the allocated one that is returned by nvidia-smi.

In reality, for GPUs, TensorFlow will allocate all the memory by default rendering using **nvidia-smi** to check for the used memory in your code useless. Even if, `tf.config.experimental.set_memory_growth` is set to true, Tensorflow will no more allocate the whole available memory but is going to remain in allocating **more memory than the one is used** and in a **discrete manner**, i.e. allocates 4589MiB then 8717MiB then 16943MiB then 30651 MiB, etc.

A small note concerning the `get_memory_info()` is that it **doesn't return correct values if used in a `tf.function()` decorated function**. Thus, the peak key shall be used after executing `tf.function()` decorated function to determine the peak memory used.

For **older versions** of Tensorflow, `tf.config.experimental.get_memory_usage('DEVICE_NAME')` was the only available function and only returned the used memory (no option for determining the peak memory).

Final note, you can also consider the **Tensorflow Profiler available with Tensorboard** as @Peter Mentioned.

Hope this helps :)