Question: Tensorflow tends to preallocate the entire available memory on it's GPUs. For debugging, is there a way of telling how much of that memory is actually in use?

Answer: 
Here's a practical solution that worked well for me: 

Disable GPU memory pre-allocation using TF session configuration: 

```
config = tf.ConfigProto()  
config.gpu_options.allow_growth=True  
sess = tf.Session(config=config)  

```

run **nvidia-smi -l** (or some other utility) to monitor GPU memory consumption.

Step through your code with the debugger until you see the unexpected GPU memory consumption.
```
tf.config.experimental.get_memory_info('GPU:0')

```

Currently returns the following keys:

```
'current': The current memory used by the device, in bytes.
'peak': The peak memory used by the device across the run of the program, in bytes.

```