@stas - again, much appreciate your input here. Appreciate your time helping me diagnose this.
I’ll describe the setup:

GPU cluster with a broad mix of different gpu types (Tesla K40m, GeForce Titan X, GeForce GTX Titan X, GeForce Titan X (Pascal))
Slurm job scheduler to coordinate job submission:

There are many users and my job will begin after another job has just finished
When my job begins, I have exclusive access to that GPU - the GPUs are only ever used by one user’s job at a time


It’s a service locally hosted by my university, so I can submit support tickets etc. I have reported the issue and we are struggling to fix. I’m here because I’m trying to find a simple workaround!

At the beginning of the job I report the usage with the tool GPUtil 41  - but this uses nvidia-smi under the hood. The usage reported is always 0 - as expected, e.g.:
| ID | GPU | MEM |
------------------
|  0 |  0% |  0% |

I know that my variable is smaller than the available RAM because I’ve measured the size of my model (it’s a few megabytes), and because the error message is slightly different from yours; mine follows the format - tried to allocate {small_number} ... {much_larger_number} free; ...). For example:
RuntimeError: CUDA out of memory. 
Tried to allocate 4.50 MiB (GPU 0; 11.91 GiB total capacity;
213.75 MiB already allocated; 11.18 GiB free; 509.50 KiB cached)

This is what has led me to the conclusion that the GPU has not been properly cleared after a previously running job has finished.
Your proposed solution to bail if there isn’t enough RAM at the start will not work - there is enough RAM according to nvidia-smi and indeed the error message. I imagine there is not enough contiguous memory!
Regardless, to fix, I think all I need to do is to clear the GPU’s memory at the beginning of my job (or simply wait until this is done). Is there a way to force this?
Alternatively, it could be that the GPU is clear, but the first variable is sent to the GPU memory in an extremely fragmented way. Is there any reason why this would happen?