Question: I have 10 servers running on Ubuntu 14.04 x64. Each server has a few Nvidia GPUs. I am looking for a monitoring program that would allow me to view the GPU usage on all servers at a glance.

Answer: 
[munin](http://munin-monitoring.org/) has at least one [plugin](http://munin-monitoring.org/browser/munin-contrib/plugins/gpu/nvidia_gpu_) for monitoring nvidia GPUs (which uses the `nvidia-smi` utility to gather its data).

You could setup a `munin` server (perhaps on one of the GPU servers, or on the head node of your cluster), and then install the `munin-node` client and the nvidia plugin (plus whatever other plugins you might be interested in) on each of your GPU servers.

That would allow you to look in detail at the munin data for each server, or see an overview of the nvidia data for all servers. This would include graphs charting changes in, e.g., GPU temperature over time

Otherwise you could write a script to use ssh (or [pdsh](https://computing.llnl.gov/linux/pdsh.html)) to run the `nvidia-smi` utility on each server, extract the data you want, and present it in whatever format you want.
Or simply use

<https://github.com/PatWie/cluster-smi>

which acts exactly in the same way as `nvidia-smi` in the terminal but gathers all information of nodes across your cluster, which are running the `cluster-smi-node`. The output will be

```
+---------+------------------------+---------------------+----------+----------+
| Node    | Gpu                    | Memory-Usage        | Mem-Util | GPU-Util |
+---------+------------------------+---------------------+----------+----------+
| node-00 | 0: TITAN Xp            |  3857MiB / 12189MiB | 31%      | 0%       |
|         | 1: TITAN Xp            | 11689MiB / 12189MiB | 95%      | 0%       |
|         | 2: TITAN Xp            | 10787MiB / 12189MiB | 88%      | 0%       |
|         | 3: TITAN Xp            | 10965MiB / 12189MiB | 89%      | 100%     |
+---------+------------------------+---------------------+----------+----------+
| node-01 | 0: TITAN Xp            | 11667MiB / 12189MiB | 95%      | 100%     |
|         | 1: TITAN Xp            | 11667MiB / 12189MiB | 95%      | 96%      |
|         | 2: TITAN Xp            |  8497MiB / 12189MiB | 69%      | 100%     |
|         | 3: TITAN Xp            |  8499MiB / 12189MiB | 69%      | 98%      |
+---------+------------------------+---------------------+----------+----------+
| node-02 | 0: GeForce GTX 1080 Ti |  1447MiB / 11172MiB | 12%      | 8%       |
|         | 1: GeForce GTX 1080 Ti |  1453MiB / 11172MiB | 13%      | 99%      |
|         | 2: GeForce GTX 1080 Ti |  1673MiB / 11172MiB | 14%      | 0%       |
|         | 3: GeForce GTX 1080 Ti |  6812MiB / 11172MiB | 60%      | 36%      |
+---------+------------------------+---------------------+----------+----------+

```

when using 3 nodes.

It uses NVML to read these values directly for efficiency. I suggest to **not** parse the output of `nvidia-smi` as proposed in the other answers.
Further, you can track these information from `cluster-smi` using Python+ZMQ.