So I get lots of questions from Academics when it comes to using Azure GPUs https://docs.microsoft.com/en-us/azure/virtual-machines/windows/sizes-gpu typically the most common is what's the best practice for monitoring GPU cores/RAM usage on N-series DSVM(s)?
So there are solutions like logging into each VM and run "watch nvidia-smi" but this simply is not scalable. But what would be great is to have a single it be great dashboard visual.
Well you now can thanks to some Microsoft colleagues Mathew Salvaris and Miguel Fierro. They have created an app for monitoring GPUs on a single machine and across a clusters.
You can use it to record various GPU measurements during a specific period using the context based loggers or continuously using the gpumon cli command. The context logger can either record to a file, which can be read back into a dataframe, or to an InfluxDB database.
Data from the InfluxDB database can then be accessed using the python InfluxDB client or can be viewed in realtime using dashboards such as Grafana.
Examples in Jupyter notebooks can be found here
Below is an example dashboard using the InfluxDB log context and a Grafana dashboard
You can download the installation and source from https://github.com/msalvaris/gpu_monitor