What you will monitor
A graphics processing unit, or GPU, is the chip that runs most machine learning and graphics work. Its health is easy to lose track of until a job slows down or fails. Monitoring gives you a steady record of how hard the chip works and whether it runs safely.
A small set of measurements covers most needs. Utilization shows how busy the compute cores are. Memory use shows how much video memory, sometimes called VRAM, is in use. Temperature and power draw show whether the card runs hot or pulls more energy than expected.
Collect these readings at regular intervals, not only when you connect to the host to check. A trend over hours or days tells you far more than a single snapshot.
Before you start
A few checks save time later. Work through them before you install anything on the host.
- Confirm that the host has a working GPU driver and that the GPU appears among the system's devices.
- Confirm that you have administrator access to install software and to open a network port on the host.
- Confirm that your provider allows you to run your own monitoring software, and check their documentation for any limits.
Setup steps
Work through these steps in order. Each one depends on the step before it.
- Install the GPU driver's management library on the host, so that software can read the GPU's values.
- Install a GPU metrics exporter, which is a small program that reads those values and serves them over a local web address.
- Start the exporter as a service, so it keeps running after a restart.
- Open the exporter's address in a browser on your private network, and confirm that utilization, memory, temperature and power values appear as text.
- Add a time-series data store, which is a database built for readings tagged with the time they were taken, and point it at the exporter so it collects readings at regular intervals.
- Add the same data store to a dashboard tool as a data source, and confirm that a test chart shows live values.
Keep the exporter's port off the public internet. Limit access to your private network or to a firewall rule that allows only your monitoring server.
Build the dashboard and alerts
A dashboard turns raw readings into charts you can scan quickly. Build a panel for each measurement, and start with a time range of a few hours. Once the charts look right, save the dashboard and share it with the people who run the hosts.
Alerts tell you when a reading crosses a line you set. Create a rule for sustained high temperature, for memory that stays close to its limit, and for a GPU that shows no utilization while a job should be running. Set each rule to wait for the condition to last a short while, so brief spikes do not wake anyone. Send alerts to a channel that someone actually reads.
What to do if it fails
If the exporter shows no GPU values, the driver or its management library is usually the cause. Restart the host if you have just installed a driver, then check again.
- Check that the driver loads and that the management library can see the device.
- Check the port setting and the firewall on the host if the exporter runs but its address shows nothing.
- Check that the data store can reach the exporter's address, and that the address in its configuration is correct, if charts have gaps.
- Compare suspicious readings with the values the driver reports directly, and check the metric names in the exporter's documentation.
If your provider runs the host for you, the driver and exporter may need their help. Check their own documentation before you change anything system-wide.
Where this matters when choosing a provider
Not every GPU host lets you install software or read hardware values. Some managed services hide the machine, so you only see the readings the provider chooses to share. Check that before you commit, and ask about the specific measurements you need.
Read the guide to choosing a GPU provider to compare hardware, access and billing. The GPU models entry explains the names providers use for their hardware. For a wider view, browse the GPU provider board.