gpuledger

What it answers

The join is per GPU: the reservations Nomad made, the tenants that actually hold it, and whether each tenant was reserved. From that, the findings:

Worst first. The exit code is 0 whatever the findings — a check that ran is a success — unless --exit-on warn|bad|error asks for one, for CI and for check runners.

Run it

Once, on a node:

gpuledger ls
gpuledger check --json
gpuledger check --exit-on bad --allow-unmanaged

As a system job, one per GPU node, scraping /metrics on 9877: deploy/nomad/gpuledger.nomad.hcl. It uses raw_exec on purpose: the ledger reads the host's nvidia-smi, /proc and Docker socket, which a container would have to be handed anyway.

V=0.3.0
curl -fsSLO https://github.com/HiWay-Media/gpuledger/releases/download/v$V/gpuledger-v$V-checksums.txt
nomad job run -var version=$V \
  -var checksum=sha256:$(grep linux-amd64 gpuledger-v$V-checksums.txt | cut -d' ' -f1) \
  deploy/nomad/gpuledger.nomad.hcl
curl -s http://<node>:9877/metrics | grep gpuledger_gpu_tenants

Metrics: gpuledger_up, gpuledger_findings{code,level}, gpuledger_worst_level, gpuledger_gpu_info{model,bus}, _utilization_percent, _memory_used_bytes, _memory_total_bytes, _temperature_celsius, _power_watts, _encoder_sessions, _thermal_margin_celsius, _thermal_slowdown, _tenants, _reservations, gpuledger_gpu_state{state} and, with a history, gpuledger_gpu_state_since_timestamp_seconds{state} — alert on time() - gpuledger_gpu_state_since_timestamp_seconds{state="reserved-idle"} > 6*3600 — and per tenant gpuledger_tenant_memory_bytes{kind,container,container_id,nomad_job,task,alloc,namespace} and gpuledger_tenant_reserved. Labels carry names and ids, never a process name, a pid, an image or a path. /ledger and /findings return the same as JSON; /healthz is 503, with the failing sources in the body, while a source is unreadable.

Since when. With --history FILE, serve remembers each GPU's state — free, reserved-idle, held or unaccounted — and when it entered it, so reserved-idle says for 6h12m, the number a scheduling decision needs. One record per GPU, no tenant in it; written atomically each refresh. ls and check with the same flag read the file and never write it. A silence longer than three refresh intervals, or a refresh where a source failed, restarts the clock rather than claim a continuity nobody saw. The system job keeps the file in its allocation's data dir on a sticky disk, so it survives a deploy on the same node.

The fleet, from anywhere that reaches the nodes: gpuledger fleet reads every node's /ledger — found through Consul, where the system job registers as gpuledger, or listed with --targets — and counts each GPU as exactly one of held, reserved-idle, unaccounted (a tenant Nomad did not reserve the card for, or cannot vouch for) or free, per node and per job. fleet check evaluates every node with one policy, worst first; a node that cannot be read is a source-unavailable ERROR, not a failure of the command.

gpuledger fleet --consul consul.service:8500
gpuledger fleet ls --targets gpua:9877,gpub:9877 --json
gpuledger fleet check --exit-on bad

Illustrative output, the shape fleet ls prints:

gpuledger fleet · 7 node(s), 1 unreachable · 12 GPU(s)
│ node  │ gpus │ held │ reserved-idle │ unaccounted │ free │ memory              │
├───────┼──────┼──────┼───────────────┼─────────────┼──────┼─────────────────────┤
│ gpua  │ 2    │ 1    │ 1             │ 0           │ 0    │ 2048/16384 MiB      │
│ gpud  │ 2    │ 0    │ 1             │ 1           │ 0    │ 3120/16384 MiB      │
│ …     │      │      │               │             │      │                     │
│ gpug  │ —    │ —    │ —             │ —           │ —    │ unreachable: …      │
│ fleet │ 12   │ 5    │ 3             │ 2           │ 2    │ 21504/98304 MiB     │

│ job                          │ reserved │ held │
├──────────────────────────────┼──────────┼──────┤
│ default/gpu-gpud-restreamer  │ 1        │ 1    │
│ default/tngrm-video-worker   │ 3        │ 1    │

Consul over TLS works the same way as Nomad's: --consul https://… (or a bare address with CONSUL_HTTP_SSL=true), --consul-ca-cert, --consul-ca-path, --consul-client-cert, --consul-client-key, --consul-tls-server-name, each defaulting to the Consul CLI's variable (CONSUL_CACERT, …).

Without Consul, Nomad's own service discovery (1.3+) does the same job: set provider = "nomad" in the system job's service block, then gpuledger fleet --nomad-service gpuledger — with the same --nomad-addr, token and TLS as the rest, and the same policy file (listing a service needs read-job, which it has). The job spec leaves provider out, so it still validates on Nomad before 1.3.

It asks the nodes' gpuledger only, never their Nomad, Docker or driver: each node is seen exactly as its own agent sees it. --consul defaults to $CONSUL_HTTP_ADDR, the Consul token comes from the variable named by --consul-token-env (CONSUL_HTTP_TOKEN), --consul-service gpuledger, --timeout 5s per node.

Flags: --nomad-addr (default $NOMAD_ADDR or http://127.0.0.1:4646), --nomad-token-env NOMAD_TOKEN — the name of the variable holding the ACL token, so the token is never on a command line — --docker unix:///var/run/docker.sock, --podman auto, --nvidia-smi, --proc /proc, --node, --json, --no-nomad, --no-docker, --encoder-max, --temp-max, --allow-unmanaged, --no-idle, --listen, --interval, --history; for fleet, --targets, --nomad-service, --nomad-namespace, --consul, --consul-service, --consul-token-env, --timeout.

Alerts and a dashboard, for Prometheus and Grafana: deploy/prometheus/gpuledger.rules.yml — source down, an unreserved or unmanaged tenant, a card hot or thermally slowing down, reserved and idle for over six hours — driven by gpuledger_findings{code,level}, the same verdict check gives, with a zero for every code so > 0 always has a series; gpuledger_worst_level is check's exit-code scale. The rules are unit-tested with promtool test rules. deploy/grafana/gpuledger.json imports into any Grafana with a Prometheus datasource; in CI it is imported into the latest Grafana, and every one of its queries — and every rule — runs against a real Prometheus scraping gpuledger on real Nomad allocations.

With ACLs, the token needs deploy/nomad/gpuledger.policy.hcl: agent:read, node:read and read-job on the namespaces that run GPU jobs. The last one is easy to miss: without it Nomad answers the node's allocations without that namespace's, and no error — gpuledger reports source-unavailable for each Nomad container whose allocation it cannot see rather than calling it unreserved.

Without a static token, on Nomad 1.5+: the task's workload identity is its token. Bind the same policy to the job and run the workload identity variant of the job:

nomad acl policy apply -namespace default -job gpuledger gpuledger-job deploy/nomad/gpuledger.policy.hcl
nomad job run -var version=V -var checksum=sha256:… deploy/nomad/gpuledger.wi.nomad.hcl

It passes --nomad-node-id ${node.unique.id} — both job specs do — because before Nomad 1.11 the local agent's /v1/agent/self refuses a workload identity (HTTP 500); with the node id given, that call is not made. Nomad 1.4 has job-bound policies but gives the task no token; before 1.4 neither exists, and the static token is the way.

Over mutual TLS, an agent with tls { http = true } and verify_https_client: --nomad-addr https://127.0.0.1:4646 and the certificates by path — --nomad-ca-cert (or --nomad-ca-path), --nomad-client-cert, --nomad-client-key, --nomad-tls-server-name — each defaulting to the variable the Nomad CLI reads (NOMAD_CACERT, NOMAD_CAPATH, NOMAD_CLIENT_CERT, NOMAD_CLIENT_KEY, NOMAD_TLS_SERVER_NAME), so a shell already set up for nomad works as is. Key material never goes in a flag. A refused handshake is a source-unavailable that says TLS; half a key pair is refused before any request.

Podman is read next to Docker when /run/podman/podman.sock exists (--podman off to skip it, or an endpoint of your own); a Podman endpoint you name that cannot be read is a source-unavailable, the default one that does not exist is simply no Podman.

Turn on the docker driver's extra_labels (job_name, task_name, namespace) on GPU nodes: without them a Nomad tenant is known by its allocation id and container name only.

The JSON contract

ls --json, check --json, /ledger, /findings, fleet --json and the history file carry "schema": 1. The rule, for anything that reads them:

• Adding a field keeps the schema. Ignore the fields you do not know.
• Removing or renaming a field, or changing its type or its meaning, bumps it — and the CHANGELOG says so under the release.
• schema absent (or 0) is a gpuledger before 0.3; fleet shows such a node as gpua (schema 0), and judges it from what its ledger has.

The reference is testdata/golden/: the full output of ls, check and fleet ls on the test fixtures. Any change to it fails the tests until it is regenerated on purpose (go test ./cmd/gpuledger -run Golden -update), so no field changes by accident.

Per-card numbers

What the driver does not report, gpuledger takes from NVIDIA's Video Encode and Decode GPU Support Matrix, read 2026-09-25 (internal/cards):

So on the farm's cards a session count is never a limit, and encoder-saturated stays silent unless --encoder-max N asks for one; -1 turns it off everywhere.

Thermal limits are not in the table: the datasheets give only the ambient range (0 – 50 °C for L4, T4 and A10), and each card's own slowdown temperature comes from the driver. gpuledger asks for it with two optional queries, each on its own so a driver that lacks a field never breaks the main one: temperature.gpu.tlimit (°C left before slowdown) and the hardware and software thermal slowdown flags, under clocks_event_reasons.* and, on drivers before the rename, clocks_throttle_reasons.*. Both are in /ledger and in /metrics as gpuledger_gpu_thermal_margin_celsius and gpuledger_gpu_thermal_slowdown — only when the driver reports them.

Compatibility

The Nomad matrix runs on every change and weekly: the latest patch of every Nomad minor from 1.0, read from releases.hashicorp.com at run time, plus 1.7.3. On each, integration/nomad_test.go starts nomad agent -dev with ACLs on (and, in a test of its own, with its API over mTLS, verify_https_client and verify_server_hostname on) and Nomad's example device plugin rebuilt as vendor nvidia, type gpu, runs two Docker jobs that ask for device "nvidia/gpu" — one in a second namespace — and a container outside Nomad, and checks with the policy file's token that gpuledger says held, reserved-idle, unreserved-tenant, unmanaged-tenant and contended on the right cards; that a token without read-job or without node:read is an ERROR naming it; that /metrics passes promtool check metrics; that the system job validates; and that fleet counts the node's jobs right, directly and through a Consul dev agent's health API with the /healthz check the system job declares — and once more over HTTPS with verify_incoming. A second test runs a task under the latest nomad-driver-podman (0.6.5), without extra_labels, and checks the card comes out held from the container's name alone.

What the matrix established, on every version: the docker driver labels containers com.hashicorp.nomad.alloc_id; /v1/node/<id>/allocations filters by the token's read-job without an error; the task's cgroup is /system.slice/docker-<id>.scope or, on 1.3 – 1.6, /nomad.slice/docker-<id>.scope. Under the podman driver: no label at all without extra_labels, the name <task>-<alloc id>, the cgroup /nomad.slice/libpod-<id>.scope/container — on every version, 1.0 included.

What it does not do

• Write anything: no nomad, docker or nvidia-smi command that changes state is ever invoked. Reads only — the one file it writes is its own --history, when asked to.
• Bind NVML. Shelling out to nvidia-smi's documented CSV queries keeps the binary static and the driver dependency out of the build; the cost is one process spawn per refresh, tens of milliseconds.
• See encoder-only sessions as processes. NVENC sessions without a CUDA context appear in encoder.stats.sessionCount but not always in query-compute-apps; the count is reported per GPU, the owner is attributed only when a process is visible.
• Read containerd directly (Nomad's community containerd driver): its API is gRPC, which the standard library does not speak — GL-23.
• Know about MIG partitions or Kubernetes. Nomad and plain Docker; the fleet view reads the nodes' own ledgers, it does not replace them.

Install

Static binaries for linux/amd64 and linux/arm64 on the releases page, with a checksums file and — from the release after 0.2.0 — build provenance you can verify: that the binary was built by this repository's release workflow from the tagged commit.

V=0.3.0
gh release download v$V --repo HiWay-Media/gpuledger
sha256sum -c gpuledger-v$V-checksums.txt
gh attestation verify gpuledger-v$V-linux-amd64 --repo HiWay-Media/gpuledger \
  --signer-workflow HiWay-Media/gpuledger/.github/workflows/release.yml

The release workflow runs that same verification before it publishes, and a dry run of it — build, attest, verify, and a tampered binary that must not verify — runs on every change to the workflow. The system job takes the checksum as a required variable, so Nomad refuses a download that does not match: -var checksum=sha256:$(grep linux-amd64 gpuledger-v$V-checksums.txt | cut -d' ' -f1). From source: go install github.com/hiway-media/gpuledger/cmd/gpuledger@latest (Go 1.27). Requires nvidia-smi on the node and, for tenant resolution, read access to the Docker socket and /proc.

Prior art

• nomad-device-nvidia: the plugin that fingerprints and schedules; gpuledger reads what it scheduled and compares it with what runs.
• dcgm-exporter: per-GPU telemetry with pod labels on Kubernetes; no Nomad, no notion of an unmanaged tenant.
• checkfleet: the same findings contract — worst first, exit 0 by default — for a fleet of domain checks.