Prometheus + Grafana Monitoring: Making LLM Services Visible

An LLM process that “looks fine” on CPU is often already serving a worse model, a slower first token, or a truncated completion. Prometheus + Grafana will show requests; a default node dashboard will not show tokens, TTFT, or a model label exploding cardinality.

The path is scrape → query → rules → panels. Every llm_* name below is an example metric name, not a standard export from Prometheus or from any particular gateway. Match /metrics before copying queries.

Choose / skip

Scrape and relabel

In the official scrape_config, relabel_configs rewrite target labels before the scrape; metric_relabel_configs rewrite sample labels after. Unbounded labels have to die in the second layer. Dropping them after they hit disk is too late.

Sketch prometheus.yml (host, port, paths follow the deploy; this file has not been run on this box):

global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - /etc/prometheus/rules/*.yml

scrape_configs:
  - job_name: llm-gateway
    metrics_path: /metrics
    scrape_interval: 15s
    scrape_timeout: 10s
    static_configs:
      - targets:
          - 127.0.0.1:8000
        labels:
          service: llm-gateway
    relabel_configs:
      - source_labels: [__address__]
        regex: "([^:]+):\\d+"
        target_label: instance
        replacement: "${1}"
    metric_relabel_configs:
      - regex: "user_id|prompt_id|request_id|session_id"
        action: labeldrop

Keep job stable. Strip the port from __address__ into instance so a rolling restart that changes the port does not mint a new series. Edit the labeldrop list to match whatever the library leaked; a prompt-text label is not worth keeping.

Kubernetes belongs on kubernetes_sd_configs plus __meta_kubernetes_* relabel, not a handwritten static_configs that rot. Static is fine for a single process or compose.

Instrumentation can be a client library or OpenTelemetry exported as Prometheus text / remote write. OTel exponential histograms become native histograms; classic histograms can ingest as NHCB. Query shape follows the export. The PromQL below assumes the still-common classic _bucket series.

RED is not enough: quantiles for duration

RED still comes first: Rate, Errors, Duration. An LLM gateway is an HTTP (or gRPC) service. Nail those three, then add tokens.

llm_http_requests_total and llm_request_duration_seconds_* are example names. Duration has to be a histogram (or a native histogram). A Summary cannot be aggregated across replicas.

# Rate
sum by (job, model, code) (
  rate(llm_http_requests_total[5m])
)

# Errors: non-2xx share. The code label has to exist at export time
sum by (job, model) (
  rate(llm_http_requests_total{code!~"2.."}[5m])
)
/
sum by (job, model) (
  rate(llm_http_requests_total[5m])
)

Averages lie. One 30s timeout drowned in a hundred 200ms successes still looks “fine” as _sum / _count. histogram_quantile estimates φ from buckets. Classic histograms must keep le in the aggregation:

# do not ship this as "latency"
rate(llm_request_duration_seconds_sum[5m])
/
rate(llm_request_duration_seconds_count[5m])

# p95: rate the buckets, sum by (le, …), then quantile
histogram_quantile(
  0.95,
  sum by (le, job, model) (
    rate(llm_request_duration_seconds_bucket[5m])
  )
)

histogram_quantile is an estimate; error is bounded by bucket width. An SLO at 300ms with buckets at 0.1 / 1 / +Inf parks p95 in the wrong bin. Native histograms interpolate by resolution and skip the “guess the boundaries at instrumentation time” step. φ and the window change in the query, not in the binary — the reason to prefer histograms over summaries when aggregation is required.

In Grafana time series panels, use $__rate_interval rather than a hardcoded 15s under a 1m min-step. The Prometheus query editor documents this quantile shape.

LLM metrics: tokens, TTFT, model cardinality

CPU can stay flat while the router swaps models, the first token stalls, and completions get clipped by max_tokens. None of that lives in node_cpu_seconds_total.

Example names only:

• llm_tokens_total: counter. Label direction="prompt|completion" (or two counters). Unit is tokens, not dollars.
• llm_ttft_seconds: histogram. Time to first token — meaningful for streaming. Non-streaming endpoints should omit it rather than fake TTFT from total duration.
• llm_request_duration_seconds: histogram. Accept to last token (or the full non-streaming response).
• model: a closed enum. gpt-4.1 / claude-sonnet is fine. User-typed model strings, LoRA filenames, and per-request prompt hashes are not.

# example: token rate
sum by (job, model, direction) (
  rate(llm_tokens_total[5m])
)

# example: completion tokens per request (mean, not a quantile)
sum by (job, model) (rate(llm_tokens_total{direction="completion"}[5m]))
/
sum by (job, model) (rate(llm_http_requests_total[5m]))

# example: TTFT p95
histogram_quantile(
  0.95,
  sum by (le, job, model) (
    rate(llm_ttft_seconds_bucket[5m])
  )
)

Metric and label naming is blunt: every unique label set is a new series. Three model values × five code values × two instances is survivable. Ten thousand user_ids is not. le belongs to the histogram; do not also glue quantile="0.95" Summary labels onto the same histogram.

Quality scores (human eval, LLM-as-judge, hallucination samples) are not a scrape-time standard. An offline job can write a gauge onto its own row. Inventing llm_hallucination_ratio with no exporter is worse than leaving the panel off.

Recording rules and an alert sketch

A dashboard that re-evaluates histogram_quantile(sum by (le, …)(rate(…))) every 10s will stall once replica count grows. Recording rules materialize hot expressions. Use level:metric:ops names so they do not collide with raw series.

Sketch /etc/prometheus/rules/llm.yml:

groups:
  - name: llm.recording
    interval: 30s
    rules:
      - record: job_model:llm_http_requests:rate5m
        expr: sum by (job, model, code) (rate(llm_http_requests_total[5m]))
      - record: job_model:llm_request_duration_seconds:p95
        expr: |
          histogram_quantile(
            0.95,
            sum by (le, job, model) (
              rate(llm_request_duration_seconds_bucket[5m])
            )
          )
      - record: job_model:llm_ttft_seconds:p95
        expr: |
          histogram_quantile(
            0.95,
            sum by (le, job, model) (
              rate(llm_ttft_seconds_bucket[5m])
            )
          )
      - record: job_model:llm_tokens:rate5m
        expr: sum by (job, model, direction) (rate(llm_tokens_total[5m]))

  - name: llm.alerts
    rules:
      - alert: LLMHighP95Latency
        expr: job_model:llm_request_duration_seconds:p95 > 8
        for: 10m
        labels:
          severity: page
        annotations:
          summary: "LLM p95 latency high ({{ $labels.job }} {{ $labels.model }})"
          description: "placeholder threshold 8s; take the number from the SLO"
      - alert: LLMHighErrorRatio
        expr: |
          (
            sum by (job, model) (job_model:llm_http_requests:rate5m{code!~"2.."})
            /
            sum by (job, model) (job_model:llm_http_requests:rate5m)
          ) > 0.05
        for: 5m
        labels:
          severity: page
      - alert: LLMTTFTRegressed
        expr: job_model:llm_ttft_seconds:p95 > 2
        for: 15m
        labels:
          severity: ticket
        annotations:
          summary: "TTFT p95 high; a CPU-only dashboard still looks idle"

> 8, > 0.05, and > 2 are placeholders, not measured SLOs. Read a week of quantiles, then set for: so a single spike does not page. Alerting rules decide firing; Alertmanager owns routing, silence, and grouping.

promtool check rules validates syntax before load. A rule group that overruns the next evaluation_interval is skipped, which punches holes in recorded series — another reason heavy quantiles belong in rules, not in every Grafana panel.

Grafana dashboard layers

A dashboard is read top to bottom. Forty node_exporter panels with tokens in a corner is the wrong order.

1. Variables: job, model, instance. Multi-select model. Populate from label_values(llm_http_requests_total, model), not a handwritten list.
2. Row 1 — RED: QPS (stat + timeseries), error ratio, p50 / p95 / p99. Quantiles from recording rules, or from histogram_quantile with $__rate_interval.
3. Row 2 — LLM: tokens / s split by direction, completion tokens per request, TTFT p95. On streaming services, TTFT next to total duration separates queueing from generation.
4. Row 3 — distribution: duration heatmap (query format Heatmap, _bucket series). Bimodal traffic (fast successes + long timeouts) is two bands, not a bland average.
5. Row 4 — saturation: CPU, memory, queue depth, GPU util if exported. This row explains. It does not close the incident. Low CPU is not a verdict.

Fewer panels, same expression. Recording rules compute once; panels reference the recorded name. Alert thresholds and graph thresholds should be the same PromQL so a green board cannot coexist with a firing pager.

What a silent quality drop looks like on CPU

A familiar false green:

• rate(node_cpu_seconds_total{mode!="idle"}[5m]) is flat. The box is “not busy”.
• rate(llm_http_requests_total[5m]) holds. HTTP code="200" stays near 100%.
• Meanwhile: TTFT p95 climbs (upstream queue or cold start); completion tokens per request fall (max_tokens cut, or the model stops early); prompt tokens climb (context keeps growing); a new model value appears (router moved to a cheaper or slower model).

Any one of those can happen at constant CPU and zero HTTP errors. Callers see a slower first token, a shorter answer, or a different model. The process looks idle. Hallucinations, refusals, and schema breakage do not appear in Prometheus unless an eval job exports them. Drawing a fake llm_hallucination_ratio with no scrape is worse than omitting the panel.

Put RED and TTFT / tokens above CPU. Scale from saturation. Catch quality regressions from quantiles and tokens, not from idle percent.