arosplatforms™AI consultancy
ar
← All articles

LLM Model Monitoring: What to Track in Production

LLM Model Monitoring: What to Track in Production

Hand-drawn monitoring motifs frame the title area

LLM model monitoring means continuously tracking whether an AI system in production is healthy, accurate and affordable, and being able to explain why when it is not. For an operations leader, the practical answer is a short list of signals tied to business outcomes, owned by your team and built into the system from day one. That is how we build AI operating systems at Arosplatforms: monitoring is part of the delivery, and clients keep ownership of it with no vendor lock-in.

Arosplatforms
arosplatforms.com
Build Monitoring Into Your AI System
Arosplatforms builds customized AI operating systems with monitoring integrated, while keeping ownership with your team and avoiding vendor lock-in.
Visit

TL;DR

  • Monitoring watches known measures (latency, errors, token use, cost) and alerts when they move. Observability records enough detail to explain a problem nobody predicted. You need both.
  • LLM failures often don’t show up as errors. They show up as wrong or low-quality answers, so quality and safety signals matter as much as uptime.
  • Track three groups: system health, business outcomes (task success, resolution time) and model risk (hallucinations, policy violations, drift).
  • Run safety checks on all traffic, sample quality checks, and score a fixed test set on a schedule to catch drift.
  • Decide who owns the dashboards and alerts before launch, not after the first incident.

Table of Contents

What LLM model monitoring is, and how it differs from observability

Monitoring is the practice of continuously tracking predefined metrics such as latency, error rates, token usage, cost and throughput to tell whether an application is healthy right now. Observability goes further: it is the ability to understand why a response was produced by examining the full request flow, including retrieval, model inference, post-processing and tool calls.

A simple way to hold the difference: monitoring tells you that your intake assistant got slower or more expensive. Observability lets you open one bad conversation and see which document was retrieved, which prompt ran and which tool call failed.

The distinction matters because, as Wikipedia’s overview of AI observability notes, a system can look healthy on standard infrastructure measures while still producing answers that are wrong, irrelevant or unsafe. A lease-inquiry bot can respond in 800 milliseconds with zero errors and still quote the wrong pet policy.

You will also see “AI model monitoring” used for the same job. That term is broader: AI observability covers LLMs, classic ML models, agents, RAG pipelines and fine-tuned models. If your business runs a claims-scoring model next to a document-summarizing LLM, one monitoring practice should cover both.

What LLM model monitoring is, and how it differs from observability

Why production LLMs need their own monitoring

Traditional software fails loudly. LLM applications often don’t. According to Datadog, outputs are non-deterministic, model chains are opaque, and failures surface as wrong or low-quality responses rather than errors. The same guidance lists hallucinations, cost and performance problems from third-party APIs, prompt injection and data privacy risk among the issues observability has to address.

AWS Prescriptive Guidance names the cost of skipping it: blind spots in agent behavior, undetected cost anomalies or regressions, limited insight into quality and difficulty finding root causes. Its framing is useful for non-engineers: you monitor behavior, cost and correctness rather than servers.

There is also an operational reason. The OpenTelemetry project notes that usage, cost, latency and rate limits are the day-to-day pressure points for applications that call external models. Hit a rate limit during month-end close or a busy leasing weekend and a core workflow stops.

Analyst expectations point the same way. Anaconda cites a Gartner prediction that 40% of organizations deploying AI will use dedicated AI observability tools by 2028.

What to track: three groups of signals

A workable scheme, drawn from Anaconda’s LLMOps guide, organizes monitoring into system health, business outcomes and model risk. Here is what each looks like for the industries we serve.

Group What it measures Example signals Where it shows up in your business
System health Is it up, fast and affordable? Latency, error rate, throughput, token use, cost per request Any workflow; cost spikes hit budgets first
Business outcomes Is it doing the job? Task success, resolution time, CSAT, support deflection Tenant messages answered, claims triaged, intake forms completed
Model risk Is it safe and accurate? Hallucination indicators, policy violations, prompt injection attempts, output drift Patient intake, contract review, underwriting notes

System health

These are the classic signals. Galileo describes them as performance (invocation latency, time to first token), cost and resource use (tokens per request, GPU and CPU use) and reliability (error rates, API failures, throughput). One published example of alert thresholds from Systems Hardening: latency alerts when P99 exceeds 5 seconds or time to first token exceeds 2 seconds, and cost alerts when hourly spend exceeds budget or spikes 2x above the 24-hour average. P99 means the response time that 99 of 100 requests beat. Treat those numbers as a starting template and tune them to your own workflow.

System health metrics and example alert thresholds

Cost deserves a better unit than the token. Future AGI argues for measuring dollars per million successful tasks: a cheaper model that fails 30% of the time is rarely the right pick, while a slightly pricier one that fails 5% of the time often is. For a deeper look, see our piece on five metrics to check before cutting LLM spend.

Business outcomes

This is the layer operations leaders care about, and the one generic dashboards skip. Pick two or three numbers that already exist in your operation: minutes to resolve a tenant request, claims triaged per adjuster per day, percentage of intake forms completed without staff correction. Our approach is built around measurable results, with weekly demos on real data so these numbers are visible while the system is being built.

Model risk

This is where hallucinations, policy violations and prompt injection sit. Sentrial’s analysis says quality signals are the layer most teams under-instrument, and that quality evaluations should run against every interaction rather than occasional spot checks.

Evaluation metrics and drift

Evaluation metrics are automated checks that answer whether an output was good. Common ones include factual accuracy, retrieval relevance, goal completion and instruction following. Academic work cautions that static benchmarks face validity challenges and argues for metrics that reflect real-world performance and are refined over time. In practice, that means building your test set from your own documents and conversations, not from a public leaderboard.

Drift is the slow decline in quality over time. Weights & Biases separates four types:

  • Data drift: the inputs change from what the system was built for.
  • Model drift: the model’s quality shifts, including silent provider updates.
  • Retrieval drift: the knowledge base returns irrelevant or outdated context.
  • Behavioral drift: external changes such as API format shifts or traffic spikes.

Retrieval drift is the one that surprises property managers, law firms and municipalities most. Policies, lease templates and ordinances change, the index gets stale, and the assistant keeps answering confidently from old text. The simplest defense, per Future AGI, is scoring the same fixed evaluation set every week and watching week-over-week change.

A production monitoring workflow

Here is a practical sequence. The numbers come from Future AGI’s production evaluation workflow; treat them as starting points.

  1. Test before you deploy. Curate 100 to 1,000 prompts that look like real production traffic, attach expected behavior, run them on every code change and block releases that regress past a set tolerance.
  2. Sample live traffic. Score 5% to 25% of real requests with the same evaluators and alert on day-over-day drops.
  3. Guard everything that is safety-critical. Run input and output screens on 100% of traffic. Sampling is fine for quality, not for safety.
  4. Watch for drift. Track a rolling seven-day mean per evaluator and alert if today’s mean drops more than 1.5 standard deviations.
  5. Keep traces. Capture the full path of each request so you can explain failures.

When a user reports a bad answer, LangChain’s recommended loop is to find the trace, capture the messages, retrieved context and tool calls, add it to a dataset (anonymizing sensitive content), write the expected behavior and run offline evals before shipping a fix. Every complaint becomes a permanent test. If you want to see how requests behave across several AI models at once, a multi-model audit tool can help you compare outputs while you investigate.

For teams that want a staged rollout, Dash0 outlines eight phases: traces, configuration provenance, performance and economics, outcomes, evaluation, production feedback loops, governance, and operational practices such as quality SLOs and incident response. You do not need all eight on day one. Start with traces, outcomes and a weekly evaluation run.

Monitoring has its own cost

One academic survey describes a “monitorability tax”: monitoring consumes real compute. That is another reason to sample quality checks and reserve full coverage for safety and compliance signals.

Regulated work needs monitoring you can show

For clinics, insurers, financial teams and municipalities, monitoring doubles as evidence. A log of what the model saw, what it answered and which safety screens ran is what a compliance reviewer will ask for. Our AI compliance monitoring use case and AI pipeline monitoring pages describe how we approach it, and healthcare teams can look at AI governance and compliance for healthcare. Keep patient and client data out of logs unless it has been anonymized, and have your privacy officer or counsel confirm what your logs may contain.

Perspective

Most monitoring advice is written for engineering teams that already have an ML platform. Our clients are operations leaders running a leasing desk, a clinic, a dispatch office or a claims team. They need to know one thing: is this system still doing the job we bought it for?

We think monitoring belongs inside the build, not bolted on afterward. That is why our engagements include weekly demos on real data, fixed scope and pricing, and full ownership of what we deliver. If your team owns the dashboards, the evaluation sets and the alerts, you can run the system without us and without vendor lock-in.

Our advice: choose a few business outcome metrics first, add safety checks on all traffic, and score a fixed evaluation set every week. Everything else can grow from there.

arosplatforms team

Build monitoring into your AI system from the start

If you are planning an AI system or already running one without visibility, a readiness assessment is the natural first step: it clarifies which workflows to automate, what to measure and how monitoring will be owned. Our AI readiness assessment covers that ground, and our custom AI development engagements deliver production systems with monitoring and ownership built in. For the underlying infrastructure, see AI infrastructure and MLOps.

FAQ

What is the difference between LLM monitoring and LLM observability?

Monitoring tracks predefined metrics such as latency, errors, tokens and cost, and alerts when they move. Observability records traces, inputs, outputs and retrieved context so you can explain problems you did not predict. Production systems need both.

Which LLM evaluation metrics should I start with?

Start with task success on your own workflow, factual accuracy against your documents, retrieval relevance and latency. Add safety and policy checks on every request. Score a fixed test set weekly so you can see drift.

How much traffic should I evaluate?

One published workflow samples 5% to 25% of live traffic for quality scoring and runs safety guardrails on 100%. Adjust the sample rate to your volume and risk, since monitoring itself consumes compute.

Does AI model monitoring cover more than LLMs?

Yes. AI model monitoring and observability also cover classic ML models, agents, RAG pipelines and embedded AI features. A single practice with shared dashboards and alerts is easier to run than separate ones.

Sources