Why LLM applications need their own observability
Ordinary application monitoring sees an HTTP request to api.openai.com. It doesn't see what matters about a model call:
- Cost is per call, not per server. Every request is billed by tokens, so one busy feature or one large customer can move the bill. Provider invoices show a total per key, not per service, feature or customer.
- Failures are often quiet. Rate limits, timeouts and retries that succeed on the third try don't crash anything, but they add latency and cost.
- Latency is long and uneven. Calls take seconds, streaming makes time to first token matter, and the slow tail (p95) is what users feel.
- Models change under you. Prices change, models are retired, and a service still calling an old model stops working on a known date.
What to measure
Record these for every call, and break each down by service, model, environment and, if you serve many customers, by customer.
| Metric | Why it matters | Alert when |
|---|---|---|
| Cost | What each service, model and customer spends | Spend passes a limit, or the month is on track to exceed the budget |
| Tokens | Input, cached input, output and reasoning tokens drive cost | Tokens per call jump after a prompt change |
| Latency and time to first token | What users wait for; p95 shows the slow tail | p95 goes above your target |
| Errors | Rate limits (429), timeouts and server errors by type | Error rate goes above a threshold, such as 2% |
| Retries | Hidden cost and latency behind calls that eventually work | Retries rise for one model or provider |
| Calls and heartbeats | Volume per service, and whether a service is still sending | A service stops reporting |
How LLM observability works
There are three common ways to collect this data. They can be combined.
| Approach | How it works | Trade-offs |
|---|---|---|
| SDK in your service | A library wraps the provider's client and records each call, sending records in the background. | No extra network hop; sees your service, customer and trace context. Needs a line of setup per service. |
| Proxy or gateway | You point the client's base URL at a proxy that forwards to the provider and logs the traffic. | One config change; can also cache and route. Adds a hop in the request path and knows less about your code. |
| OpenTelemetry | Your code emits spans that follow the OpenTelemetry GenAI conventions, exported to any compatible backend. | Vendor-neutral and fits existing tracing. Needs instrumentation and a backend that understands GenAI spans. |
Tools in this space lean different ways. Gateways such as Helicone and Portkey sit in the request path. Langfuse, Arize Phoenix and Opik are open-source tools that combine tracing with evaluation. LangSmith is built around LangChain. Datadog offers LLM monitoring as part of its APM.
Alye uses an SDK and also accepts OpenTelemetry. It focuses on production cost and reliability: correct prices per model, region and marketplace, spend per service and customer, traces, and alerts by email, Slack and Microsoft Teams.
Set it up in five minutes
With Alye, each service needs an SDK key and two lines of code. Calls through the official OpenAI, Anthropic and Google GenAI clients are then recorded automatically.
# Python 3.10+ pip install alye export ALYE_API_KEY=alye_live_... import alye alye.init(service="support-reply")
// Node.js 20+
npm install alye
import * as alye from "alye";
alye.init({ service: "support-reply" });
const openai = alye.wrap(new OpenAI());Express and NestJS have one-line integrations, and existing OpenTelemetry exporters can point at Alye instead. See the docs.
How to choose an LLM observability tool
- Does it price calls correctly for your models, regions and cloud marketplaces (Azure OpenAI, Vertex AI, Bedrock), and your own prices for fine-tunes?
- Can it split spend by service and by your own customers, and set budgets?
- Does it sit in your request path? If so, what happens to your app when it is slow or down?
- Does it alert where your team already works, and on silent services, not just errors?
- Are prompts stored by default? Is personal data masked before it leaves your service?
- Is pricing predictable: what happens when you go over, and do you pay per seat?
- Does it accept OpenTelemetry, so you aren't locked in?