SuperPenguin vs LLM Observability vs Gateways

LLM observability vs gateways vs SuperPenguin

These tools can all show a dollar next to a model call. They are not the same job.

LLM observability (Langfuse, LangSmith) is for debugging and evaluating LLM applications. You wrap the app, inspect traces, score outputs, and run evaluations. You can tag a trace with a user, session, or metadata. Cost is an estimate on those traces. You can see which step in a run was expensive, and you can roll that estimate up by user or tag. That is enough when you only need a debug-time cost number. It is not a provider invoice, and it is not coding-tool spend unless the agent exports traces.

Gateways (Helicone, OpenRouter) sit between your application and the model. Your app calls the gateway; the gateway forwards the request to OpenAI, Anthropic, or another provider. That hop is the product: log every request, route across providers, and cap or rate-limit traffic. A gateway only sees calls that passed through it. Cursor, Claude Code, and Codex talk to the vendor from the laptop, so they never hit the hop.

SuperPenguin is a holistic way to manage AI cost. On product APIs, you attach metadata to each request so you can slice cost by any dimension you want. On the coding side, the Mac app shows each engineer their usage and rate limits, the web app aggregates that for the team, and coding ROI is cost per merged pull request.

Keep observability to debug. Keep a gateway to route or rate-limit. Stay on Langfuse or LangSmith alone if a tagged cost estimate on the trace is the whole job. Use SuperPenguin when the job is billed spend, including coding tools a tracer never sees.

Quick comparison

Published documentation as of August 28, 2026. All five products change quickly, so check the linked page before you plan around a row.

SuperPenguin: best for AI cost from APIs through coding tools

What it is: SuperPenguin is a holistic way to manage AI cost. It captures spend on the model APIs your product already calls. It also captures Cursor, Claude Code, and Codex spend: per-engineer usage and rate limits in the Mac app, the team rollup on the web, and coding ROI as cost per merged pull request.

Known for: An SDK wrap on the client you already use. You attach metadata to each request, then slice cost by any dimension you want. That works with 12 platforms: OpenAI, Anthropic, Gemini, AWS Bedrock, Azure OpenAI, Deepgram, ElevenLabs, OpenRouter, Vercel AI Gateway, LiteLLM, Fireworks, and Together AI, plus other OpenAI-compatible endpoints.

On the coding side, each engineer runs a Mac app next to Cursor, Claude Code, or Codex and sees their own usage and remaining rate-limit allowance. The web app is the team rollup of that same data. You can also connect GitHub to get coding ROI as cost per merged pull request.

Commonly used for: On the API side, teams want to know what it costs to serve a customer or run a feature, so they can set prices and watch margin. They also use it after a prompt or model change, to see whether the product got more expensive. On the coding side, engineers watch their own usage and rate limits in the Mac app. Leads use the web app to see the same picture for the team, then use cost per merged pull request to judge coding ROI.

Worth noting: Prompt and response capture is optional and off by default. Turn it on for optimization.

LLM observability

Langfuse: open-source tracing and evaluation

You use Langfuse to see what an LLM app actually did. Wrap the application, inspect each step in a trace, score the output, run evaluations, and iterate on prompts. That is the product: debugging and evaluation.

It is open source under an MIT license. Teams self-host it, or use Langfuse Cloud, including a HIPAA region. ClickHouse acquired the company in January 2026. The project stayed MIT.

Cost is a secondary number on those traces. You can attach a user, session, tags, and metadata, then see estimated cost by user in the UI. That estimate is calculated when the trace is ingested, so a later price change does not rewrite old traces. Only generation and embedding observations carry cost. Evaluations, including LLM-as-a-judge, add to the Langfuse bill as well as the model bill.

LangSmith: tracing and evaluation for LangChain teams

You use LangSmith to debug and evaluate chains and agents. The question is why a run failed, which step went wrong, or whether the output passed an eval. If you already build on LangChain or LangGraph, tracing is largely automatic. Other stacks can still send runs into it.

Hosted cloud is the default. Self-host is an Enterprise option.

Cost sits on those runs after the debug job. LangSmith can attach a dollar figure to a step so you can see which one dominated spend. Paid tool calls such as search, scraping, or retrieval can sit in that unit cost with the model, not only generations. The figure still lives on the trace, not on an invoice, and dashboard grouping is limited.

Gateways

Helicone: request logs and spend caps

Helicone is a gateway you put in front of your model calls. Point the app at Helicone and it logs every request, shows per-user metrics, and can enforce spend caps. You can also send logs without putting a hop in front of inference, but rate limits only work when traffic actually goes through Helicone.

Mintlify acquired Helicone in March 2026, put it in maintenance mode, and closed new signups. Existing users can keep using it. Cost is most accurate on Helicone's own gateway path. Other ingest paths estimate from a price table that stopped getting model updates after the acquisition.

OpenRouter: one API in front of many providers

OpenRouter is an inference gateway: one API and one account in front of models from many providers. You call OpenRouter instead of calling OpenAI or Anthropic directly. Stripe announced an agreement to acquire OpenRouter in August 2026. That had not closed at the time of writing.

The cost number is what OpenRouter charged your account, which is useful when OpenRouter is the bill. Bring-your-own-key usage is estimated at list rates, and spend guards on that path can look healthier than they are. Cached responses report zero usage, so they contribute nothing to a token-share chargeback.

Pricing

List prices as of August 28, 2026. Inference spend is separate from what Langfuse, LangSmith, and Helicone charge to log that traffic. SuperPenguin meters the AI spend it manages, not traces.

Helicone's published prices still appear on its pricing page. Weigh them against maintenance mode and closed signups before treating Helicone as a 2026 buying option.

Using them together

You do not have to pick one category. Observability is still how you debug a run. A gateway is still how you route or cap traffic. SuperPenguin sits beside them for spend.

The piece the others do not offer is coding-tool cost on the laptop: each engineer's usage and rate limits in the Mac app, the same picture rolled up on the web, and coding ROI as cost per merged pull request. SuperPenguin also attributes product-API spend: you attach metadata to each request and slice cost by any dimension you want. Keep the tracer or gateway you already have. Add SuperPenguin when you need that picture.

Which LLM cost tracking setup should you use?

SuperPenguin captures AI spend from the APIs you wrap and the coding tools a gateway never sees. It is not a tracer and not a proxy. It works alongside Langfuse, LangSmith, Helicone, and OpenRouter rather than replacing them, because debugging an agent, routing traffic, and managing AI spend are different jobs.

Sources

1. Token and cost tracking, Langfuse docs.
2. Metrics API, Langfuse docs.
3. Billable units, Langfuse docs.
4. Pricing, Langfuse.
5. Cost tracking, LangSmith docs.
6. Threads, LangSmith docs.
7. LangSmith plans and pricing, LangChain.
8. Cost tracking cookbook, Helicone docs.
9. Proxy vs async, Helicone docs.
10. Helicone pricing, Helicone.
11. Mintlify acquires Helicone, Mintlify.
12. Usage accounting, OpenRouter docs.
13. Broadcast, OpenRouter docs.
14. Guardrails and BYOK, OpenRouter docs.
15. Pricing, OpenRouter.
16. SuperPenguin pricing, SuperPenguin.
17. Set up AI Coding and PR Costs, SuperPenguin docs.
18. Pricing and costs, SuperPenguin docs (OpenRouter billed cost and upstream).