Measure the operating system, not the conversation.
We do not read prompts. We do not read responses. We measure the structure of AI operation — input, output, cache, context, leverage, yield, stability, and trajectory. The operating system is observable without reading the conversation.
Design a 30-Day Evaluation See How We DiagnoseFour primitives. Everything else is derived.
Every AI interaction reduces to four measurable primitives. All higher-order metrics — leverage, yield, token SNR, context reuse — are constructed from these. Nothing else is required to measure operation.
INPUT
Tokens submitted to the model in a given turn. Includes prompt, context, attachments, and reused prior output.
measuredOUTPUT
Tokens returned by the model in a given turn. The generated response, before any downstream use.
measuredCACHE READ
Tokens retrieved from context cache — prior context reused without re-submission. Indicates context persistence.
measuredCACHE WRITE
Tokens written to context cache for future reuse. Indicates deliberate context construction.
measuredThese four primitives are content-free. No prompt text. No response content. No code. The structure of operation is fully observable from token counts and cache behavior alone.
Five metrics. Each derived from the primitives.
From four primitives we construct five core metrics. Each has a construct definition, a mathematical definition, and a valid domain. Each is labeled MEASURED because it is computed directly from telemetry — not inferred.
| Metric | Definition | Status |
|---|---|---|
| leverage | Output tokens relative to net new input tokens. How much value the operator extracts per unit of input they author. High leverage = the operator makes the model do more with less new prompting. | measured |
| yield | Usable output tokens relative to total output tokens. How much of what the model generates is actually retained, committed, or carried forward. High yield = low waste. | measured |
| token_snr | Signal-to-noise ratio across the token stream. Distinguishes productive tokens from redundant or discarded ones. High SNR = the operator's token budget is spent on signal. | measured |
| log_leverage | Log-scaled leverage to normalize heavy-tailed distributions. Compresses extreme values so a few high-leverage sessions do not dominate the operator's profile. | measured |
| construction | Ratio of cache writes to total input. Measures how deliberately the operator builds reusable context versus re-prompting from scratch each turn. | measured |
Every metric is traceable to the primitives. Every score is reproducible from the telemetry. No metric is inferred from behavior, personality, or self-report.
Every operator becomes a profile. Every label is traceable.
The operator profile is the core deliverable of evaluation. It aggregates the five core metrics plus derived fields — context reuse, stability, trajectory, volatility, and operating pattern — into a single traceable record.
Every diagnostic label must be traceable to underlying measures. "RECURSIVE / BUILD-HEAVY" is not a personality assessment — it is a pattern derived from high construction, high context reuse, and stable yield across sessions. Show the measures, then the label.
What we do NOT measure.
Measurement discipline means knowing what to exclude. We collect the minimum evidence required to answer the measurement question. The following are explicitly NOT required for core evaluation.
Not required
- Prompt content — the text of what operators type
- Response content — the text the model returns
- Code content — the source code in inputs or outputs
- Email content — the body text of messages
- Slack content — the text of channel messages
Core
- Token telemetry — input, output, cache read, cache write counts
- Cache behavior — read/write ratios, reuse patterns
- Session structure — turn count, session length, provider, model
- Temporal patterns — cadence, gaps, burstiness
We do not collect everything because it exists. We collect the minimum evidence required to answer the measurement question. Token telemetry is core. Content is not.
Twelve requirements before a metric ships.
A number is not a metric. A metric is not a measure. A measure is not a construct. Before any metric enters the operator profile, it must satisfy twelve requirements. This is what separates measurement from dashboarding.
Construct definition
What does the metric claim to represent? Named, defined, and bounded in plain language.
Mathematical definition
The exact formula. Reproducible from the primitives by anyone with the telemetry.
Valid domain
The range of inputs over which the metric is defined and meaningful. Where it breaks.
Unit
The unit of measurement. Tokens, ratios, percentiles, log-scaled values.
Directionality
Which direction is better? Explicitly stated. No metric is "good" without a defined polarity.
Distribution
The expected distribution across operators. Heavy-tailed, normal, bimodal — known and accounted for.
Normalization
How the metric is normalized across providers, models, and session types. Comparable across contexts.
Reliability
Does the metric produce the same score for the same telemetry? Test-retest consistency.
Validity
Does the metric measure what it claims to measure? Construct validity, not face validity.
Sensitivity
Does the metric detect meaningful change? Not so coarse that real improvement is invisible.
Decomposability
Can the metric be broken down by provider, model, session type, and time window? Diagnosable, not opaque.
Traceability
Every score traces back to the primitives and the telemetry that produced it. No black boxes.
Canonical labels
Every claim in the system carries a canonical status label. The label tells you how much trust the claim has earned — and what would be required to promote it.
No claim is promoted without evidence. A HYPOTHESIS becomes VALIDATION REQUIRED before it becomes PROVEN. The label is the contract.
Profiles become diagnoses. Diagnoses become interventions.
Evaluation produces the operator profile. Diagnosis turns the profile into actionable strengths, weaknesses, and hypotheses.
See How We Diagnose Workflow Fit