Strengths and weaknesses, not scores.
A score tells you who is high and who is low. A diagnosis tells you why — and what to do about it. We separate measurement from pattern, pattern from hypothesis, hypothesis from intervention, and intervention from validated result. Five layers. Never collapsed.
See Workflow Fit Back to EvaluateFour levels. Examine in order.
Diagnosis is not a single act. It is a hierarchy. Before blaming the organization, check the operator. Before blaming the workflow, check the tool. Work upward only after the current level is examined and ruled out — or noted as a contributor.
Operator
Is this operator's behavior — leverage, yield, context reuse, stability — the source of the observation? Start here.
measuredTool / Model
Is the tool or model the operator is using the source? Different providers and models produce different operating envelopes.
measuredWorkflow
Is the stage of work the source? Some stages reward high leverage; others reward high persistence. Fit matters.
hypothesisOrganization
Is the system — incentives, access, training, tooling policy — the source? The highest level. Reach it last.
hypothesisDo not advance to a higher level until the current level has been examined and ruled out (or noted as a contributor).
Never collapse these into one claim.
A measurement is not a pattern. A pattern is not a hypothesis. A hypothesis is not an intervention. An intervention is not a validated result. Each layer has its own evidence standard. Conflating them is the most common error in AI workforce analysis.
| Layer | What it is | Evidence standard | Status |
|---|---|---|---|
| MEASUREMENT | A computed metric from telemetry — leverage, yield, context reuse, stability. | Reproducible from primitives. Traceable to token counts. | measured |
| OBSERVED PATTERN | A recurring structure in the measurements — e.g., high input / low output across sessions. | Consistent across time windows and session types. | observed |
| DIAGNOSTIC HYPOTHESIS | A proposed explanation for the pattern — e.g., "operator re-prompts from scratch instead of reusing context." | Consistent with the pattern. Falsifiable. Has alternatives. | hypothesis |
| RECOMMENDED INTERVENTION | A proposed action targeting the hypothesis — e.g., "train context construction technique." | Declares a target metric and a follow-up window. | pilot |
| VALIDATED RESULT | The measured change after the intervention — target and non-target deltas. | Re-measured against baseline. Outcome joins labeled ASSOCIATION. | validation required |
Never collapse these into one claim. "The operator is bad at context reuse" is a measurement, not a diagnosis. "Train them" is an intervention, not a validated result.
The pattern library.
Patterns are recurring structures in operator measurements. Each pattern is a HYPOTHESIS — a named, observable configuration of metrics that suggests a diagnostic direction. Patterns are not diagnoses. They are starting points for diagnosis.
Low leverage
High input, low output. The operator feeds the model a lot and gets little back. Suggests over-specification or poor prompt efficiency.
hypothesisPoor cache / context reuse
Low cache reads, high cache writes that are never read again. Context is built but not reused. Suggests context construction without persistence.
hypothesisHeavy input / low output
Input dominates the token budget. The operator authors far more than the model returns. Suggests the model is being used as a reader, not a generator.
hypothesisHigh output / low persistence
The model generates a lot, but little is retained or carried forward. High output, low yield. Suggests output is discarded or not integrated.
hypothesisContext-rich / execution-weak
Strong context construction but low yield or low leverage. The operator sets up well but does not convert context into output.
hypothesisHigh usage / low Yield
Many sessions, many tokens, but yield is low. Volume without quality. The most common divergence pattern.
hypothesisLow usage / high Yield
Few sessions, but each one is productive. The operator is efficient but underutilized. A hidden strength.
hypothesisHigh volatility
Metrics swing widely session to session. Stability is low. Suggests inconsistent technique or high sensitivity to task type.
hypothesisModel sensitivity
Performance shifts sharply across providers or models. The operator's results depend heavily on tool selection.
hypothesisStage-specific performance
Strong at some workflow stages, weak at others. Not a generalist. Fit is stage-dependent.
hypothesisImproving trajectory
30-day movement is positive. Metrics trending up. The operator is getting better — at what, exactly, is the next question.
hypothesisDeclining trajectory
30-day movement is negative. Metrics trending down. May indicate tool change, task shift, or disengagement.
hypothesisOne operator is a data point. A cohort is a distribution.
Individual profiles are necessary but not sufficient. Cohort views reveal distributions, divergences, and segments that individual profiles cannot. These are the standard cohort views.
Field position
Each operator's percentile rank across the cohort on each core metric. Who is high, who is low, and by how much.
Distribution
The shape of each metric across the cohort. Heavy-tailed, normal, bimodal. The distribution determines how to read the percentiles.
30-day movement
Trajectory across the cohort. Who is improving, who is declining, who is stable. Movement is a first-class metric.
Usage vs operation divergence
Operators ranked by usage versus operators ranked by operation quality. The gap between the two rankings is the most actionable view.
Pattern clusters
Operators grouped by shared operating patterns. Clusters reveal segments — e.g., "recursive builders" vs "one-shot generators."
Provider / model breakdown
Cohort metrics decomposed by AI provider and model. Reveals whether differences are operator-driven or tool-driven.
Stability ranking
Operators ranked by consistency across sessions. Low stability is a diagnostic signal even when average performance is high.
Volatility map
Session-to-session variance across the cohort. High-variance operators need different interventions than low-variance operators.
From measurement to validation. One operator.
A complete diagnostic flow for OPERATOR 041 — from the initial measurement through the validated result. Each step is a separate claim with its own evidence standard.
MEASUREMENT
OPERATOR 041: Leverage 8.1x (22nd percentile), Yield 3.2 (18th percentile), Context Reuse LOW, Stability 64, 30-Day Movement −6%.
measuredOBSERVED PATTERN
High input / low output across 28 of 30 sessions. Cache writes present but cache reads near zero. Context is built but never reused. Pattern is consistent across providers and time windows.
observedDIAGNOSTIC HYPOTHESIS
The operator constructs fresh context each session instead of reusing prior context. Low cache reads + high cache writes + low leverage = context construction without persistence. Alternative: the operator's tasks are genuinely novel each session (ruled out — task types repeat).
hypothesisRECOMMENDED INTERVENTION
Train context reuse technique: session chaining, context carry-forward, cache-aware prompting. Target metric: cache reads per session. Follow-up window: 30 days. Non-target metrics: leverage, yield, stability.
pilotVALIDATION
Re-measure at 30 days. Cache reads per session: +210%. Leverage: 8.1x → 19.4x. Yield: 3.2 → 6.1. Stability: 64 → 78. Non-target metric (session count): unchanged. Outcome join (PR cycle time): −12% — labeled ASSOCIATION, not CAUSATION.
validation requiredThis is a synthetic example. Real validations require a defined comparison condition. The outcome join (PR cycle time) is ASSOCIATION — it co-occurred with the intervention but was not causally isolated.
All diagnoses are HYPOTHESIS — not causal findings. A hypothesis becomes a validated result only after re-measurement against a defined comparison. Outcome joins are ASSOCIATION, never CAUSATION, unless a validated design permits causal claims.
Diagnoses become interventions. Interventions get re-measured.
Once you know the pattern and the hypothesis, the next question is: which stage of work does this operator fit best — and which intervention moves the needle?
See Workflow Fit Verification