Measuring progress

Metrics should reflect actual user outcomes and system behavior. Vanity metrics may be useful as context, but they cannot establish product quality on their own.

​

20.1 Product metrics

Possible measures include:

• Completion rate for defined user workflows.
• Time required to complete a task compared with a baseline.
• Frequency of users correcting or abandoning results.
• Retention for users who have completed a meaningful task.
• Accessibility issues found and resolved.
• Support volume by feature and root cause.

​

20.2 Model metrics

Depending on the task, measures may include:

• Accuracy or task success on held-out evaluations.
• Hallucination or unsupported-claim rate.
• Tool-selection and tool-execution success.
• Robustness to malformed inputs and prompt injection.
• Latency and resource use.
• Regression rate between versions.

​

20.3 Reliability metrics

Track availability, error rate, latency, time to detect incidents, time to mitigate incidents, backup restoration success, and the number of unresolved high-severity vulnerabilities.

​

20.4 Safety metrics

Track permission violations, sensitive actions completed without required confirmation, cancellation failures, policy regressions, and time to remediate high-impact issues. A low observed incident count is not sufficient if testing coverage is weak.

​

20.5 Reporting discipline

Metrics need definitions, time windows, sample sizes where relevant, and context. A number without a denominator can be misleading. Internal dashboards should distinguish measured data from estimates and forecasts.