> ## Documentation Index
> Fetch the complete documentation index at: https://documentation.orbitdev.org/llms.txt
> Use this file to discover all available pages before exploring further.

# Measuring progress

> Metrics should reflect actual user outcomes and system behavior. Vanity metrics may be useful as context, but they cannot establish product quality on their own.

Metrics should reflect actual user outcomes and system behavior. Vanity metrics may be useful as context, but they cannot establish product quality on their own.

## 20.1 Product metrics

Possible measures include:

* Completion rate for defined user workflows.
* Time required to complete a task compared with a baseline.
* Frequency of users correcting or abandoning results.
* Retention for users who have completed a meaningful task.
* Accessibility issues found and resolved.
* Support volume by feature and root cause.

## 20.2 Model metrics

Depending on the task, measures may include:

* Accuracy or task success on held-out evaluations.
* Hallucination or unsupported-claim rate.
* Tool-selection and tool-execution success.
* Robustness to malformed inputs and prompt injection.
* Latency and resource use.
* Regression rate between versions.

## 20.3 Reliability metrics

Track availability, error rate, latency, time to detect incidents, time to mitigate incidents, backup restoration success, and the number of unresolved high-severity vulnerabilities.

## 20.4 Safety metrics

Track permission violations, sensitive actions completed without required confirmation, cancellation failures, policy regressions, and time to remediate high-impact issues. A low observed incident count is not sufficient if testing coverage is weak.

## 20.5 Reporting discipline

Metrics need definitions, time windows, sample sizes where relevant, and context. A number without a denominator can be misleading. Internal dashboards should distinguish measured data from estimates and forecasts.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.