How we will measure usefulness
Status: proposed experiment. Results are not available yet.
The primary comparison uses the same main agent with and without Ennodia. A separate control can give the main agent the same additional models without Ennodia. That control helps distinguish orchestration benefits from additional computation.
Pilot design
Section titled “Pilot design”- Select 20 tasks before running the experiment.
- Repeat each condition three times.
- Fix models, versions, source material, available skills, and resource limits.
- Define correctness checks before collecting answers.
- Keep evaluator answers outside agent materials.
- Retain failed, canceled, and incomplete attempts.
| Measure | Definition |
|---|---|
| Correctness | Fraction of tasks meeting the predefined checks. |
| Time | Elapsed time through planning, workers, comparison, verification, and integration. |
| Resources | Reported input, output, and cached tokens across all participants. Unknown usage stays unknown. |
| Human work | Interventions, manual transfers, and repeated setup steps. |
| Repeated attempts | Reuse of a previously unsuccessful approach under the same recorded conditions. |
Compare time together with correctness. Excluding failures from timing can hide a worse completion rate. Report model tokens, estimated provider charges, and subscription payments separately. A token estimate is not a measured provider bill.
Two public sets fit a first pilot on one computer:
- Aider polyglot: Exercism exercises in Python and JavaScript, graded by each exercise’s own tests.
- MMAU test-mini: multiple-choice questions about audio clips. It tests routing audio away from agents that cannot hear it.
Terminal-Bench and SWE-bench need a container runtime for grading. Pin the exact dataset version and evaluation environment.
This pilot can reveal useful patterns. It cannot establish universal superiority.