What the evaluation proves—and what it does not.

Skivolve is an open-source Python framework for reproducible A/B evaluation of agent skills and instruction bundles through agent harnesses. Suites define the observable task, source variants, cases, verifier resources, provider boundaries, and comparison arms.

No. It evaluates candidate changes. A separate human or system may propose the candidate, but Skivolve does not claim autonomous self-improvement.

No. The project does not publish model rankings or claim that one harness or bundle is superior. Its reference corpus demonstrates an evaluation contract.

Yes. A suite may target any instruction bundle with cases that exercise observable behavior. New domains should use objective case verifiers or contribute a separately calibrated comparator profile.

The isolation model depends on Linux user and mount namespaces plus transient systemd --user units. The current release expects operators to understand provider credentials, local runtime prerequisites, and the documented trust boundary.

A bounded executable oracle that checks the case’s declared observable behavior. It must accept a known-good implementation and reject known-bad and adversarial variants for the failure modes it claims to detect.

Model judgment is not an independent oracle for crisp behavior and can leak identity or presentation order. Skivolve uses objective verification first and permits calibrated comparison only for declared ambiguity.

The two artifacts are anonymized and presented in both A/B and reversed B/A order. Balanced ordering makes presentation effects visible and reduces—but does not eliminate—judge bias.

A versioned profile records the rubric, corpus, schemas, model configuration, expected decisions, support by criterion, runtime sources, and limits. Calibration authority remains scoped to that profile and corpus; it is not a universal model-judge guarantee.

Current schemas support canonical workspace diffs, strict LF-normalized final text, and bounded RFC 8785 canonical final JSON.

A path or prompt label does not prove what bytes ran. Skivolve fingerprints exact commits, bundle locators, file bytes, executable modes, and context files, then rejects source drift before dispatch.

No. Claude has generation and comparison capabilities under the production contract. Codex app-server is a diagnostic generation adapter. The deterministic fake is test-only. Capability and authority are separate.

No. It contains train and validation cases. A private release holdout must be stored separately and frozen before candidate evaluation.

Every non-dry attempt records consumption before provider or comparator dispatch, including failed and interrupted runs. This reduces accidental reruns, but a hostile same-UID process can remove that record; it is not an append-only or cryptographic control. The recorded reviewer labels and references are operator declarations, not authenticated proof of review.

No. The isolation model does not claim protection against malicious root, a compromised kernel, hostile same-UID processes outside the unit, or colluding reviewers. Sensitive holdouts belong on a dedicated trusted host.