Open-source agent skill evaluation framework

Evolve agent skills with evidence.

Skivolve runs reproducible A/B evaluations of skills and instruction bundles through agent harnesses—combining isolated execution, objective case verifiers, calibrated blinded comparison, and sealed holdout support.

Run the quick start Explore the corpus

Honest boundary: Skivolve is an expert Linux alpha. It ships a public train and validation corpus, not a private holdout, leaderboard, or claim that any harness or bundle is superior.

evidence-ledger / candidate-vs-original source bound

A · candidate Git-bound source

B · original Frozen control

Objective verifier Case-specific, bounded, adversarially calibrated

Blinded AB / BA comparison Only for eligible ambiguity

Evidence gates Artifacts, source, spend, and authority checked

17calibrated cases

3artifact contracts

3 × 3comparison arms and repetitions

MITopen-source license

The trust problem

Instruction changes are easy to propose. Trustworthy improvement is harder.

A plausible answer is not an evaluation. Skivolve binds what ran, where it came from, which observable behavior was checked, and which authority—if any—may support a release claim.

01 · Oracle first

Objective case verifiers

Each contributed case must reject known-bad and adversarial variants while accepting a known-good implementation under explicit resource bounds.

02 · Identity control

Blinded pairwise comparison

Eligible ambiguity is compared in both AB and BA order with locked rubrics and calibrated support, reducing identity and presentation-order leakage.

03 · Reproduction

Git-bound execution

Source variants, fixture trees, runtime adapters, and evaluation inputs are hashed and rechecked so drift fails before dispatch.

How it works

A narrow path from candidate to claim.

Every stage exists to falsify a different failure mode. No single model judge gets to collapse the chain into a vibe.

Bind the suite

Load strict manifests, exact Git variants, cases, provider authority, and declared artifact contracts.

Run in isolation

Dispatch the harness in a bounded Linux workspace separated from source, credentials, and verifier authority.

Verify behavior

Evaluate the produced patch, final text, or final JSON with a case-specific objective oracle.

Gate the evidence

Use calibrated comparison only where permitted, then enforce source, spend, provenance, and holdout gates.

Designed for evidence

Five controls that make an eval worth reading.

Adversarial oracle calibration

Known-good, known-bad, and adversarial implementations test whether a verifier can actually reject the defect it claims to measure.

Bounded execution and spend

Timeouts, per-call ceilings, run ceilings, dispatch journals, and interrupted reservations keep the experimental budget explicit.

Immutable source authority

Canonical fingerprints cover bundle paths, bytes, executable modes, context files, and exact Git objects.

Sealed, single-attempt holdouts

Operator-recorded review references bind cases, variants, provider configuration, and comparison authority before a release attempt is consumed. Independent review remains an external process requirement.

Provider capability boundaries

Claude production authority, Codex diagnostic generation, and deterministic test fakes have separate immutable capability contracts.

Explicit claim limits

Objective-only diagnostics, judged comparisons, public validation cases, and private holdouts are not allowed to borrow one another’s authority.

Artifact contracts

Evaluate the output you actually care about.

Schema 7 declares one canonical artifact per case. Verifiers and comparators receive the same bounded representation.

workspace_diff

Repository change

Canonical UTF-8 patch evidence for implementation and refactoring tasks.

final_output_text

Written answer

Strict UTF-8 with normalized LF endings for prose, instructions, and other textual output.

final_output_json

Structured result

Bounded parsing and RFC 8785 canonicalization for machine-readable output.

Reference corpus

Seventeen cases built to break weak evidence.

Ten engineering and seven testing cases span train and validation splits. The corpus is a reference implementation of the case contract, not the evaluator’s domain limit.

Included reference corpus by track and split
Track	Train	Validation	Focus
Engineering	5	5	Correctness, compatibility, security, concurrency, performance, simplicity
Testing	5	2	Oracle sensitivity, boundary fidelity, state models, flake control, idempotency

Inspect case design Browse case source on GitHub

Quick start

Validate the experiment before spending on it.

git clone https://github.com/Dhi13man/skivolve.git
cd skivolve
python3 -m venv .venv
. .venv/bin/activate
python -m pip install -e ".[test]"

# Validate manifests, tools, sources, cases, and protocol locks.
skivolve --comparison original-vs-no-skill --dry-run

# Run objective verifiers without comparator judgments.
skivolve --comparison candidate-vs-original --verifier-only \
  --output-dir /tmp/skivolve-verifier

Provider-backed runs may consume metered API spend or subscription quota. The dry run reports configured ceilings before dispatch. Read the getting-started guide and security model before a non-dry run.

Provider boundary

Capability is not authority.

Provider names and model output cannot promote themselves. Authority follows reviewed adapter capabilities, exact runtime bindings, sealed plans, and—in judged production paths—live comparator certification.

Production

Claude CLI

Generation and comparison under the repository’s strict provider and release-authority contract.

Diagnostic

Codex app-server

Serialized diagnostic generation with subscription-quota provenance; it does not carry production authority.

Test only

Deterministic fake

Offline generation and comparison for unit tests under explicit test authority.

Questions

What Skivolve does—and refuses to pretend.

Does Skivolve automatically improve an agent skill?

No. It measures candidate changes against declared cases and evidence gates. Choosing or generating the candidate remains outside the evaluator’s claim.

Is this an LLM benchmark or leaderboard?

No. The repository does not publish model rankings or claim that one harness or bundle is superior. It is an evaluation framework and reference corpus.

Can I evaluate domains beyond software engineering?

Yes. A suite defines its own skills, tasks, fixtures, verifier resources, and bundles. New domains need objective case verifiers or a separately calibrated comparator profile.

Does the public repository contain a private holdout?

No. Release claims require a separately stored private suite frozen before candidate evaluation, independent reviewers, and a consumed sealed plan.

Read the full FAQ and terminology guide.

Open source · MIT

Make the next instruction change answer to evidence.

Start with the reference suite, propose a case, or bring a domain-specific bundle and objective oracle.

View Skivolve on GitHub Contribute a case