Coding OS
Sign in
Browse docs▾

title: "ADR-0016: Gated skill-evolution — extend the existing learning loop, do not build a training stack" domain: META layer: adr status: accepted updated: 2026-07-23

ADR-0016: Gated skill-evolution — extend the existing learning loop, do not build a training stack

Purpose: Record how coding-os should (and should not) adopt the "skill-document-as-trainable-weights" idea from an external, published skill-optimization method — so a future agent extends the existing subsystems instead of standing up a parallel training stack. Read when: proposing anything that auto-improves skills/rules from experience, or evaluating a "let's train our skills" idea. Skip when: hand-authoring a single skill or rule. Read next: raptor-consolidation.md · anti-overengineering.md · memory.md.

Nav: ADR Index | Docs Index

Status

Accepted (2026-07-23).

Context

An external, published method (2026) treats an agent's natural-language skill document as the trainable "weights" of a frozen model and optimizes it with a deep-learning-style loop — rollout (run scored tasks) → reflect (an optimizer model emits bounded add/delete/replace edit patches, the "gradient") → aggregate → select/clip (keep at most learning_rate edits) → update → gate on a held-out validation split (accept only a strict improvement) — plus a nightly offline companion that harvests real session transcripts, mines recurring tasks, replays them, consolidates behind the gate, and stages a proposal for a human to adopt. The deployed artifact is a compact skill doc run against the unchanged model, so it adds zero inference-time cost.

A subsystem-by-subsystem audit of coding-os found the fit is real but narrow: cos has already, independently, built most of the machinery, and stops one step short of the method's one load-bearing property.

Method component Existing cos home
Confidence/SGD-step/gradient-clip + weight store learned_patterns + LTP/LTD (boost_success/penalize_failure) + Ebbinghaus decay (learning.py, decay.py)
Nightly offline "sleep" cycle nightly.py CRON A (gated legs) + responsive_extract (Stop hook) + CRON B weekly narrative
Replay / rollout engine sdk_dispatcher.dispatch — a fresh sub-session against an agent_file + input_slice
Optimizer-model role the CRON B distiller agent + cos_learn_narrative
Stage-proposal → human-adopt generalize_lessons → .coding-os/memory/drafts/, cos_promote, governance task-marker + block-protected-files.sh
Rejected-edit buffer decay archive-at-floor (promoted_to='archived') + LTD

What cos lacks — and it is the hard piece: (1) no held-out validation split exists anywhere — every gate (verify ledger, DoD, anti-ambiguity) runs the project's own in-band suite against the current tree, never a retained task set scoring a guidance edit; (2) the consolidation loop never emits bounded edits to canonical SKILL.md / rule .md bodies (it emits new-file drafts and confidence scalars); (3) the reward label (agent_metrics.outcome) is agent-self-reported and gameable.

Decision

  1. Extend three existing subsystems; do not build a training stack. The value of the external method for cos is conceptual (the gate discipline + an independent reward), not its implementation. Reuse learning.py, nightly.py, and sdk_dispatcher.
  2. Sequence smallest-first, defer the L-effort behind a feasibility spike:
    • [TASK-850] (S, ship-alone value) — derive an additive derived_outcome reward label from the tree-keyed verify ledger, not self-report.
    • [TASK-851] (spike) — before any eval build, measure whether the real task stream has enough recurring, checkable instances to form a held-out set with signal above stochastic-rollout variance, under a per-night cost ceiling. Go/no-go gates the L-effort.
    • [TASK-852] (feature, depends on TASK-851) — only then, bounded learning_rate-clipped prose-only edit patches on SKILL.md/rule .md, staged as drafts and applied only behind the gate and a human governance task, with a draft-expiry policy.
  3. Immediate hardening (done in [TASK-849]): block-protected-files.sh now guards the src/core/skills/ and src/core/rules/ sources, not just their rendered .claude/ copies — the source propagates to every consumer via live symlinks, so it must carry at least the same governance-marker guard. This is the precondition that makes any future automated skill-editing (TASK-852) staged and adopted rather than silently live.
  4. An automated skill/rule editor targets SKILL.md prose bodies + the 8 hand-written rule .md + stack.yaml source rows — never the derived skill-enforcement.md/dimension-registry.md (regenerated, freshness-gated) and never front-matter (globs/description are load-bearing for enforcement and trigger accuracy).

Deliberately not built (parasitic parts avoided)

A new belief/confidence engine · a new scheduler/cron · a new replay engine · a freeform record(insight) tool or a from-scratch optimizer agent · a standalone rejected-edit table · a branch/worktree A-B canary (Rule 23 forbids branches) · an optimizer editing the derived rule files. Each already has a home above; adding it would duplicate the learning.py loop.

Raptor-lens review

Per raptor-consolidation.md — a design that adds parts must name the capability that pays for each one.

  • Component consolidation (1) + parasitic-complexity elimination (3): the roadmap adds zero new subsystems — every step is a leg, a column, or a patch inside an existing unit. The "deliberately not built" list is the parasitic mass this ADR refuses.
  • Zero-overhead abstraction (2): the immediate hardening adds two path checks to one existing hook and reuses its escape-hatch verbatim — denser coverage, no new mechanism. It is a no-op in consumer projects (they have no src/core/ tree), so blast radius is meta-repo-scoped despite the live-symlink reach.
  • High cohesion (4): the reward label lives where the outcome is recorded; the eval leg lives in nightly.py beside the mining it depends on; the edit-patch path lives in the distiller that already drafts.
  • Part-count honesty: the one genuinely new capability — a held-out validation gate — is explicitly gated behind a feasibility spike (TASK-851) because at solo-repo scale its signal-vs-noise is unproven; we do not pay for a part whose capability we have not measured.

Spike verdict — NO-GO, measured 2026-08-11 (TASK-851)

The recorded risk materialised. Measured over task_outcomes on the dogfood repo, after TASK-850 landed the derived_outcome label:

Measure Value
Labeled outcomes 61 of 757 (8%)
Class balance 59 success, 1 rework, 1 blocked — 96.7% one class
Largest recurring cluster (domain × complexity × type) 14 instances
Binomial SE at that rate and size 4.8pp
Headroom above the current success rate 3.3pp
Smallest observable change (one flipped instance) 7.1pp

The noise floor exceeds the entire measurable improvement. A held-out split built from this stream cannot register a guidance improvement at all: the best achievable gain (3.3pp) is smaller than one cluster's sampling error (4.8pp), and the smallest change the metric can even express (7.1pp) is more than double the headroom. Spending per-night rollout cost to gate on that would be paying for a part whose capability we have now measured to be zero.

So stage 2 (eval leg) and stage 3 (TASK-852, bounded edit patches) are not built; TASK-852 is archived rather than parked, because "wait for volume" with no trigger is the icebox-parking failure mode this ADR itself names.

Re-open trigger — re-run this measurement when the labeled stream reaches two classes with ≥30 instances each, in a single recurring cluster. Until then the hand-authored status quo is strictly safer than an ungated auto-editor reaching every consumer through live symlinks.

Consequences

  • The strongest, lowest-regret move (TASK-850) improves the current loop immediately and stands even if the held-out gate proves infeasible.
  • Biggest risk (recorded, not solved): a held-out gate may be statistically infeasible on a single-dev repo whose task stream is a long tail of one-offs; stochastic rollouts scoring a thin set decide on sampling noise while spending real per-night cost. TASK-851 exists to kill the build early if so. Without a working gate, auto-editing SKILL.md through live symlinks is more dangerous than the hand-authored status quo.
  • Reward-hacking: the ledger label must be paired with a lagging quality signal (rework/reopen rate over following sessions) so an edit that narrows verify scope cannot be accepted as "improvement" (TASK-850/TASK-852 scope note).
  • Human throughput is the real rate limiter: staged drafts need a human governance task each; without a draft-expiry policy they pile up stale (the icebox-parking failure mode) — TASK-852 owns the decay policy.

References