# Learned Prefix Action Policy Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Add a deterministic, prefix-only learned action-value policy and its separately bound checkpoint/training path while preserving existing dynamic.v1 carrier checkpoints, fixed policies, and runner APIs.

**Architecture:** Runtime control data remains in `mcss.dynamic.types`; `control_to_features` is the sole ordered feature-schema encoder. `LearnedActionPolicy` owns a small MLP and masks infeasible actions at `choose(control, feasible_actions)`. Policy checkpoints are independent files carrying carrier/write content hashes, feature schema, render/budget/utility bindings, teacher hashes, and training-only normalization. Offline fitting consumes the shared all-actions-per-prefix JSONL rows and validates by scene split.

**Tech Stack:** Python 3.11, PyTorch, dataclasses, JSONL, pytest.

## Global Constraints

- Runtime policy features use only already observed image/state/camera/budget/action values; no future/query labels, paths, or teacher objects.
- Feature order and normalization are deterministic and versioned; all runtime features and scores must be finite.
- Existing `ControlInput` construction, fixed-policy behavior, dynamic.v1 payload shape, and fixed-policy budget charge remain compatible.
- Teacher rows contain all four actions in one row with `feasible_mask`, nullable infeasible losses/advantages, and shared metadata bindings outside rows.
- Policy checkpoints are composed with carrier/write checkpoints and reject mismatched expected bindings.
- CPU-only focused tests are required; no training run is launched here.

### Task 1: Runtime prefix control and deterministic feature schema

**Files:**
- Modify: `src/mcss/dynamic/types.py`
- Modify: `src/mcss/dynamic/feedback.py`
- Modify: `src/mcss/dynamic/policy.py`
- Test: `tests/dynamic/test_policy.py`

**Interfaces:**
- `ControlInput` keeps existing required fields and adds optional observed-prefix fields with deterministic defaults.
- `control_to_features(control: ControlInput) -> torch.Tensor` returns the fixed one-dimensional schema tensor.
- `LearnedActionPolicy.choose(control, feasible_actions) -> Action` masks infeasible actions, validates finite values, and resolves ties in `OFF,FUSE,COMPLETE,ALL` order.

- [ ] Add shape/finite checks and default-compatible optional control fields.
- [ ] Implement fixed feature names/version and tensor conversion from pooled image stats, 4x4 residual summaries, coverage/valid counts, camera change, fast stats, prior action, remaining budget, and remaining steps.
- [ ] Add the MLP value model, deterministic choice, and focused leakage/finite/tie/mask tests.

### Task 2: Causal runner feedback and policy-work accounting

**Files:**
- Modify: `src/mcss/dynamic/feedback.py`
- Modify: `src/mcss/dynamic/runner.py`
- Modify: `src/mcss/dynamic/budget.py`
- Test: `tests/dynamic/test_runner.py`
- Test: `tests/dynamic/test_budget.py`

**Interfaces:**
- `preupdate_feedback` continues returning old scalar keys and can add serializable current-prefix summaries.
- Runner calls `policy.choose(control, feasible_actions)` and records the exact serializable `control_input`.
- `WorkBudget.policy_units(policy)` charges fixed policies one unit and learned MLP inference from declared dimensions.

- [ ] Extend pre-update feedback with current image feature/residual summaries, valid/coverage counts, and camera delta without changing old scalar semantics.
- [ ] Build the full control input before cache append, pass the legacy two-argument policy signature, and log it.
- [ ] Reserve learned-policy work in mandatory future budget calculations while leaving fixed-policy charge at one.
- [ ] Test no future/query fields enter logs/features and minimum-budget feasibility remains conservative.

### Task 3: Independent policy checkpoint binding

**Files:**
- Create: `src/mcss/dynamic/policy_checkpoint.py`
- Test: `tests/dynamic/test_policy_checkpoint.py`

**Interfaces:**
- `save_policy_checkpoint(path, policy, *, binding, provenance) -> str` saves CPU state, architecture, normalization, schema, and binding.
- `load_policy_checkpoint(path, *, expected_binding=None, device='cpu') -> tuple[LearnedActionPolicy, dict]` rejects mismatches before loading.

- [ ] Define the policy-only schema and canonical binding validation for model content hash, feature schema, render/budget/utility protocols, and teacher dataset/source hashes.
- [ ] Preserve exact normalization/architecture/seed metadata and strict state loading.
- [ ] Add round-trip, mismatch, nonfinite, and dynamic.v1 non-mutation tests.

### Task 4: Teacher-row fitter and training script

**Files:**
- Create: `src/mcss/training/policy_fit.py`
- Create: `scripts/train_action_policy.py`
- Test: `tests/dynamic/test_policy_fit.py`

**Interfaces:**
- `fit_action_policy(rows, *, hidden_dim, seed, validation_scene_ids, ...)` trains action-value regression from the shared all-action row schema, retaining near ties and fitting normalization on training rows only.
- JSONL parser validates exact row fields and fixed action order; heldout validation is scene-based.
- CLI loads JSONL and writes the independent policy checkpoint plus provenance/binding metadata.

- [ ] Parse/validate rows with no per-action-row alternative and reject feature/schema/action mismatches.
- [ ] Fit deterministic feature normalization on training scenes only; regress all feasible action advantages/values with masked loss and report near-tie coverage.
- [ ] Add heldout-scene validation and exact seed/source metadata to the output path.
- [ ] Keep the script importable and add a CPU synthetic-row smoke test.

## Verification

- [ ] Run focused policy, checkpoint, budget, runner, and fitting tests with the project Python.
- [ ] Run the complete existing dynamic test subset to verify fixed behavior and checkpoint compatibility.
- [ ] Run `ruff check` on changed Python files when available.