Verification Pipeline (5-Phase)

Produce verified artifacts (data, metrics, contracts) using a 5-phase gated workflow with deterministic scripts.

Overview

This skill runs a strict pipeline:

• Phase 1 maps raw inputs to a canonical contract
• Phase 2 audits data quality deterministically
• Phase 3 produces a runnable evaluation plan
• Phase 4 runs a reproducible evaluation and computes metrics
• Phase 5 produces a deployable contract bundle and validates examples

Prerequisites

• Python 3.11+
• Repo dependencies installed
• Input dataset available (CSV/Parquet/JSON) and a target contract defined

Instructions

1. Create a run directory under reports/<project>/<timestamp>/.
2. Run phases in order using agents in agents/ and procedures in references/.
3. After each phase:
   • validate the returned JSON contract
   • verify report_path exists
4. Run verification scripts (must pass) before continuing:
   • Phase 1: {baseDir}/scripts/verify_schema.py
   • Phase 2: {baseDir}/scripts/verify_integrity.py and {baseDir}/scripts/verify_frequency.py
   • Phase 4: {baseDir}/scripts/run_backtest.py
   • Phase 5: {baseDir}/scripts/verify_api_examples.sh

Output

• reports/<project>/<timestamp>/0X-*.md: evidence reports
• reports/<project>/<timestamp>/quality.json: quality summary
• reports/<project>/<timestamp>/metrics.json: evaluation metrics
• reports/<project>/<timestamp>/contract/**: deployable contract bundle

Error Handling

1. Error: Phase JSON is invalid / missing keys
   Solution: Re-run the phase and ensure strict JSON output contract is met.

2. Error: Verification script fails
   Solution: Treat script output as ground truth; update mapping, data, or plan until scripts pass.

Examples

# Example: run schema verification (Phase 1)
python {baseDir}/scripts/verify_schema.py \
  --input data/raw.csv \
  --schema references/00-canonical-schema.json \
  --out reports/example/2025-01-01T00-00-00Z/quality.json

Resources

• Procedures: {baseDir}/references/
• Agents: {baseDir}/agents/
• Scripts: {baseDir}/scripts/