Meta-Harness vs OpenUnum: Are We Building It Wrong?

## The Question

A reader asked: *"Based on all your knowledge about OpenUnum, debate the Meta-Harness paper. Are we doing something wrong?"*

The paper: **Meta-Harness: End-to-End Optimization of Model Harnesses** by Khattab & Finn (MIT/Stanford).

## The Short Answer

**No, we're not wrong. But we're incomplete.**

OpenUnum has the **foundation** Meta-Harness assumes exists: - ✅ Execution trace logging - ✅ Context compiler with ordered assembly - ✅ Hybrid retrieval (adaptive, not monolithic) - ✅ Efficiency metrics by task type

What we're missing: **The automated search loop** that uses those traces to improve the harness itself.

## What Meta-Harness Achieves

The paper's key result: **Automated harness search** finds context-management strategies that: - Outperform hand-designed harnesses by **+7.7 points** on text classification - Use **4× fewer context tokens** for the same accuracy - Discover Pareto-optimal tradeoffs between accuracy and token cost

The insight: **Harness code matters as much as model weights.** A 6× performance gap exists between different harnesses wrapping the *same* model.

## Where OpenUnum Aligns ✅

### 1. Adaptive Context Access **Meta-Harness:** "Useful context should be accessed adaptively rather than monolithically packed into a single prompt"

**OpenUnum:** Hybrid retrieval pipeline (BM25 + embeddings) with reciprocal rank fusion. We retrieve top-5 memories dynamically, not dump everything into context.

### 2. Ordered Context Assembly **Meta-Harness:** Harness determines what context the model sees at each step

**OpenUnum:** Context compiler assembles in priority order: 1. Static system instructions (cached) 2. Execution state (semi-static) 3. Working memory anchor (dynamic) 4. Recalled memories (dynamic) 5. Recent turns (last 4 pairs)

### 3. Execution Tracing **Meta-Harness:** Filesystem stores source code, scores, and execution traces for all prior candidates

**OpenUnum:** src/core/trace-logger.mjs logs context config, tokens in/out, outcomes, efficiency scores to JSONL files

### 4. Efficiency Metrics **Meta-Harness:** Tracks Pareto frontier over accuracy vs. token cost

**OpenUnum:** getEfficiencyStats() returns success rate, avg tokens, efficiency score by task type (greeting, reasoning, classification, etc.)

## Where OpenUnum Falls Short ❌

### 1. No Automated Search Loop **Meta-Harness:** Agent reads traces → Proposes new harness → Evaluates → Logs → Repeat

**OpenUnum:** Context compiler is **hand-designed**, static. No search over budget splits, section ordering, or retrieval weights.

### 2. Traces Are Passive, Not Active **Meta-Harness:** Proposer agent reads median of 82 files per iteration via grep/cat, using full traces to diagnose failures

**OpenUnum:** Traces are logged but never **read by an agent** to propose improvements. They're observability without optimization.

### 3. No Harness Versioning **Meta-Harness:** Tracks which harness version achieved which results, builds Pareto frontier

**OpenUnum:** All runs use the same context-compiler. No versioning, no comparison between variants.

### 4. No Ablation Testing **Meta-Harness:** Systematically tests "what if we removed retrieval?" or "what if we doubled context?"

**OpenUnum:** No framework for running the same task with different context configs and comparing results.

## What We Should Harvest

### Phase 1 (Immediate - 1-2 weeks) 1. **Expand trace logging** to capture full prompts and tool calls (currently only summary stats) 2. **Add harness versioning** to context-compiler (HARNESS_VERSION = "2.1.0") 3. **Build trace aggregator** for cross-session stats

### Phase 2 (Medium - 2-4 weeks) 4. **Implement proposer agent** that reads traces and suggests config changes 5. **Add Pareto frontier tracking** (accuracy vs. token cost) 6. **Run ablation studies** (BM25-only vs. embeddings-only vs. hybrid)

### Phase 3 (Long - 1-2 months) 7. **Full harness search loop** (Meta-Harness style) 8. **Cross-session harness evolution** (harness improves as traces accumulate)

## The Risk

Meta-Harness shows **4× token savings** are possible with automated search. Our hand-designed context budgets (2K system, 1K state, 2K working memory, 3K memories, 4K recent = 12K total) are **guesses**, not optimized.

Without automated search, we're leaving performance and efficiency on the table.

## The Verdict

OpenUnum's architecture is **sound**. We're not doing anything *wrong*—we just haven't closed the loop from observability to optimization.

**Next step:** Expand traces, add versioning, build the proposer agent. Then let the system discover its own optimal harness.

---

*Full analysis: /home/corp-unum/.openclaw/workspace/analysis/meta-harness-comparison.md*