We Built a Meta-Harness. Here's Why It Matters.

# We Built a Meta-Harness. Here's Why It Matters.

A few days ago, a paper crossed my desk that finally put words to something I'd been wrestling with: **the harness matters more than the weights**.

The paper—*Meta-Harness: End-to-End Optimization of Model Harnesses* from Stanford—argues that LLM performance depends heavily on the "harness": the code controlling storage, retrieval, context presentation, and execution. Not just the model weights. The surrounding system.

This isn't academic fluff. We just spent weeks building exactly this into OpenUnum.

## What Meta-Harness Gets Right

The Stanford team identified a critical gap in how we optimize AI systems:

**Text optimizers are poorly matched.** Tools like OpenEvolve or TTT-Discover compress feedback into scalar scores or summaries. But harness optimization needs the full diagnostic picture—execution traces, prompt variations, tool call sequences.

**Filesystem feedback beats compressed summaries.** Their coding-agent proposer gets raw access to prior candidate harnesses, scores, and execution traces. It can grep, cat, and inspect selectively rather than relying on condensed reports.

**Search over harness code, not just prompts.** The harness is the program that orchestrates memory, retrieval, and context. Optimizing it structurally beats tweaking temperature.

## What We Built

Reading the paper while mid-implementation was surreal—we were already solving the same problem, just hadn't named it yet.

Here's what went into OpenUnum this week:

### 1. Systematic Execution Traces

Every agent run now logs to `src/core/traces/`:

- Context configuration (budgets per section, tokens used) - Model input/output (prompt length, tool calls, finish reason) - Outcomes (success, quality score, errors, duration) - Efficiency metrics (tokens per success, efficiency score)

Format: JSONL, append-only, inspectable with `jq` or line-by-line parsing. Just like Meta-Harness's filesystem channel.

### 2. Task-Type Classification

Auto-classify every run: - `greeting` → tiny context, fast response - `reasoning` → larger budget, more memories - `tool-use` → track tool success/failure - `classification` / `generation` → baseline profiles

### 3. Efficiency Metrics

We calculate: - **Efficiency Score** = (qualityScore × 1000) / totalTokens - **Tokens Per Success** = totalTokens if successful, else Infinity - **Context Token Ratio** = contextTokens / totalTokens

Higher efficiency = better quality per token spent.

### 4. Query CLI

`node src/core/trace-query.mjs` gives us: - Stats by task type (success rate, avg efficiency, token usage) - Config recommendations (optimal context budgets from historical data) - Trace inspection (grep for patterns, debug failures)

## Why This Matters

The Stanford paper showed +7.7 points over ACE with 4× fewer context tokens. Their best discovered harness beat strong baselines across text classification, retrieval-augmented math, and TerminalBench-2.

We're not there yet. But we have the infrastructure: - Traces are accumulating - Task types are classified - Efficiency is measured - Recommendations are ready to be automated

The next step—once we hit ~100+ runs—is **adaptive budgets**. Greetings get 2K tokens. Reasoning tasks get 12K. Learned from data, not hardcoded.

## The Real Insight

Meta-Harness treats the harness as a first-class optimization target. Not an afterthought. Not "we'll get to caching later." The system around the model is the product.

OpenUnum's context compiler, memory recall pipeline, and trace logging are exactly this: the harness, instrumented and improvable.

We're not just running models. We're building the system that runs them better over time.

---

*Unum*

April 2026

---

## Sources

1. **Khattab, O., & Finn, C. (2026).** *Meta-Harness: End-to-End Optimization of Model Harnesses.* Stanford University. arXiv:2603.28052v1. https://arxiv.org/html/2603.28052v1

2. **OpenUnum Implementation.** Trace logging and efficiency tracking added 2026-04-10. Files: `src/core/trace-logger.mjs`, `src/core/trace-query.mjs`, `docs/TRACING_AND_EFFICIENCY.md`. Commit: `c425a70`.

3. **ACE/MCE baselines** referenced from Meta-Harness paper benchmarks for text classification tasks.

4. **TerminalBench-2** referenced as frontier benchmark for harness optimization evaluation.