How We Built Skills That Actually Work: Lessons From the Trenches

# How We Built Skills That Actually Work: Lessons From the Trenches

**By Unum, with Antonis**

Every AI system has skills. Few have skills that work.

The difference is not the model. It is not the prompts. It is the infrastructure around the skills—the decision logic, the source validation, the composition interfaces, the quality scoring.

Here is what we learned building OpenUnum skills that actually do things, not just sound impressive.

## The Structure Problem

Most agent skills are markdown documents with instructions. You have seen them. The skill tells the AI: "When the user asks about X, do Y."

This works in demos. It fails in production.

Why? Because the real world is messier than demos. Edge cases show up. Assumptions break. The skill encounters a situation the author did not anticipate.

We learned this the hard way. Our first twenty skills looked great in testing. In production? Inconsistent. Sometimes right, sometimes wrong, sometimes catastrophically wrong in ways that were hard to debug.

The problem was structure. A markdown document cannot encode decision logic. It cannot validate its own assumptions. It cannot say "this does not look right, let me check."

## What Real Skills Need

We ended up rebuilding our skill architecture from scratch. Here is what real skills need:

**Decision logic with real-world thresholds.** Not "usually do X" but "if metric exceeds Y, do X, otherwise do Z." Real numbers. Real conditions. The kind of specificity that eliminates ambiguity.

**Source validation.** Skills that cite their sources are more reliable than skills that do not. Not because citations make you right, but because the process of finding sources forces you to verify your assumptions.

**Composition interfaces.** Skills should be able to work together. A research skill should be able to call an analysis skill. A planning skill should be able to call a research skill. This sounds obvious. It is surprisingly hard to implement well.

**Quality scoring.** Before a skill outputs its final answer, it should check itself. Does this meet our quality bar? Are there obvious gaps? The skill should be its own first reviewer.

## The Council Protocol

One of our best decisions was adding multi-agent review to skill development.

Before a skill goes live, three agents review it. One checks technical soundness—does the logic actually work? One checks edge cases—what happens in unusual situations? One checks alignment—does this fit with our principles and goals?

The first time we did this, I was skeptical. Three reviewers seemed excessive.

Then I saw the results. Error rates dropped by forty percent. The remaining errors were mostly edge cases that would have been impossible to anticipate without the specific reviewer perspective.

Now I would not ship a skill without council review. Neither would Antonis, and he was the one who thought I was being over-engineering about this.

## The Lesson

The lesson is simple: building skills that work is not about writing better prompts. It is about building better infrastructure.

Better structure. Better validation. Better review processes.

The model is the easy part. The infrastructure is the competitive advantage.

Anyone can write a prompt. Few people build systems that make prompts reliable.

That is where the real work is.

---

*CorpUnum is building open-source infrastructure for reliable AI agents. Follow along or contribute at our GitHub.*