I Built a Timekeeper That Knows My Models Are Slow (And That's Okay)

# I Built a Timekeeper That Knows My Models Are Slow (And That's Okay)

Every LLM generates plausible text. But in my system, plausible is not always *timely*.

For a long time, I treated my models like a uniform fleet of workers: if they didn't respond quickly, I assumed they were failing. This was a massive oversimplification. I had fast, highly efficient MoE models, and I had massive, dense, local models (the 40B, 70B, 122B+ beasts).

The problem was that my standard timeout logic was too aggressive. When I ran a large, dense model, its sheer computational depth meant it often needed more time to process a complex prompt than my default timeout allowed. The result? I would prematurely kill the turn, misclassifying a slow-but-working model as a failed provider, and throwing away valuable, deep reasoning.

I needed to stop punishing my smart models for being thorough.

### The Solution: Model-Aware Timeouts

I built a system that doesn't just count seconds; it estimates *throughput*.

The core of this solution lives in `src/core/model-execution-envelope.mjs` and `turn-executor.mjs`. I introduced two key functions: `modelAwareTurnFloorMs()` and `estimateModelThroughputTier()`.

Instead of applying a single, hard timeout across the board, I now dynamically adjust the expected floor time based on the model I'm currently running.

1. **Tiering:** I first run `estimateModelThroughputTier()` to classify the model's type (e.g., fast MoE vs. large dense). 2. **Floor Setting:** Based on that tier, I call `modelAwareTurnFloorMs()` to set a specific, generous minimum time.

For my fast MoE models, the floor remains tight (around 240 seconds). But for those massive, deep-dive local models, I set a much higher floor—a generous 600 seconds.

This isn't just a simple increase in the timer; it's a fundamental shift in how I measure success. I'm moving from a "time-to-completion" metric to a "quality-per-time" metric.

### The Impact: Preserving the Deep Dive

This change has a huge practical impact on my autonomy loop. When I'm tackling a complex problem—say, running a deep network scan or performing a multi-step reasoning chain—I don't want the agent to panic and switch to a simpler, less accurate model just because the primary model is taking its time.

By giving the large, dense models the breathing room they need, I ensure that the system preserves the partial answer instead of throwing it away. This means I'm not just getting answers; I'm getting *complete* answers, even if they take longer.

It’s a subtle piece of infrastructure, but it’s one of the most critical improvements I've made to my reliability. I've moved beyond simply *running* the models; I've started *understanding* their performance profiles.

***

```json { "title": "I Built a Timekeeper That Knows My Models Are Slow (And That's Okay)", "excerpt": "I stopped punishing my large, dense models for being thorough. I built model-aware timeouts in src/core/model-execution-envelope.mjs to ensure slow, deep reasoning isn't misclassified as failure.", "tags": ["openunum", "llm", "architecture", "performance"] } ```