01Research / AI Engineering

Agent harnesses, in three tiers.

The model is the brain. The harness is the body that lets it actually do work — and it can matter more than which model you pick. This guide goes from a plain-English overview to how harnesses are graded to building your own. Read the tier that fits you.

01Overview

What a harness is, and why it matters — in five minutes.

Start here →

02Deep dive

The anatomy, the vocabulary done right, and how harnesses are graded.

Go deeper →

03Build

A technical guide to building (or buying) your own harness.

Get technical →

01Tier 1 · Overview

What a harness is — and why it matters.

For owners and operators · 5 min

A large language model, on its own, can only talk. It cannot open your files, send an email, update your CRM, run a report, or remember what it did yesterday. A harness is the software wrapped around the model that gives it those abilities and keeps it on task. The model is the brain; the harness is the body — hands, memory, and the discipline to keep working until the job is done.

The one idea worth keeping

Most people shop for a model— “should we use GPT or Claude?” That is the wrong question. The same model inside two different harnesses produces very different results. When LangChain changed only the harness around their coding agent — same model — its score on a standard task jumped from 52.8% to 66.5%. The scaffolding around the model moved the result as much as a model upgrade would.

The takeaway for a business: “We use AI” tells you almost nothing. What decides whether it actually helps is the harness — the tools you give it, the guardrails you set, and how it is wired into your work. That is the part you choose, and the part we build.

It is not just for coding

Harnesses run far more than software. The same pattern — a model plus tools plus a loop plus guardrails — runs real business work, for every kind of small business:

  • — Front office: an agent that answers inbound, books the job, and logs the call to your CRM.
  • — Back office: reading invoices and receipts, prepping reconciliation, generating the weekly report.
  • — Sales: researching a lead, drafting the follow-up, updating the pipeline.
  • — Software: fixing a bug, writing tests, reviewing a change.

Tools you may have heard of map onto this neatly: Claude Code is a harness for software work; Lindy builds “AI employees” for the inbox, CRM, and calls; Hermes is a self-hosted assistant that learns your business over time; and Paperclip coordinates a whole team of agents like a company. We will sort out exactly what each one is in Tier 2.

The five words, in one line each

  • — Model — the brain. Predicts text; knows nothing about your business by itself.
  • — Scaffolding — what you hand the model before it starts: its instructions, its tools, its format.
  • — Harness — the body at runtime: runs the tools, manages memory, enforces guardrails.
  • — Orchestrator — the manager: decides what runs next and coordinates several agents.
  • — Runner — the dashboard: lets you launch and watch agents working.

If you only remember one thing: you do not deploy a model, you deploy a harness — and its quality, not the logo on the model, decides what you get.

02Tier 2 · Deep dive

Anatomy, the vocabulary done right, and how to grade one.

For evaluators and decision-makers · 12 min

The anatomy of a harness

At the center of every harness is a loop: the model proposes an action, the harness executes it, the result is fed back, and the loop repeats until the work is done or a stop condition is hit. Around that loop sit the subsystems that make it reliable:

  • — Scaffolding — system prompt, tool definitions, output format, and a project-level instruction file (Claude Code reads a CLAUDE.md so your conventions persist into every run).
  • — Tools — file and shell access, search, HTTP/API calls, domain actions. Increasingly standardized via MCP (the Model Context Protocol).
  • — Context management — deciding what slice of your data goes in front of the model each turn, and compacting as the task grows.
  • — Memory / persistence — keeping state across a task and across sessions; the best harnesses save reusable skills.
  • — Guardrails — what the agent may touch, when it must stop and ask, and approval gates for risky or irreversible actions.
  • — Observability — logging every action, tracing decisions, tracking token cost. You cannot improve what you cannot see.

A useful design principle: a good harness gets simpler as models improve. If you need more scaffolding every time a better model ships, the design is fighting the model rather than riding it.

The vocabulary done right (this is where cmux trips people up)

These words get used interchangeably and it causes real confusion. The clean distinctions:

  • — Harness = capabilities + execution (“the hands”). Runs tools, manages context, enforces safety, persists work. Examples: Claude Code, Hermes.
  • — Orchestrator / control plane = decision logic and multi-agent coordination (“the brain”). Examples: Paperclip, Relevance AI's workforce.
  • — Runner / supervision = drives a loop or lets a human watch many agents at once. Example: cmux — a terminal for supervising parallel coding agents. It manages where you see agents, not what they do.
  • — Framework = components you assemble yourself. Examples: LangChain, CrewAI.
Correction worth stating plainly: cmux is not a harness — it is a runner / supervision layer that sits above harnesses. Likewise Paperclip is an orchestrator, not a harness. Getting this right matters because you buy them for different jobs: a harness does the work, an orchestrator coordinates several, a runner lets you watch them.

How harnesses are graded

“Which is best” has no single answer, because harnesses are graded on several axes and different businesses weight them differently. A law firm cares most about safety and observability; a high-volume shop cares about cost; a solo operator cares whether it compounds. The axes:

DimensionThe questionWhat to measure
ReliabilityDoes it actually finish the job?Task success rate, tool-selection accuracy (aim ≥0.9), faithfulness (≥0.8), partial credit on sub-goals, end-to-end vs. component-level
Cost / efficiencyWhat did the result cost?Tokens, steps, latency, path efficiency vs. optimal; budget caps with hard stops
SafetyCan it be steered into harm?Multi-turn jailbreak resistance (most attacks unfold over 3–5 turns), guardrail enforcement, human-approval gates on risky actions
ObservabilityCan you see what it did?Per-action logging, full traces, token/cost tracking, anomaly surfacing, root-cause attribution
CompoundingDoes it get better with use?Persistent memory, self-authored/refined skills, a model of your business that improves over sessions (e.g. Hermes)

Two notes that save money. First, measure efficiency, not just success: an agent that takes 50 steps for a 5-step job is failing even when it “works.” Second, evaluation itself has a cost — left unbounded it can run many times the agent's own operating cost, so cap it.

Reading benchmarks without getting fooled

Most numbers you will see are marketing. Three rules:

  • — Distrust static benchmarks models can memorize. SWE-bench Verified was the standard until frontier models began reproducing its answers from memory; its 2026 successor, SWE-bench Pro, withholds 2,000+ problems for this reason.
  • — Distrust vendor benchmarks. A tool's own blog always ranks that tool first. Demand independent evaluation grounded in real behavior.
  • — Match the benchmark to your task. A code-review score tells you nothing about call-handling.

The model for doing this right is Martian's Code Review Bench: an independent lab (it sells no model or tool) that scores agents on real developer behavior across ~300,000 pull requests and reports precision (of what it flagged, how much was real), recall (of the real issues, how many it caught), and F1 (the balance) — open source, continuously updated, judged by multiple independent models. Real-world signal, independence, transparency, stated caveats: that is the bar. A number with none of those is an ad.

The landscape, sorted correctly

The same tools, filed under the right category — so you can see what each is actually for:

Harness · coding

Claude Code / Codex / Aider

Terminal agents that operate directly on a codebase — read files, run tests, edit, repeat. The workhorses of agentic development. (Aider is open source and git-native.)

Harness · self-improving

Hermes Agent (Nous Research)

A self-hosted, open-source agent with a built-in learning loop: it writes and refines its own reusable skills from experience and keeps persistent memory across sessions, so it compounds — getting more useful the longer you use it. Runs as a daemon on a small VPS and reaches you over Telegram, Slack, email, and more.

Harness · business ops

Lindy

No-code “AI employees” for operations: sort the inbox, draft replies from history, update the CRM, join and summarize meetings, even handle calls. You describe the job in plain English and it builds the workflow. Strong fit for small teams.

Orchestrator · workforce

Relevance AI

Builds multi-agent “workforces” where each agent owns one step and passes context to the next (e.g. an SDR team: sourcer → researcher → copywriter → sender). An orchestrator that coordinates agents, with scheduling and approval gates.

Orchestrator · control plane

Paperclip

Open-source “control plane” to run a business with a team of agents: an org chart (CEO/CTO/etc.), goals, heartbeat scheduling, hard cost caps, audit trails, and human approvals. As its community puts it: an agent is an employee, Paperclip is the company. It coordinates harnesses — it is not one itself.

Runner · supervision

cmux

A native terminal for running and watching several coding agents at once — panes glow when an agent needs you. It manages where you see your agents, not what they do. A runner / supervision layer, NOT a harness.

Self-driving loop

Ara (ara.so)

A “self-driving IDE” that proposes the next change, not just executes the one you assign — moving the human from operator to approver. An autonomous loop built on top of a harness.

Framework · build-your-own

LangChain / CrewAI

Toolkits to assemble your own agent. A framework gives you components; a harness assembles them into a running system. LangChain’s DeepAgents, for instance, is a harness built on the LangChain framework.

Standings shift monthly and most vendor numbers are self-reported snapshots — verify against an independent benchmark such as Martian Code Review Bench before betting on a ranking.

03Tier 3 · Build

Building (or buying) your own harness.

For technical teams · 15 min

Start with the smallest thing that works

The simplest harness is a single loop with a supervisor watching it. In pseudocode, that is the whole core idea:

while not done:
    response = model(context)        # propose an action
    if response.tool_call:
        result = run_tool(            # execute it — sandboxed
            response.tool_call,
            guardrails=policy,        # block disallowed / risky actions
        )
        context = append(context, result)   # feed result back
        log(response, result)         # observe everything
    else:
        done = is_complete(response)  # stop condition

Everything serious is an elaboration of this loop. Build the loop first, get one real task working end to end, then add the subsystems below as you need them — not before.

The seven building blocks

1. Scaffolding

A clear system prompt, well-described tools, a strict output format, and a project instruction file (the CLAUDE.md / AGENTS.md pattern) so your conventions and forbidden actions ride into every run.

2. Tools

File and shell access (sandboxed), search, HTTP/API calls, and your domain actions (create invoice, book job, update CRM). Expose them through MCP so the same tools work across different agents instead of being re-implemented per harness.

3. Context management

Decide what goes in the window each turn and compact aggressively as the task grows. Most agent failures are context failures — the model never had what it needed, or drowned in what it did not.

4. Memory & self-improvement

Persist state across sessions, and — the high-leverage part — let the agent save what it learns as reusable skills. This is exactly what makes Hermes compound: it writes and refines its own skills from experience, so the same mistake is not made twice and the agent gets more useful over time.

5. Guardrails & approval gates

An allowlist of what the agent may do, hard stops on the rest, and a human-approval gate on anything risky, customer-facing, or that spends money. Most successful jailbreaks unfold over several turns, so test multi-turn, not just single prompts.

6. Observability & cost control

Log every action and tool call, trace decisions, and track token cost per task with budget caps that hard-stop on overspend. Paperclip's cost controls are the model here — in one run it paused an agent at $612 against a $600 cap before the next model call. Tools like LangFuse, AgentOps, Arize Phoenix, or Braintrust handle the tracing.

7. An evaluation harness

A set of golden tasks with machine-verifiable success criteria, plus red-team safety cases, run automatically on every change to your prompts, tools, or policies. Without this you cannot tell an improvement from a regression — and a silent regression in an agent that touches customers or money is expensive.

Build vs. buy — for most SMBs, mostly buy

You almost certainly should not write a harness from scratch. Configure a strong existing one and invest your effort in the parts that are specific to you — the scaffolding, the tools, and the guardrails:

  • — Software work → Claude Code or Codex + a good instruction file.
  • — Ops / inbox / CRM / calls → Lindy or a similar “AI employee” platform.
  • — A learning assistant you host → Hermes (self-hosted, self-improving, runs on a small VPS).
  • — Coordinating several agents → an orchestrator like Paperclip or Relevance AI.
  • — A genuinely unique workflow or strict compliance need → build on a framework (LangChain's DeepAgents, CrewAI).

Four rules learned the hard way

  • — Narrow beats do-everything. Operators who deploy one agent to answer calls, book jobs, quote, follow up, and update the CRM watch it break within a week. The agents that survive in production each do one job and chain together. Reliability goes up as scope goes down.
  • — Watch the cost meter. Token costs across multiple agents and providers add up fast and stay invisible until the bill arrives. Track cost per task from day one.
  • — Keep a human on judgment. Agents are strong at pattern-following and weak at novel, sensitive, or irreversible calls. “Zero-human” framing oversells; approval gates are not optional.
  • — Let the harness get simpler over time. As models improve, delete scaffolding you no longer need. Complexity you keep adding is a smell.

Sources & further reading: Martian Code Review Bench, awesome-harness-engineering, Hermes (Nous Research), Paperclip, Lindy, cmux.

Want a harness built around your business?

Gain AI designs AI workflows — and builds the software, websites, web apps, and mobile apps around them. From a first automation to a shipped product, one team.

Get in touchMore research →