I’m constantly exploring available open harnesses and open-weight models, especially as major cloud AI providers tighten their acceptable use policies. No two harnesses behave identically. Each server interacts with MCP servers in slightly different ways. As complaints about MCP server inefficiency continue to grow, I decided to investigate the actual data and establish a straightforward testing methodology.

The idea

I want to verify whether claims of MCP inefficiency stem from systemic protocol flaws or simply naive, “vibe-coded” implementations. While the MCP JSON protocol isn’t particularly concise, I suspect the tokens spent on protocol overhead are minuscule compared to the inefficiencies introduced by poor server design choices.

Because no two harnesses handle MCP identically, symptoms of inefficiency likely manifest differently across platforms. Testing a cross-compatibility matrix is therefore essential. Rather than optimizing for the quirks of individual harnesses, most developers seem to test against a single preferred client and call it a day.

As an added benefit, this testing ground will serve as a stepping stone toward building a reliable, local-first environment.

The setup

Harnesses

New agent harnesses pop up every day. Rather than reinventing the wheel, I’m starting with five of the most established options:

  • OpenClaw
  • Hermes
  • OpenCode
  • Claude Code
  • Codex CLI

Each harness is wrapped in a customized Bubblewrap launcher and profile. This keeps them isolated from the host OS except for intended touch surfaces, without inhibiting their functionality.

LLM provider

Local instances of LiteLLM and Ollama are deployed via Docker. All harnesses are configured to route through this setup as their model provider, with Bubblewrap configured to allow HTTP API access to the LiteLLM instance.

Models used

To establish a baseline, I selected a cross-section of well-known, generally available models:

  • Anthropic / Claude Haiku 5.5
  • Anthropic / Claude Sonnet 5
  • Anthropic / Claude Opus 4.7
  • OpenAI / GPT-4o
  • OpenAI / GPT-5 nano
  • OpenAI / GPT-5.1
  • Alibaba Group / Qwen3 (8B)
  • Alibaba Group / Qwen3.8 (27B)
  • Liquid AI / LFM3 (1.7B)

The broader goal is to eventually evaluate a wider variety of tool-capable models.

Method of analysis

Every harness is tested using the same model, the same set of MCP servers, and the exact same prompt-defined task. The evaluation criteria are:

  • Task completion: Did the model successfully complete the task within the provided environment?
  • Context bloat: Is the MCP server/harness combination wasting context capacity with unnecessary input tokens?
  • Iteration overhead: Did the model require extra inference iterations due to ambiguous or confusing inputs?
    • Root cause: Which component is responsible: the harness or an unoptimized MCP server?

Runtime metrics from every session are exported for deeper analysis. These insights will directly inform how I design and optimize my own MCP servers moving forward.

Furthermore, wherever possible, all components are cloned from their Git repositories and built directly from source. This provides deeper insight into their internals and allows for inserting custom probing patches. This is drastically improving the quality and granularity of AI-assisted assessments.