Illustration: Setting context length for local LLMs's

Outsourcing work to specialized coding models via a proxy and task routing

A coding harness or agent framework is essentially an orchestrator: the final code quality and the associated bill are determined by the underlying language model that carries out the tasks. In this practical guide, you'll learn how to have orchestrators such as Claude Code or similar CLI assistants outsource heavy programming work to cost-efficient models via an intermediate OpenAI-compatible proxy, including a watertight audit trail and real-time insight into your usage per task.

Why decouple a harness from the executing model?

Many popular coding harnesses and assistants are shipped as a tightly coupled monolith. The system prompt, tool definitions, conversation history, context management, and API connections are all embedded in a single binary or closed package. If you want to use a different model or connect an external toolchain, you have to tinker with the source code of third-party software. As soon as the underlying software is updated, you risk losing your own customizations or having them break.

By functionally separating the interface and orchestration from the actual model call, a flexible, modular architecture emerges. In such a setup, the orchestration layer remains responsible for the interaction with the developer and the local context (such as searching a git repository or reading local files), while the computationally intensive programming work is forwarded via a flexible adapter to a model that is the most suitable and cost-effective for that specific task.

This decoupling offers three major advantages for indie developers and small teams:

Cost control: Heavy, repetitive tasks (such as generating unit tests, data migration scripts, or extensive boilerplate) don't always have to run through the most expensive frontier model if a specialized reasoning model can handle the work for a fraction of the price.
Avoiding vendor lock-in: When a new generation of models becomes available, you only adjust the routing in your intermediate layer. Your development environment, keyboard shortcuts, and scripts remain untouched.
Central registration and auditing: All interactions from different assistants pass through a single point, so you know exactly which task consumed how many tokens and what the costs were.

The premise: strictly separating orchestration and execution

In a robust development pipeline, each component fulfills a well-defined role. An orchestrator (such as an interactive agent in your terminal) understands the context of your codebase, creates an action plan, and determines which files need to be edited. The orchestrator then delegates the generation of complex logic, mathematical algorithms, or refactoring to a specified execution layer.

The architecture consists of three functional layers:

The orchestration layer: The CLI tool or plugin that translates your instructions into concrete subtasks.
The delegation layer (harness/wrapper): An automated script or pluggable runtime that packages the task description and presents it as a structured prompt.
The routing and proxy layer: A self-hosted, OpenAI-compatible proxy server (such as LiteLLM or a similar gateway) that captures the call, adds metadata, writes logs, and forwards it to the API endpoints of the chosen model provider.

Separating these responsibilities aligns seamlessly with the principles of robust API integrations, where error handling, retries, and logging are centralized instead of scattered across individual CLI tools.

The architecture of the call chain

The data path of a delegated programming task runs sequentially from the local workstation to the provider and back:

Ontwikkelaar / CLI Orchestrator (bv. Claude Code, Kimi Code)
  │
  ▼
Delegeer-script / Lokale wrapper (taakdefinitie & contextbundeling)
  │
  ▼
Uitvoerings-harness (headless worker)
  │
  ▼
Zelfgehoste OpenAI-compatibele proxy (bv. LiteLLM op http://localhost:PORT/v1)
  ├─ Registreert: Traceer-ID (cid), taaktype, timestamp
  ├─ Berekent: In- en out-tokens, geschatte kosten
  │
  ▼
Externe Provider API (bv. DeepSeek, Qwen of OpenAI-compatibel eindpunt)
  │
  ▼
Respons keert via de proxy terug naar de lokale werkomgeving

Because the proxy mimics a standard OpenAI interface, the calling harness does not notice that there is an inspection and logging layer in between. The configuration only requires setting an alternative base URL.

Step by step: setting up the configuration

Step 1: Setting up the proxy for routing and logging

The core of this setup is a local or self-hosted reverse proxy that translates and logs API calls. A well-known open-source example of this is LiteLLM, but you can use any OpenAI-compatible proxy or gateway for this. The proxy runs as a background process or within a container.

If you run the infrastructure locally or on a home server, check the instructions for running local models in Docker to neatly isolate containers and access them via an internal network.

A typical configuration for such a proxy links a logical model to a specific provider endpoint and stores the session data locally. Below is a conceptual example configuration (for example proxy-config.yaml):

# proxy-config.yaml - Voorbeeldconfiguratie voor een lokale LLM-proxy
model_list:
  - model_name: code-worker-heavy
    litellm_params:
      model: deepseek/deepseek-chat
      api_base: https://api.deepseek.com/v1
      api_key: "os.environ/DEEPSEEK_API_KEY"
  - model_name: code-worker-fast
    litellm_params:
      model: openai/gpt-4o-mini
      api_base: https://api.openai.com/v1
      api_key: "os.environ/OPENAI_API_KEY"

general_settings:
  master_key: "sk-jouw-lokale-proxy-sleutel"
  database_url: "sqlite:////pad/naar/proxy_tracking.db"
  logging_format: "jsonl"
  log_file_path: "/pad/naar/logs/api_access.jsonl"

Then start the proxy on a port of your choice (denoted in examples as PORT):

# Starten van de proxy (voorbeeld)
litellm --config /pad/naar/proxy-config.yaml --port PORT

Step 2: Connecting the execution harness

A harness is the framework that receives the coding task, analyzes any file trees, and constructs the actual prompt. Configure the harness so that it communicates with your local proxy instead of directly with the cloud provider.

In your harness's configuration file (often a YAML or JSON file in your home directory), set the base URL and the API key:

# settings.yaml van de harness
model_provider:
  name: "custom-proxy"
  baseURL: "http://localhost:PORT/v1"
  apiKey: "sk-jouw-lokale-proxy-sleutel"
  defaultModel: "code-worker-heavy"

Pay attention to the URL structure: Most SDKs and harnesses automatically add /chat/completions to the base URL. Therefore, specify only the base path up to and including /v1 (for example http://localhost:PORT/v1 or https://jouw-proxy.example/v1). If you accidentally end with /chat/completions, this will lead to 404 error messages.

Step 3: The delegation script and automated skills

To smoothly delegate work from an interactive CLI session, you create a small wrapper script. This script starts the harness in so-called 'headless' mode (without an interactive TUI), forwards the instruction, and logs the outcome.

#!/usr/bin/env bash
# delegate-task.sh — Delegeer een afgebakende taak naar het gespecialiseerde model
set -euo pipefail

TASK_DESCRIPTION="$1"
LOG_DIR="/pad/naar/project/logs"
TIMESTAMP=$(date +"%Y%m%d_%H%M%S")
CORRELATION_ID="task-$(openssl rand -hex 4)"

mkdir -p "$LOG_DIR"

echo "[INFO] Start taak ${CORRELATION_ID}: ${TASK_DESCRIPTION}"

# Aanroep van de headless harness met injectie van het correlatie-ID
HARNESS_TASK_ID="${CORRELATION_ID}" execute-harness --headless \
  --endpoint "http://localhost:PORT/v1" \
  --task "$TASK_DESCRIPTION" \
  > "${LOG_DIR}/${TIMESTAMP}_${CORRELATION_ID}.log" 2>&1

echo "[SUCCESS] Taak ${CORRELATION_ID} afgerond. Zie logboek voor details."

To ensure that orchestrators such as Claude Code consistently use this workflow, you define an instruction file (a so-called skill or prompt instruction). Place this file in the configuration folder of your orchestrator (for example .claude/skills/delegatie/SKILL.md or .agent/skills/delegatie.md):

# Richtlijn voor taakdelegatie

Wanneer je een taak tegenkomt die voldoet aan de volgende criteria:
1. Het betreft een opzichzelfstaande module, refactor of testsuite met duidelijke specificaties.
2. De taak vereist substantiële codegeneratie (meer dan 100 regels code).
3. Er zijn geen multimodale invoerbestanden (zoals afbeeldingen of UI-mockups) vereist die het doelmodel niet ondersteunt.

Voer de taak NIET zelf direct uit, maar delegeer deze via het script:
`./scripts/delegate-task.sh "<volledige instructie inclusief context en bestandspaden>"`

Controleer na uitvoering het resultaat via het gegenereerde logbestand en valideer de gewijzigde bestanden.

Cost control and measuring usage

The main argument for this modular setup is insight and savings. When developers blindly send all coding tasks to their orchestrator's default frontier model, costs quickly add up with large context windows. By routing heavy work to models optimized for coding and reasoning tasks, you maintain high throughput at manageable costs.

In the article on rate limits and cost management goes deeper into budget monitoring and concurrency limits for intensive API usage.

Illustrative calculation example: Costs per development task

To illustrate the financial impact, we compare a hypothetical refactoring task. Suppose a complex restructuring of a backend module requires 50,000 input tokens of context (files, types, previous tests) and generates 4,000 output tokens of new code.

Model category (illustrative)	Nature of the task	Context (tokens)	Generated (tokens)	Cost factor
General frontier model	Orchestration & planning	50,000 in	4,000 out	100% (reference)
Specialized reasoning model	Delegated code execution	50,000 in	4,000 out	~10% to 20%
Local open-weight model	Boilerplate & simple tests	50,000 in	4,000 out	Electricity/hardware only

In an active development week where dozens of such subtasks run, this routing ensures that the total API bill drops significantly, without sacrificing the quality of the software architecture.

Choosing the right model per programming task

Not every model excels at the same discipline. The advantage of a central proxy is that you can switch dynamically per task type. Consult the overview of specialized models for code to determine which models are currently leading for languages such as TypeScript, Python, Rust, or Go.

In addition, the guide on selecting the right model per task helps in drawing up routing rules based on reasoning power, context size, and latency. The benchmarks at model quality per task offer guidance for objectively comparing programming performance.

Type of programming task	Recommended model class	Why via the router?
Architecture design & planning phase	General high-reasoning model	Requires broad abstraction ability and strong instruction following.
Implementation of algorithms & refactoring	Specialized coding model (e.g., DeepSeek, Qwen)	High density of correct syntax at low token rates.
Writing unit tests & type definitions	Fast, smaller model or local model	Repetitive work with low complexity where speed matters.
Documentation & changelogs	Lightweight language model	Minimal reasoning power needed, focus on natural language use.

Pitfalls from practice

Setting up a multi-step chain brings a number of practical considerations:

Model name validation by clients: Some interactive CLI tools check in advance whether the specified model name appears in their hardcoded list of supported models. If you use an alias in your proxy (such as code-worker-heavy), the client may throw an error. Resolve this by having the proxy rewrite the model name, or by returning the known names via the /v1/modelsendpoint of the proxy.
Context management and token loss: When a task is delegated to a headless worker, all necessary context (such as type definitions and environment variables) must be explicitly included in the prompt or the accompanying files. After all, a standalone worker has no knowledge of the earlier chat history of your interactive session.
Error handling and timeouts: Complex reasoning tasks can take considerably longer before the first chunk of data arrives (time to first token). Make sure the timeouts in your reverse proxy and in your HTTP client are set generously enough (for example, at least 120 to 300 seconds for heavy coding tasks).
Image and multimodal data: Not all specialized coding models support image input (vision). Always route tasks with wireframes or screenshots to a multimodal model via a routing rule to prevent parsing errors.

Setting up logging, auditing, and governance

An important aspect of professional software development with AI is accountability. By routing all requests through a central proxy, you automatically build up a JSONL log file. This file serves as the central log of your automated development pipeline.

A representative line from such a log file looks like this:

{
  "timestamp": "2026-08-19T14:32:10Z",
  "correlation_id": "task-8f3a1b2c",
  "orchestrator": "claude-code",
  "routed_model": "deepseek-chat",
  "provider": "deepseek",
  "task_type": "unit-test-generation",
  "tokens": {
    "prompt_tokens": 14250,
    "completion_tokens": 1820,
    "total_tokens": 16070
  },
  "cost_estimate_usd": 0.0052,
  "status": "success",
  "latency_ms": 4210
}

Using simple command-line utilities such as jq , you can directly run analyses on your daily or weekly usage:

# Bereken het totale aantal tokens dat vandaag via de proxy is verwerkt
jq -s 'map(.tokens.total_tokens) | add' /pad/naar/logs/api_access.jsonl

# Toon alle taken die langer dan 10 seconden duurden
jq 'select(.latency_ms > 10000) | {id: .correlation_id, model: .routed_model, duur: .latency_ms}' /pad/naar/logs/api_access.jsonl

Who is this approach valuable for?

This modular architecture is particularly suitable for:

Indie developers: Who make intensive use of AI assistants and want to keep a grip on their monthly API spending without sacrificing the usability of modern CLI tools.
Small development studios: Who need to be able to demonstrate which code was generated using which model and who want to accurately charge costs per project or client.
Teams with strict privacy and compliance requirements: Who want to route certain sensitive repositories exclusively via local models or specific private endpoints, while generic tasks may go to public APIs.

By decoupling orchestration from the actual model execution and placing an intelligent proxy in between, you transform a collection of loose AI tools into a scalable, cost-conscious, and transparent development environment.