From a security talk to a local AI lab

The idea started at a cybersecurity talk. Several founders were describing the same shift from different angles: the model matters, but the harness around it may matter just as much.

They were talking about smaller, purpose-selected models with unusual strengths—models that become far more useful when a purpose-built harness controls their instructions, tools, context, permissions, and operating boundaries. That framing stuck with me. It sounded less like “pick the smartest chatbot” and more like systems engineering.

A model is potential. A harness turns that potential into a controlled capability. The idea that started the project

The hardware decision

The next day, I pulled the trigger

The idea stayed with me after the talk. By the next day, curiosity had won: I ordered a MacBook Pro with an M5 Max, 128 GB of unified memory, and a 4 TB SSD. It was a bigger machine than I had ever bought, but I was already thinking less about the laptop and more about the lab I could build around it.

The storage gave me room to keep different model families and quantizations close at hand instead of constantly deciding what to delete. The unified memory was the more interesting part. On Apple silicon, the CPU and GPU work from the same memory pool. For local AI, that changes what one machine can realistically attempt.

That connection led me to MLX. Apple’s current MacBook Pro specifications list the M5 Max with up to 128 GB of unified memory, and the official MLX project is designed for the same shared-memory architecture. Its arrays can be used by the CPU or GPU without an explicit copy between separate memory pools. The more I learned, the more obvious the starting point became: run the models locally, observe what happens, and build the controls I wished existed.

Learning the ecosystem

First, I had to learn what a model actually is

I started where many people start: Hugging Face. I wanted to understand how people discover models, how weights and configuration files are stored, how revisions are shared, what model cards do, and how a model moves from a repository into a running application.

The first surprise was how many decisions appear before the first useful prompt: architecture, quantization, context size, chat template, tool-call behavior, licensing, disk size, memory pressure, and the difference between loading a model and merely having it downloaded. “Run an open model” is not one decision. It is a chain of them.

Codex became my pair programmer through that process. Together we built the project in layers: command-line generation first, then reproducible benchmarks, then a managed server, and finally the visual Control Center. Each layer appeared because the previous one exposed a new operational problem.

The MLX server

A stable local endpoint for changing models

The managed MLX server gives local applications one consistent endpoint even while I experiment with different weights. It wraps mlx-lm with stable aliases, explicit lifecycle controls, and runtime guardrails. The HTTP surface is compatible with the common chat-completions shape, so clients do not need to know where a model is stored on disk.

scripts/mlx-server load qwen3-coder-30b-8bit
scripts/mlx-server status

# OpenAI-compatible endpoint
curl http://127.0.0.1:48080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen3-coder-30b-8bit",
       "messages":[{"role":"user","content":"Explain this function."}]}'

Upstream mlx-lm describes its HTTP interface as similar to the OpenAI chat API. It also explicitly says the server is not recommended for production because it implements only basic security checks. That warning shaped my default: the lab binds to 127.0.0.1, and remote exposure is out of scope until proper authentication and hardening exist. See the official server documentation.

The OpenCode harness

Keep the coding harness stable while the backend changes

OpenCode is the harness that made the local server feel useful rather than experimental. It brings repository context, a tool loop, commands, and MCP connections to an OpenAI-compatible model endpoint. I can select a concrete local model when I want to test a particular set of weights.

The setup I use day to day is simpler. OpenCode points to one stable entry, local-mlx/active. That entry sends default_model to the API, and the managed MLX server resolves it to the model that was explicitly loaded and pinned at startup.

{
  "model": "local-mlx/active",
  "provider": {
    "local-mlx": {
      "npm": "@ai-sdk/openai-compatible",
      "options": {
        "baseURL": "http://127.0.0.1:48080/v1"
      },
      "models": {
        "active": {
          "id": "default_model",
          "name": "Active pinned MLX model"
        }
      }
    }
  }
}

That reverses the ownership in a useful way. OpenCode keeps the same model selection while the backend decides which model is resident. Switching weights happens through scripts/mlx-server load or the Control Center, where memory, cache, lifecycle, and safety controls already live. With pinning enabled, a client cannot cause a silent model switch by requesting a different concrete model.

The security testing toolkit

Turn a local open-weight model into a bounded security tester

Once OpenCode could use the pinned local model reliably, I could test the original idea that inspired the lab: a specialized harness can turn a general model into a controlled capability. I built appsec-audit-mcp with the local open-weight coding model working through OpenCode, then packaged it as a separate, installable Python toolkit instead of embedding security scanners in the model server or teaching the model a long list of improvised shell commands.

The FastMCP server exposes 54 namespaced tools for runtime source selection, static analysis, HTTP interaction, authentication and session inspection, proof-of-concept validation, evidence and coverage tracking, Markdown and HTML reporting, and read-only AWS posture checks. It can run over stdio, or as a macOS login service on the loopback-only Streamable HTTP endpoint http://127.0.0.1:48090/mcp. OpenCode connects to that service once as the global audit MCP.

{
  "mcp": {
    "audit": {
      "type": "remote",
      "url": "http://127.0.0.1:48090/mcp",
      "enabled": true
    }
  },
  "permission": {
    "skill": {
      "security-audit": "allow",
      "appsec-audit-mcp": "allow"
    },
    "audit_*": "allow"
  }
}

A small installer links the two project skills and the command into OpenCode's global configuration, so the toolkit is available while the local model is testing any authorized application workspace. The model can map an attack surface, form a concrete hypothesis, validate it with a controlled comparison, preserve evidence, and render a report without losing the audit trail between unrelated shell commands.

The Control Center

I needed to see the machine, not just the model

A terminal is excellent for automation and a poor place to maintain situational awareness. The Control Center makes the server’s actual state visible: the resident model, server process, system and process memory, swap, prompt cache, active requests, and the limits currently protecting the machine.

The architectural turning point

Memory management became a product feature

My early experiments made the problem obvious. Downloading many models is mostly a storage question. Keeping models resident is a system question. Large weights, long contexts, active generation, and retained KV caches all compete for the same unified memory used by macOS and every other application.

That is why the server now treats memory as a controlled resource instead of hoping the machine will sort it out:

1. Only one explicitly selected model stays resident. Switching is an operator action, not an accidental side effect of an API request.
2. Context and output are admitted before generation. The server checks token limits and an estimated memory budget before expensive work begins.
3. A memory reserve protects the rest of the Mac. New work can be rejected, retained cache can be released, and active work can be cancelled as memory falls.
4. Every important action remains visible and stoppable. The Control Center shows the policy, current budget, last safety event, and active request state.

The defaults visible in this build cap effective context at 128,000 tokens, output at 32,000 tokens, reserve 16 GiB for admission, and use an 8 GiB emergency-cancellation threshold. They are operating limits, not promises that every model should use the maximum.

The model harness

A fast loop for learning model behavior

The Model Harness gives me a controlled place to talk to the active model. I can change the system prompt and generation settings, keep multi-turn history in the browser tab, stream the answer, inspect exact token usage and output speed, and cancel a request through the same server control path.

Benchmarking

Fast is useful. Comparable is better.

Once several models were available, impressions were not enough. I needed repeatable prompt sets and saved evidence. The benchmark runner records model load time, prompt and generation tokens, tokens per second, wall time, finish reason, and peak memory. Each run is written as JSONL and can be rendered as an HTML report.

The next promotion gate is a task-quality scorecard: secure-code-review accuracy, patch correctness, false-positive rate, instruction adherence, structured tool-call validity, and hallucinated file or API references. Until that evidence exists, speed remains only one axis.

Telemetry

Observe the harness without recording everything

The telemetry view answers operational questions: Which client is using the server? Which model handled the request? How many input and output tokens moved? How much prompt cache was reused? What was the latency? Did the model emit structured tool calls, and can any of them be attributed to an MCP namespace?

I did not want observability to become silent conversation logging. The retained JSONL history is bounded. It stores counts, timing, client identity, offered tool names, structured tool-call names, and an optional short preview of the latest user query. It does not retain complete conversations, model responses, reasoning text, or tool arguments. Query previews can be disabled entirely.

Telemetry became a harness debugger

A long-running agent can appear productive while repeatedly calling tools, rebuilding an oversized prompt, or losing the prefix cache that made earlier steps fast. The telemetry service groups related requests by client, model, address, and bounded query preview, then scores the signals that matter together: sustained tool-call chains, repeated tool patterns, context growth, prompt-cache loss, and latency degradation.

This security-audit run crossed several thresholds at once. OpenCode had completed 29 consecutive tool steps, the context had grown to roughly 87,400 tokens, recent cache reuse had fallen to zero, and recent latency had climbed to more than three minutes. The Control Center raised a high-risk advisory with the evidence and a concrete operator action: review or stop the client, then compact the conversation or start a fresh agent session.

Where the project stands

The Control Center is now the beginning, not the finish line

The difficult part was not getting a model to produce text. It was operating models predictably: controlling residency, memory, context, clients, tools, and evidence. That is the system I am continuing to build.

I started by collecting models. I ended up building a way to operate them.