Running 235B Parameters on the Machines Under Your Desk

Here's the thing: when people picture running a large language model, they picture a data center. Racks of GPUs that belong to someone else, a metered API, and a bill that grows every month you succeed. You send your prompts off to a black box and hope the price, the model, and the privacy policy all stay the way they were when you signed up.

For a lot of teams, that's a bad trade. You give up control over when models change, where your data goes, and what hardware runs your workloads. And as usage grows, so does the bill, with no lever to pull except "pay more." It's a surrender wrapped in convenience.

Mesh LLM, published yesterday by the iroh team, is a different shape. It pools the GPUs and memory you already have — across as many machines as you want to add — and exposes the whole thing as one OpenAI-compatible API at http://localhost:9337/v1. Start one node. Add more later. Let the mesh decide whether a model runs on the box in front of you, routes to a peer, or splits across several machines.

I've been tracking the iroh project for a while. Their stack is built on an identity-addressed, NAT-traversing QUIC transport that just works between any two devices, and it's already running in production on hundreds of thousands of devices. Mesh LLM is the first time they've applied that networking layer to the problem that's been quietly eating the industry: AI is expensive, and it's somebody else's.

The problem with monolithic models

The popular models are monoliths. Most people reach them through a UI or an API key and pay a large provider to run everything. That model is convenient, and it's also a slow erosion of agency. You do not control when the model gets updated, what memory it runs in, or what hardware sits underneath. If the provider deprecates a version, your fine-tuned pipeline breaks. If their pricing shifts, your margins shift with it.

Plenty of businesses that depend on these models want the opposite. More control. More pluggability. Lower cost. They have GPUs sitting in offices, in closets, under desks. What they're missing is a way to make those machines act like one. That's the gap Mesh LLM steps into.

The pitch is straightforward. Run bigger models without buying bigger GPUs. Share compute privately with your team, or publicly with the world, to power agents and chat. Point any OpenAI client at localhost:9337/v1 and stop caring where the work actually happens.

Under the hood, it distributes model compute across a mesh of iroh endpoints. A request can be served three ways: run it locally on this machine's GPU, route it to a peer that already has the model loaded, or split a model too big for any single box across several machines as a pipeline.

How splitting works — and why it matters

For the giants — the 70B, 180B, even 235B parameter mixture-of-experts models — Mesh LLM has a split mode they call "Skippy." A model gets partitioned by layer ranges into stages. Layers 0 to 15 on one node, 16 to 31 on the next, and so on down the pipeline. Activations flow from one stage to the next, so several modest machines can run a model none of them could hold alone.

The OpenAI client never sees any of this. It still just talks to localhost. The abstraction is thick enough that you don't need to know what hardware is where, yet thin enough that you get actual control if you want it. That's the kind of layering that earns trust from engineers who've been burned by opaque platforms.

The catalog ships with over 40 models, from half-a-billion-parameter models that fit on a laptop to those 235B Mixture-of-Experts giants. The plugins are pluggable, each one declaring what it provides in a manifest. The runtime starts them, routes calls, and exposes capabilities over MCP, HTTP, inference, and mesh events. It's not a monolithic binary; it's a composition layer that lets you add or swap models without touching infrastructure.

The networking that makes it possible

Every node, whether it serves models or only sends requests, boots an iroh endpoint. That endpoint is the node's identity — a public key — and its only network surface. There is no central server. iroh handles the hole-punching, NAT traversal, and relay fallback needed to open a direct, authenticated QUIC connection between any two nodes, wherever they sit.

To keep that working across the open internet, Mesh LLM runs two iroh relays in different regions. Nodes that can't reach each other directly always have a fallback path nearby. The whole protocol rides on QUIC's ALPN negotiation, with three distinct ALPNs: mesh-llm/1 for the main mesh (gossip, routing, HTTP tunnels, plugin channels), mesh-llm-control/1 for the owner control plane (config sync, ownership attestation), and skippy-stage/2 for latency-sensitive activation transport between split model stages.

Inside the main connection, everything is a bidirectional QUIC stream tagged with a single leading byte that says what kind of stream it is. One connection carries gossip, inference, route queries, and peer-lifecycle events, all demuxed by that first byte. A byte of 0x01 means gossip. 0x04 is an inference request tunneled to a peer. 0x05 asks "which models do you host?" 0x06 signals a dead peer. 0x07 is a graceful shutdown. 0x08 opens a plugin RPC channel. And 0x0e shares direct addresses for NAT traversal.

The neat part is what this buys you. iroh gives authenticated, NAT-traversing QUIC between any two machines, addressed by public key. So "route to a peer" and "stream activations to the next pipeline stage" become the same primitive as "talk to localhost," just with a different endpoint ID. The networking stops being something you have to think about.

Why this is a different shape altogether

Look, I've spent years intercepting API traffic, reverse-engineering undocumented protocols, and building automation that stitches together services that were never meant to talk to each other. What strikes me about Mesh LLM is how deliberately it strips out the middlemen. It's not a platform you rent. It's a tool you run.

The lightweight software is about 18 MB. You install it, and you either join the public mesh or configure a private deployment. It presents itself as a vanilla OpenAI-compatible endpoint. Your existing tooling — your chat clients, your agent frameworks, your monitoring — doesn't need to change. The shape of your workflow stays the same. The power dynamics shift.

A mobile app is coming, built on iroh's Swift SDK. The plan is to speak ACP, the emerging agent standard, so other clients can join the mesh too. The throughline is the same one that motivated the whole project: more peer-to-peer, fewer closed servers, and no lock-in.

It's worth pausing on "no lock-in." That phrase gets thrown around a lot, but here it's structural. Because every node is identified by a public key, and because the protocol is ALPN-negotiated on top of QUIC, there's nothing proprietary about the communication. If you wanted to, you could write your own node that participates in the mesh, speaking the same gossip and stream formats. The openness isn't just a license file; it's wired into the design.

The honest tensions

I'm not going to pretend this is a universal panacea. Running distributed inference across heterogeneous hardware, across flaky home or office connections, with split models that stream activations over the internet — that introduces latency and reliability challenges that a hyperscaler's data center doesn't face. The "Skippy" pipeline partitioning is clever, but if one node in the chain crashes, the inference fails. There's no mention yet of checkpointing or redundant stages, and those will matter at scale.

Privacy is another double-edged sword. The mesh can be private, but the default public mesh means your prompts and responses are traveling to peers you may not know. The system gives you control, but it also requires you to exercise it. For regulated industries, that's both a feature and a new set of decisions to make.

And then there's the question of whether this reshapes the economics or just relocates them. Sure, you're using GPUs you already own. But those GPUs were bought for other purposes, and now they're doing inference. Power, cooling, maintenance — someone's paying for those. The true cost savings depend on utilization patterns.

None of these tensions are fatal. They're just the kind of trade-offs that come with taking infrastructure into your own hands. For teams that have been frustrated by rate limits, model deprecations, and opaque pricing, they're vastly preferable to the alternative.

What I think happens next

The way I see it, Mesh LLM sits at the intersection of two trends that have been building for years. The first is the slow, stubborn return of peer-to-peer architectures — not as an ideological stance, but as a practical response to centralization's brittleness and cost. The second is the realization that inference, unlike training, is embarrassingly parallel and doesn't need monolithic supercomputers. You can split a model. You can route to a warmer cache. You can make do with less.

This won't kill the hyperscalers. But it might make them less essential for the long tail of AI workloads that don't need sub-50-millisecond latency and five-nines availability. It might let a startup run a 70B model on repurposed workstations in an office closet. It might let a research lab share compute across continents without an AWS bill that makes the grant officer wince.

For engineers, it's a reminder that infrastructure is a choice. You can pay someone else to handle it, and pay the price on their terms. Or you can wire together what you already have, accept a bit of complexity, and keep the keys yourself.

Mesh LLM is 18 MB of software and a bet that more and more teams will choose the second option.