SwarmLLM

Run AI on your own computer. Run bigger AI together. SwarmLLM is a free app that gives you a ChatGPT-style assistant running on your own computer. When a model is too big for one computer, computers running SwarmLLM team up over the internet and run it together, each doing part of the work. No account, no subscription, no ads, no crypto.

This site is the full user guide and reference. For source code, releases, and issues, head to enapt/SwarmLLM.

What you can do with it

• Chat with AI on your own computer — start SwarmLLM and it opens in your browser at localhost:8800, shows which models fit your computer, and downloads the one you pick.
• Use models too big for your computer — computers in the swarm each hold some parts of a model and work through your question together. It is fastest when they are close to each other; across continents it works, but slowly. Today the swarm runs medium-sized models; giant ones are the goal. No computer ever needs to download the whole model.
• Link your own devices — put your laptop, desktop and home server into one private group and use them together.
• Plug it into the apps you already use — an OpenAI-compatible API, the Anthropic Messages API (it works as a Claude Code backend), an MCP server with seven tools, and optional routing to 12 cloud providers with your own keys.
• Privacy, honestly — connections between computers are encrypted, but in the public swarm the computers doing the work see what they process, so don't send anything sensitive. On your own computer, or with Private Mode, your requests stay on your devices. Details.

Speed depends on your hardware and the model; current measurements and the settings that affect them are on the Performance page.

Where to go next

Status

An early (alpha) version that improves every week and updates itself. Splitting a model across computers works, but is only fast when those computers are close together. Every change runs through thousands of automated tests before it ships. Report issues.

Platform support

All downloads are on the Releases page.

Getting Started

SwarmLLM runs AI models on your own computer and, when a model is too big for one computer, lets computers team up over the internet to run it together. It is free and open source — no account, no subscription, no crypto.

Is it private? It depends on where the model runs.

• On your own computer — yes. When a model runs entirely on your computer, your conversation doesn't leave it.
• In the public swarm — only partly. Connections between computers are encrypted, so nobody watching the internet can read them. But the computers doing the work for you see what they are working on, and a determined person running one of them could reconstruct your question. Don't send anything sensitive through the public swarm.
• Private Mode keeps your requests on your own linked devices — and, unless you switch that off, other SwarmLLM computers on your local network. Your computer still helps others with what it can spare. Set it up under More → My Devices.

Start and finish on this computer (the button above the chat box) keeps the first and last steps on your computer, so no other computer is handed your words as text. It switches itself on for any model where you hold the first and last parts. It is an extra layer, not a guarantee: the helpers still compute on numbers that published attacks turn back into most of your text. It also makes replies slower (a few seconds extra on a short reply) and needs those two parts on your disk. Details: Security & Encryption.

This guide walks you through installation, downloading your first model, and chatting.

Prerequisites

• Windows (64-bit), a Mac with an Apple chip (M1 or newer), or 64-bit Linux
• At least 4 GB of RAM (8+ GB recommended)
• At least 2 GB of free disk space (more for larger models)
• An internet connection (for downloading models and connecting to other computers)

What leaving it running means. While SwarmLLM is open, your computer shares what it can spare with the swarm. It doesn't notice when you're busy or gaming — set a Resource Schedule in the app to hold back at the times you choose, or close the SwarmLLM window. It also uses internet data every day just staying in touch with the swarm, and more while it shares model parts, so it is not a good fit for a capped connection. It doesn't start with your computer.

Chapters

• Installation — Download and run SwarmLLM on your computer
• First Model — Download and chat with your first AI model
• Joining the Network — How your computer finds others, and how to keep your requests on your own devices

Quick Commands

On Windows, type swarmllm.exe instead of ./swarmllm.

./swarmllm run                  # Start SwarmLLM (default port 8800)
./swarmllm run -p 9000          # Start on a different port
./swarmllm run -v               # Start with more detailed logging
./swarmllm status               # Check that it is running and what it is doing
./swarmllm chat                 # Chat in the terminal
./swarmllm bench                # Measure speed
./swarmllm peers                # List connected computers
./swarmllm diagnostics          # A report for a bug report — safe to post publicly
./swarmllm version              # Show version number

Installation

Download

Download the right file for your system from the GitHub Releases page:

It stops immediately with Illegal instruction? Download the -baseline build. The ordinary Linux and Windows builds are made for processors from 2013 onwards (they use AVX2 instructions), which makes them roughly 3x faster at running models on the processor. On an older machine they stop on the very first instruction, before printing anything — and because nothing is wrong with the download, re-downloading or checking permissions changes nothing. On Linux, grep -c avx2 /proc/cpuinfo printing 0 means you want the baseline build.

Not sure which Mac? Apple menu > "About This Mac." If it says "Apple M1" (or M2/M3/etc.), use the Mac download. If it says "Intel," there is no download for your Mac yet.

Which NVIDIA cards get GPU acceleration? RTX 30-series and newer (also RTX 40, RTX 50, and the A/H data-centre cards). The RTX 20-series, GTX 16-series and anything older are below the requirement of the FlashAttention kernels SwarmLLM ships, which is what makes attention fast.

An older card is not a problem — nothing breaks and there is nothing to configure. SwarmLLM checks the card when it starts, tells you in the log and on the dashboard that it is using the processor instead, and carries on. The models the swarm manages for you run through this CUDA path on every system. The Windows graphics build also carries a Vulkan backend that works on any graphics card, but it is used only for a single model file you load yourself with -m.

To check your card: nvidia-smi --query-gpu=name,compute_cap --format=csv. A number of 8.0 or higher gets GPU acceleration.

Check the download (optional, one command)

The single-file downloads — swarmllm-linux-x86_64, swarmllm-windows-x86_64-gpu.exe and the like, plus the .deb and .rpm — each have a .sha256 file beside them on the release page. (The .zip and .tar.gz archives don't.) Put both files in the same folder and run one command to catch a damaged or incomplete download:

sha256sum -c swarmllm-linux-x86_64.sha256            # Linux
shasum -a 256 -c swarmllm-macos-aarch64.sha256       # Mac

Those .sha256 files are also signed, and the built-in updater refuses an update whose signature it cannot verify.

Install & Run

Windows

1. Download swarmllm-windows-x86_64-gpu.zip if your PC has an NVIDIA graphics card (it bundles the NVIDIA libraries, so no CUDA Toolkit is needed), otherwise swarmllm-windows-x86_64-cpu.zip (works on every PC, including ones with AMD or Intel graphics).
2. Right-click it and choose Extract All — running it from inside the zip view does not work.
3. Double-click swarmllm.exe in the extracted folder. SmartScreen shows "Windows protected your PC" because the program is not code-signed yet: click More info > Run anyway.
4. A console window opens: that is SwarmLLM running. Keep it open (minimising is fine) — closing it stops SwarmLLM. The dashboard opens in your browser.

There is no SwarmLLM-Setup.exe at the moment. The installer that bundled both variants was dropped from the release build on 2026-04-22 and has not been published since; earlier versions of this page still named it.

From PowerShell on a raw binary:

cd Downloads\swarmllm-windows-x86_64-gpu
.\swarmllm.exe run

macOS

Double-click the download in Finder to unpack it into a folder, keep that folder somewhere in your home folder (Documents or Downloads is fine), and double-click swarmllm inside it. A Terminal window opens — that is SwarmLLM running, so leave it open. Or from Terminal (the archive has no folder of its own, so unpack it into one):

mkdir -p ~/swarmllm
tar xzf ~/Downloads/swarmllm-macos-aarch64.tar.gz -C ~/swarmllm
cd ~/swarmllm
./swarmllm run

Where you put it matters on a Mac. Keep SwarmLLM in a folder you own — anywhere under your home folder, as in the commands above. Folders like /Applications need administrator rights, and SwarmLLM cannot then replace its own binary, so it will tell you an update is available and decline to install it. Nothing else about it changes; move the file and updates work by themselves.

First launch: "cannot be opened because it is from an unidentified developer." The binaries are not yet signed with an Apple developer certificate, so Gatekeeper stops the first run of anything downloaded in a browser. Either open System Settings > Privacy & Security and click Open Anyway, or remove the quarantine flag the browser added:

xattr -d com.apple.quarantine swarmllm

Downloading with curl avoids it entirely — the quarantine flag is set by the browser, not by macOS in general — and it is a one-time thing per download, not something that recurs on every update (confirmed on a Mac mini M4, 2026-09-03).

Linux

The archive has no folder of its own, so unpack it into one (use swarmllm-linux-x86_64-cuda.tar.gz for an NVIDIA RTX 30-series card or newer):

mkdir -p ~/swarmllm
tar xzf ~/Downloads/swarmllm-linux-x86_64.tar.gz -C ~/swarmllm
cd ~/swarmllm
./swarmllm run

The NVIDIA download needs NVIDIA's CUDA 12 libraries (cuBLAS, cuRAND and NVRTC); unlike the Windows download it does not carry them. On Ubuntu or Debian (multiverse / non-free):

sudo apt install libcublas12 libcublaslt12 libcurand10 libnvrtc12

or install NVIDIA's CUDA Toolkit 12. Without them SwarmLLM still works: it runs on the processor and its log says which library it could not find. Restart it after installing them to use the card.

Linux packages (.deb / .rpm)

Debian / Ubuntu:

sudo dpkg -i swarmllm_*_amd64.deb
sudo systemctl enable --now swarmllm                   # start it as a background service

Fedora / RHEL — the .rpm does not yet create the user the service runs as, so create it once before starting the service (systemd creates the data folder):

sudo rpm -i swarmllm_*.x86_64.rpm
sudo useradd --system --no-create-home --shell /usr/sbin/nologin swarmllm
sudo systemctl enable --now swarmllm

Either way SwarmLLM runs as a background service that keeps its data in /var/lib/swarmllm (the access token is in /var/lib/swarmllm/api_key). Programs installed from a package do not update themselves — install the next release's package to update. Homebrew and AUR packages are not published yet.

Docker

The fastest way to get running on any Linux server:

# 1. Get the compose file and example env
curl -LO https://raw.githubusercontent.com/enapt/SwarmLLM/main/docker-compose.yml
curl -LO https://raw.githubusercontent.com/enapt/SwarmLLM/main/.env.example

# 2. Configure (add API keys, change ports, etc.)
cp .env.example .env
nano .env

# 3. Start
docker compose up -d

For NVIDIA GPU support (requires NVIDIA Container Toolkit):

docker compose --profile gpu up -d

Pre-built images on GHCR:

The CPU image carries two builds and starts the faster one your processor can run: the AVX2 build on anything from 2013 on, a slower build that runs anywhere otherwise. Images before the first release after 0.3.204-alpha carried only the slow build, so a container ran models about 3x slower than the downloadable program on the same machine, and read prompts about 8x slower. The GPU image needs an AVX2 processor, like the GPU downloads.

An image is published when its release is — after the release has been checked and signed, the same moment the built-in updater can see it. (Before this rule, images for 0.3.199-alpha, which was withdrawn, and for 0.3.202-alpha and 0.3.203-alpha, which were never released, were published by mistake; they have been removed. If a container of yours still runs one, pull latest again.)

Data is persisted in Docker volumes. Model shards are stored in the swarmllm-models volume (or bind-mount a host directory via SWARMLLM_MODELS_DIR in .env).

View logs with docker compose logs -f. The API key is printed on first startup; to read it again — the dashboard will ask for it, because a browser on your computer reaches the container from outside it — run docker exec swarmllm cat /data/api_key.

Cargo Install

Requires Rust 1.90+:

cargo install --git https://github.com/enapt/SwarmLLM.git   # add --tag vX.Y.Z-alpha to pin a release
swarmllm run

Building from Source

git clone https://github.com/enapt/SwarmLLM.git
cd SwarmLLM
cargo build --release
./target/release/swarmllm run

For NVIDIA GPU support, build what the release builds (llama.cpp + candle on CUDA + FlashAttention; it needs the CUDA toolkit and compiles for a long time):

cargo build --release --features cuda

The build command for each release variant is in CONTRIBUTING.md.

For Apple Silicon: the default build runs on CPU. A Metal-accelerated build is on the roadmap but not yet implemented (no metal Cargo feature exists yet); until then, use the default cargo build --release.

Open the Dashboard

Once running, open http://localhost:8800 in your browser. The setup wizard will walk you through initial configuration.

Your First Model

You need at least one AI model before you can chat.

Easiest path: use what the swarm already runs

When SwarmLLM starts and connects to other computers, the Chat tab shows models you can use without downloading anything:

• "Available right now on the swarm" — click one and a fresh chat opens. Start typing.
• "The swarm is gathering these" — the swarm has some of the parts; they'll be ready once the rest arrive.
• "Popular models the swarm could adopt" — popular models nobody is running yet. Clicking one opens the model search so you can pick a version.

When the Chat tab says "No models available yet", its Get shared test model button downloads a small model everyone can use, so you can start chatting straight away.

Download a model yourself

Recommended: a shared test model. In Settings → Testing & Diagnostics, click Get my share (this computer downloads only the parts it should hold) or Get all of it (so this computer can answer on its own). From a terminal:

./swarmllm get-model                   # list the test models
./swarmllm get-model standard          # Llama 3.2 3B — "Get my share"
./swarmllm get-model standard --all    # "Get all of it"

Any other model: open the Models tab and choose Search HuggingFace (or click Find models on the Dashboard), search for a model — try TinyLlama for a small, fast one — and check its badge: ✓ Runs locally, ↗ Host parts, ○ Swarm computers only or ⚠ Too large for your swarm. The Download button starts the download. A model marked Swarm computers only is too big for this computer to hold even a part of, but other computers hold all of it — its button is Chat, which opens a conversation with it straight away.

Heads-up: models from well-known publishers (meta-llama, mistralai, Qwen, bartowski, unsloth, …) spread across the swarm faster than models from unknown publishers. If you pick an unpopular model, fewer other computers will pick up its parts.

Download via CLI

# Every part, so this computer can answer on its own
curl -X POST http://localhost:8800/api/admin/hf/download-shards \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"repo_id": "Qwen/Qwen2.5-Coder-7B-Instruct-GGUF", "filename": "qwen2.5-coder-7b-instruct-q4_k_m.gguf", "all_shards": true}'

# Or one part as this computer's share; other computers may pick up the rest
curl -X POST http://localhost:8800/api/admin/hf/download-shards \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"repo_id": "Qwen/Qwen2.5-Coder-7B-Instruct-GGUF", "filename": "qwen2.5-coder-7b-instruct-q4_k_m.gguf", "peer_fair_share": true}'

# Or download specific parts (shards) by number:
curl -X POST http://localhost:8800/api/admin/hf/download-shards \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"repo_id": "Qwen/Qwen2.5-Coder-7B-Instruct-GGUF", "filename": "qwen2.5-coder-7b-instruct-q4_k_m.gguf", "shards": [0, 1, 2]}'

Recommended Models by Hardware

On-Demand Loading

You do not need to pre-load models into VRAM. When you send a request for a model whose parts are on disk but not loaded, SwarmLLM automatically loads the model on the fly. If VRAM is full, the least-recently-used model is evicted to make room. The first request to a cold model may take a few extra seconds while loading completes.

Start Chatting

Web UI:

1. Click the Chat tab
2. Select your model from the dropdown
3. Type a message and press Enter

CLI:

./swarmllm chat
# Or with a specific model:
./swarmllm chat --model qwen2.5-coder-7b-instruct-q4-k-m

API:

curl http://localhost:8800/v1/chat/completions \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen2.5-coder-7b-instruct-q4-k-m",
    "messages": [{"role": "user", "content": "Hello!"}],
    "stream": true
  }'

What Are Model Parts?

Large AI models are split into smaller parts (called shards in settings and logs, ~512 MB each) so they can be spread across the network. Each part holds some of the model's layers. SwarmLLM handles this automatically — you just pick a model and download.

A computer never needs every part of a model. When a model is split across computers, each one loads only the layers it's responsible for.

Joining the Network

You don't have to do anything to join: a new SwarmLLM connects to the public swarm by itself through a built-in starting computer, then finds more computers from there.

Automatic Discovery

SwarmLLM finds other computers automatically:

• Public swarm: on first start SwarmLLM contacts a built-in starting computer and learns about others from it.
• Same network (LAN): computers on the same Wi-Fi or home network find each other within seconds.
• Returning users: computers you've connected to before are remembered and reconnected on start-up.
• Peer exchange: connected computers share their lists of other computers with you.

Connect to a specific computer

To connect straight to a friend's computer:

1. On the Dashboard, find the Computers panel and click Connect a computer.
2. Under Your Swarm Address, click Copy and send the swarm://… address to your friend.
3. Your friend opens the same panel, pastes it under Another User's Swarm Address, and clicks Connect.

The address is scrambled so your IP address isn't readable at a glance, but anyone who has the whole address can decode it — share it only with people you'd give your IP address to.

This connects you to the SwarmLLM network. To link your own devices into a private group, use More → My Devices instead (see Private Networks).

Manual Bootstrap

./swarmllm run --bootstrap "/ip4/203.0.113.50/udp/8800/quic-v1/p2p/12D3KooW..."

Or in your config file:

[network]
bootstrap_peers = ["/ip4/203.0.113.50/udp/8800/quic-v1/p2p/12D3KooW..."]

Private Networks

gossip_network_id puts a group of computers on their own announcement channels, with their own key, so they discover each other's models separately from the public network:

[network]
gossip_network_id = "my-private-network"

On its own this is not an isolation boundary, and it is important not to treat it as one. It separates announcements. It does not stop your computer sending work to a computer outside the group: which machines hold which model parts is also learned through the shared peer-to-peer directory, which every SwarmLLM computer takes part in regardless of this setting. Measured on a test group whose very first request was answered by a public computer.

To actually keep work inside a set of machines, use a device group and turn on Private Mode:

[pool]
private_mode_allow_lan = false   # default is true, and "LAN" includes any
                                 # other SwarmLLM computer on the same network

Then link the computers into a group: on the first one, open More → My Devices, click Link My Devices, then Create, and click Add Another Device for an invite code. On each other computer, open More → My Devices, click Join Existing Group, paste the code and click Link This Device. Then switch on Private Mode there. That is an explicit list of computers, checked before any work is handed out, and it is the only thing that decides where your prompts go. Use both together for a private cluster: gossip_network_id to keep the announcements separate, Private Mode to keep the work in.

private_mode_allow_lan applies without a restart: edit config.toml and reload it (POST /api/admin/config/reload).

Firewall & internet reachability

SwarmLLM needs TCP port 8810 (P2P primary transport) and optionally UDP port 8800 (QUIC) open. On the same LAN, computers find each other by themselves — no ports to open. To be reachable across the internet, SwarmLLM tries these by itself — usually there is nothing to do:

• UPnP (on by default) — opens the port on a cooperative home router automatically. A router gives a port to one computer only, so if you run two SwarmLLM computers behind the same router, start the second on a different port (swarmllm run -p 8801) or only the first can be reached directly. The log says so when this happens (UPnP: the router did not open port …).
• A relay (on by default) — if your computer can't be reached directly, even behind CGNAT, it is reached through a publicly-reachable SwarmLLM computer.
• Manual port-forward (TCP 8810 + UDP 8800 to your machine) plus external_address in config, if you want a direct connection the router won't open by itself.

If a My Devices invite code says "only works on your local network," your computer isn't internet-reachable yet. See docs/NETWORKING.md for the full guide — including the CGNAT check and how to run your own anchor node.

Configuration

SwarmLLM works out of the box with sensible defaults. This section covers customization.

Config Priority

Settings are read from four sources, in order of priority:

1. Command-line flags (highest) — e.g., --port 9000
2. Environment variables — seven, by name: SWARMLLM_NODE_DATA_DIR, SWARMLLM_NODE_LISTEN_PORT, SWARMLLM_LOGGING_LEVEL, SWARMLLM_API_KEY, SWARMLLM_NETWORK_BOOTSTRAP_PEERS, SWARMLLM_INFERENCE_MODEL_PATH and SWARMLLM_INFERENCE_GPU_LAYERS. There is no general SWARMLLM_<SECTION>_<KEY> rule — any other setting goes in config.toml.
3. Config file — config.toml in your data directory
4. Built-in defaults (lowest)

Provider API keys have an additional source: a .env file in the data directory or current working directory. Standard env var names are used (OPENAI_API_KEY, ANTHROPIC_API_KEY, DEEPSEEK_API_KEY, etc.). The .env file does not override existing environment variables or keys already set via the dashboard.

Config File Location

Specify a custom path: --config /path/to/config.toml

Minimal Example

[node]
listen_port = 8800
contribution = "moderate"

[resources]
max_disk_mb = 50000

[identity]
region = "US"

[inference]
gpu_layers = -1   # auto: the whole model on the graphics card when it fits

[auto_manage]
enabled = true

Chapters

• Config File Reference — Every option explained
• Shard-Only Mode — Distributed inference with partial models
• CLI Flags & Environment Variables — Command-line and env var reference

Config File Reference

Every configuration option, organized by section.

When a change takes effect

Every setting the dashboard offers applies to the running node immediately (so does the same setting edited in this file, followed by POST /api/admin/config/reload). You do not need to restart for a resource limit, a contribution level, an auto-manage threshold, or a batching setting to matter.

A setting that exists only in this file applies without a restart where its row below says so; otherwise restart the node after changing it — several are read once, when something is built at startup (a listener, a rate limiter, the network).

Two of the dashboard's settings are also decided once, when the node starts, and keep their startup value until it starts again:

Before v0.3.88 none of the dashboard's settings were live: a setting would save, report success, show its new value, and leave the running node on the value it booted with.

[node] — Basic Node Settings

[resources] — Resource Limits

[resources.schedule] — Usage Schedule

[network] — Networking

[inference] — AI Model Inference

[inference] — batching

[inference] — prefix cache

[inference] — speculative decoding

[inference] — SWARM-SPEC prefetch

[inference] — activation transfer

[logging] — Log Output

[ui] — Web Interface

[api] — API Authentication

The dashboard fetches its own access key on page load. These two options decide which networks that happens on — loopback always does, and anywhere else the page asks you to paste the key once instead of failing silently.

dashboard_trust_overlay only takes effect when this node itself holds an overlay address. The IPv4 range Tailscale uses (100.64.0.0/10) is shared carrier-grade NAT space that some ISPs also hand out, so a peer's address alone does not prove a tailnet.

dashboard_trust_lan exists for the case where you reach a node through a Tailscale subnet router or a container publish. Those rewrite the source address by default, so the request arrives from the router's private address and is indistinguishable from any other LAN client. It can be toggled from Settings → Identity & Access and applies immediately, without restarting the node.

On a network this node trusts, anything that can reach the API port can obtain the access key, and with it admin and inference. That is the intended bargain for a tailnet — devices you authorised — which is why the LAN case stays opt-in. See Tailscale / WAN.

[model] — Model Storage

[auto_manage] — Automatic Shard Management

Contributing capacity vs. contributing redundancy

default_model_shard_cap = 0 (the default) lets auto-manage take every shard of a model it decides to host. With min_replicas = 2 and a small number of participants, the usual outcome is that every node ends up holding every model in full — measured on a two-node setup as ~20 GB and ~18 GB of near-identical content.

That is the right default: it maximises availability, and a model nobody holds completely cannot be served at all.

It is not always what an operator wants. Holding a full copy of a model your neighbour already holds adds a spare, not throughput. If you are joining to add capacity — more machines able to serve different parts of more models — set a cap:

[auto_manage]
default_model_shard_cap = 2   # at most 2 shards of any one model

Rough guidance:

A cap only limits what auto-manage acquires on its own. Shards you download explicitly, and shards pinned to this node by a pool owner, are unaffected.

One caveat worth knowing before you tune this: holding fewer shards spreads storage, not necessarily load. A node that holds a model in full still serves requests for it locally rather than sharing them out, so capping shards is about what gets stored where, not about how work is distributed.

[pool] — Device Pool

[pool.credit_rates] — Credit Rates

Credits are recorded but dormant — they gate nothing: no balance affects who is served or how fast, and there is no token or payment. These rates only change numbers in the internal ledger.

[updates] — Auto-Update

swarmllm run --no-update-check turns update checking off for that run, whatever mode says, and nothing — a settings save included — switches it back on; start the node without the flag to restore it. (Until v0.3.205 the flag changed nothing: it set the legacy auto_update, which no longer decides anything.)

[identity] — Your Identity

[providers] — Cloud Providers

Set from the dashboard (Settings → Cloud Providers) or PUT /api/admin/providers, which write this section for you. Keys can also come from the environment or a .env file in the data directory, under the standard names (OPENAI_API_KEY, ANTHROPIC_API_KEY, …).

[providers.claude_subscription] — Claude Subscription (feature-gated)

Requires --features claude-subscription at build time. Managed via the dashboard or PUT /api/admin/providers.

Shard-Only Mode

SwarmLLM supports shard-only operation — a node only needs individual shard files (~512 MB each) plus a small GGUF header (~6 MB), not the full model file.

How It Works

A model directory in shard-only mode:

~/.local/share/swarmllm/models/qwen2.5-coder-7b/
├── manifest.json        # Model metadata + shard layout
├── gguf_header.bin      # First ~6MB of GGUF (metadata + tensor index)
├── shard_000.bin        # 512MB shard
├── shard_001.bin
├── shard_002.bin
└── ...

SwarmLLM automatically extracts gguf_header.bin from shard_000.bin when first needed. The ShardReader constructs a virtual GGUF from header + shard files, so the model parser works exactly as if the full GGUF were present.

Why This Matters

• A 7B model is ~4.5 GB as a full GGUF, but a single shard is only ~512 MB
• Nodes only load the layers they're assigned — no wasted disk or VRAM
• You can help serve a model bigger than your graphics card by hosting just a few of its shards

Manual Shard Assignment (--shards)

For multi-node split inference, assign each node a subset of shards:

./swarmllm run --shards "0-3"    # This node handles shards 0, 1, 2, 3

The range is saved and restored every time the node starts — even if you later start it without --shards. To go back to holding every shard, start once with --shards all.

Behavior when --shards is set:

• The node only advertises the specified shard indices
• Auto-manage prioritizes downloading missing shards in the range (100x scoring bonus)
• Smart pruning never removes shards in the configured range

Multi-Node Example

Run a 7B model across two machines:

# Machine A (shards 0-3, layers 0-13):
./swarmllm run --shards "0-3" --bootstrap "/ip4/MACHINE_B_IP/udp/8800/quic-v1/p2p/PEER_ID"

# Machine B (shards 4-7, layers 14-27):
./swarmllm run --shards "4-7" --bootstrap "/ip4/MACHINE_A_IP/udp/8800/quic-v1/p2p/PEER_ID"

Both nodes discover each other, assemble a distributed pipeline, and forward hidden-state activations between them. The pipeline is assembled automatically by the InferenceRouter.

Without --shards

If you have never used --shards (or cleared it with --shards all), the node auto-detects and advertises all local shards. This is the normal mode for most users — --shards is only needed when you want explicit control over which layers a node handles.

CLI Flags & Environment Variables

CLI Flags

Subcommands

chat Options

bench Options

pool Subcommands

Link your own devices so they serve each other privately. With the group's Private Mode on, your requests stay on these devices (and, by default, other SwarmLLM computers on your local network).

Example flow:

# Main device:
swarmllm pool create --name "My Devices"
swarmllm pool invite-code   # → swarmpool://… (a long code)

# On each other device:
swarmllm pool join 'swarmpool://…'

Note: This links YOUR own devices. It's different from connecting to the SwarmLLM network (which uses swarm:// peer addresses).

Environment Variables

A small, fixed set of options — the ones a headless or Docker deployment needs before a config file exists — can be set with a SWARMLLM_ environment variable. This table is the complete list of settings that can be given this way — there is no general SWARMLLM_<SECTION>_<KEY> rule, and any other setting goes in config.toml or the dashboard. A few diagnostic switches are also read (for example SWARMLLM_HYBRID_OFFLOAD, see gpu_layers in the configuration reference), and SWARMLLM_PORT is the port swarmllm update checks for a running node (default 8800).

Example:

SWARMLLM_NODE_LISTEN_PORT=9000 ./swarmllm run -v

Provider API Keys via Environment

Cloud provider API keys use standard environment variable names:

These can also be placed in a .env file in your data directory:

# ~/.local/share/swarmllm/.env
OPENAI_API_KEY=sk-proj-...
DEEPSEEK_API_KEY=sk-...
NVIDIA_NIM_API_KEY=nvapi-...

The .env file is loaded at startup. It does not override existing environment variables or keys already configured via the dashboard/database. The dashboard settings UI shows "From .env" for keys loaded this way.

Troubleshooting

Start here: why was that request slow, or where did it fail?

SwarmLLM logs to the window it runs in and to logs/swarmllm.log in the data directory — ~/.local/share/swarmllm/ on Linux, ~/Library/Application Support/swarmllm/ on macOS, %APPDATA%\swarmllm\ on Windows (it starts over at 50 MB and keeps three older ones, swarmllm.log.1 the newest). Docker: docker compose logs as well; .deb package: journalctl -u swarmllm as well. The examples below call that file node.log.

Every completed request writes one summary line. Read it before anything else — it usually identifies the problem on its own:

grep "DIAG: request complete" node.log | tail -5

DIAG: request complete request_id=1ddd2912-… route=distributed segments=2
  nodes=0718d8b9,96842635 regions=TH,TH queue_ms=3 sched_ms=1 ttft_ms=180
  decode_ms=1420 total_ms=1604 tokens=48 tok_per_sec=33.8
  seg0_ms=520 seg1_ms=900 outcome=ok

A missing field means "not measured", never zero. ttft_ms and decode_ms are absent on requests that did not stream, because there is no honest way to split decode out of the total there.

No log file to hand?

swarmllm diagnostics

This is the single most useful thing to attach to a bug report, and it is safe to post in public: no API key, no invite code, no file paths, and every network address — yours and your peers' — replaced by a placeholder naming only its kind. Add --full if you are debugging your own machine and need the addresses back. It includes whether your machine is reachable from the internet, the last 50 requests with their routes, per-peer serving performance (ping, ms per layer, latency, region — slowest first), what your node has served for others, and recent failures with which machine served each one. That last detail is what separates "my node has a problem" from "one peer has a problem". A peer marked "(not that peer's doing)" only carried the request: the failure was on your own computer.

Diagnosing from the client side

You do not need server access. Every response carries its route:

curl -i -X POST localhost:8800/v1/chat/completions \
  -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
  -d '{"model":"…","messages":[{"role":"user","content":"hi"}]}' \
  | grep -i '^x-swarm-\|^server-timing'

x-swarm-route: distributed
x-swarm-peers: 1
x-swarm-nodes: 0718d8b9,96842635
server-timing: queue;dur=3, sched;dur=1, ttft;dur=180, decode;dur=1420

On a streaming response Server-Timing carries only what is known before the body starts — queue and scheduling. Token-level figures arrive at the end of the stream, because a header cannot be revised once sent.

Can't Connect to Peers

Check the bootstrap address format:

/ip4/203.0.113.50/udp/8800/quic-v1/p2p/12D3KooW...

Firewall: SwarmLLM needs TCP port 8810 (P2P) and optionally UDP port 8800 (QUIC) open.

• Linux: sudo ufw allow 8810/tcp && sudo ufw allow 8800/udp
• Windows: Windows Defender Firewall > Inbound Rules > New > Port > TCP 8810 + UDP 8800
• macOS: System Settings > Network > Firewall > allow SwarmLLM

Same LAN? Use local IP (e.g., 192.168.1.x). LAN peers should be found automatically via mDNS.

Model Download Stuck

1. Check disk space — a 7B model needs ~4-5 GB free
2. Verify internet access to https://huggingface.co
3. Cancel and retry from the Dashboard
4. Start with -v for verbose logs: ./swarmllm run -v
5. Try a smaller model first (TinyLlama, ~700 MB)

GPU Not Detected

1. Verify GPU works: nvidia-smi
2. Install NVIDIA drivers if needed
3. Check you downloaded a GPU build (-gpu on Windows, -cuda on Linux)
4. On Linux the card must be an NVIDIA RTX 30-series or newer; older cards use the processor automatically and say so in the log
5. Macs run on the processor only for now

WSL2 users: The CUDA driver comes from your Windows NVIDIA driver. Check that /usr/lib/wsl/lib/libcuda.so.1 exists and add to your ~/.bashrc:

export LD_LIBRARY_PATH=/usr/local/cuda/lib64:/usr/lib/wsl/lib:$LD_LIBRARY_PATH

A Model Runs on the Processor Although There Is a Graphics Card

The model's card on the Dashboard says why ("Running on the processor — …"), and so does swarmllm status. The reasons, and what to do:

A model too big for the card whole runs partly on it ("N of M layers on the graphics card") rather than wholly on the processor; that is a split, not one of the reasons above.

Port Already in Use

./swarmllm run --port 9000    # Use a different port
lsof -i :8800                 # Find what's using 8800
./swarmllm status             # Check if another instance is running

Dashboard opens but nothing saves

Symptoms: the page loads from another device, but the setup wizard's "Start SwarmLLM" button appears to do nothing, settings won't save, and panels sit empty. The hardware panel may say "CPU only" on a machine that has a GPU.

Every admin call is returning 401 because the page was never handed an access key. The daemon only hands it out automatically over networks it trusts: loopback always, a Tailscale-style overlay when this node is on one too, and a private/LAN address only if you opted in.

A banner at the top of the page states this, and — importantly — names the address the daemon actually saw for you. Behind a NAT, a container publish, or a Tailscale subnet router that is not the address in your browser's address bar, because those rewrite the source address by default.

Two ways through:

• Paste your access key into the banner. It's in the api_key file in the data directory (read it from inside the container, if that's where SwarmLLM runs), and is remembered per node so this is a one-time step per browser.
• Turn on Allow access from my local network in Settings → Identity & Access. This applies immediately without restarting the node, which matters when the node you can't reach is the one you'd have to restart.

Inference and the OpenAI/Anthropic APIs are unaffected — they accept the key as a Bearer token from any address. Full detail in Tailscale / WAN.

A worker keeps working after the client gave up

Symptoms: a client timed out or was closed, the log says the request was cancelled, and top still shows a swarmllm model-worker process at full CPU minutes later — on a processor-only machine, sometimes long enough to trip a thermal warning.

Since v0.3.152 a cancel reaches every place a request can wait, so this should stop within one prompt chunk (about 128 tokens' worth of work, seconds on a processor). To see what the node is computing and stop a worker without kill -9:

./swarmllm status                     # "Workers:" lists each model worker, its pid,
                                      # card or processor, and requests in flight
./swarmllm unload <model-id>          # retire that worker; the files stay, it
                                      # loads again on the next request

If a worker stays busy longer than a chunk after the client has gone, the cancel did not reach it — docs/DIAGNOSTICS.md § "The client left — did the work stop?" lists the log lines to check, in order.

Slow First Request

If the first inference request to a model takes noticeably longer than subsequent ones, this is expected. SwarmLLM uses on-demand model loading — models whose shards are on disk but not loaded into VRAM are loaded when first requested. If VRAM is full, an LRU eviction occurs first. Subsequent requests to the same model will be fast.

Slow Inference

1. GPU vs CPU: CPU is 5-20x slower. Check Dashboard for GPU status.
2. Model too large: Use Q4 quantization, match model size to VRAM.
3. Far-apart helpers: a model split across distant computers is slow, because each word of the reply passes through every one of them. swarmllm diagnostics lists the slowest computers first.

Database Corrupted

# Stop SwarmLLM first, then back up just the database
cp ~/.local/share/swarmllm/db.redb ~/swarmllm-db-backup.redb
rm ~/.local/share/swarmllm/db.redb
./swarmllm run

Models and config.toml are kept, but your Access Token changes (apps using the old one need the new one) and this device leaves its My Devices group.

GPU Out of Memory

If a model is bigger than your graphics card, SwarmLLM puts as many of its layers on the card as fit and runs the rest on the processor. If loading on the card still runs out of memory, it retries on the processor and logs:

GPU OOM — retrying model load on CPU

The processor is 5-20x slower, and a model that doesn't fit in the memory SwarmLLM may use is refused rather than swapped. To avoid OOM:

• Use smaller quantizations (Q4 instead of Q8)
• Use a model that fits in VRAM (check model size vs available VRAM in the dashboard)
• For models too large for one GPU, use distributed inference across multiple nodes

GPU Memory Stays Full When Idle

Seeing high VRAM use with little activity is usually expected, not a leak. Your node keeps models loaded in GPU memory so it can serve the swarm without a cold start. How much it commits is set by your contribution level ([node] contribution — minimal / moderate / maximum, also on the dashboard).

The daemon reclaims VRAM in two ways:

• Demand-driven (before pressure): a model with no local requests for [auto_manage] idle_unload_secs (default 5 min) and low network demand is unloaded from GPU memory automatically. Its shards stay on disk, so it reloads (one cold start) on the next request — your holder status never changes. Set idle_unload_secs = 0 to keep every loaded model resident. Deliberately-held models are never idle-unloaded — reference/test models (swarmllm get-model), pinned or locked models, and encrypted-pipeline models stay resident.
• Pressure-driven (automatic): above 70% VRAM the daemon narrows loaded models to fewer shards; above 95% it fully unloads a model. Both keep shards on disk.

To free VRAM immediately, restart the daemon, or lower contribution. There is no leak here — the model-worker subprocess holding a model is killed (freeing all its GPU memory) whenever the daemon unloads it by either path above.

Distributed Inference Issues

Peers visible but inference fails:

1. Ensure both nodes have the required shards loaded (check Dashboard > Models)
2. Verify P2P TCP connectivity: port <base_port> + 10 must be reachable
3. Run with -vv and filter: ./swarmllm run -vv 2>&1 | grep "DIAG:"
4. Check for DIAG: segment TIMED OUT — indicates network or compute bottleneck

High latency per token:

• Splitting a model adds one network round trip per computer for every word of the reply — fast on a local network, slow across continents
• Tensor parallelism is off by default (inference.tensor_parallel) and only helps on a fast local network

Pipeline assembly fails:

• The scheduler needs enough shard coverage to build a complete pipeline
• Check DIAG: assemble_pipeline_for for candidate counts

"Not enough memory in the swarm for …" or "… are online but turned it down":

• Not enough memory in the swarm — the computers that hold this model cannot set aside enough memory between them to run it (the message says how many of its layers they have room for). Trying again will not help; pick a smaller model, or wait until a computer with more memory holds this one.
• Online but turned it down — the computers holding that part of the model are connected but busy, usually running other models; the message quotes the last one's own reason. Try again in a few minutes.
• Versions up to v0.3.224 reported both cases as "No reachable node holds layers X-Y … the peer that held that piece has gone", which was usually not true — look in the log for need about … MB more or Not enough free memory beside the request to tell.

Inference fails with "peer never acknowledged" or "silent drop":

• A SendDirectMessage was issued but neither a Response nor an OutboundFailure event arrived from libp2p within 10s (RR_ACK_TIMEOUT_SECS). Treated as a transient failure: the router automatically retries once with a fresh pipeline assembly that filters out the unreachable peer. If retry also fails, the user sees the error within ~20s (vs the first-token deadline: 120 s plus 0.5 s per prompt token, at most 600 s — plus 240 s for each computer that may first have to load the model, since 0.3.230).
• A peer that DISCONNECTS while you wait for its first token is noticed within 5 s (0.3.230+): the log says the peer disconnected before its first token and the request is retried elsewhere.
• Most common cause: the target peer was killed or partitioned and the local libp2p connection state hasn't yet caught up.
• Look for DIAG: rr ACK timeout — closing streaming caller in the logs to confirm the fast-fail path engaged.

Concurrent requests stall when only some get dispatched:

• This node runs at most half of inference.max_concurrent_requests of its own requests at once (5 with the default 10 — every request gets the same tier while credits are dormant); the rest queue until earlier ones finish. Raise the setting to allow more. Work arriving from OTHER computers has its own cap set by your contribution level (8 / 24 / 64 at minimal / moderate / maximum), and no single computer may take more than half of that.

Cross-Node Prefix-KV Sharing

Off by default: a computer only shares its prefix cache if its owner sets inference.share_prefix_cache_with_peers = true, because announcing it reveals hashes of the prompts it has cached. Expected logs on a successful first hit of a peer's cached prefix:

B: DIAG: cross-node prefix HIT — hydrated KV matched_tokens=N total_tokens=M
A: DIAG: served PrefixKvFetch ... hit=true

I never see cross-node prefix HIT:

• Only fires on iter 1 of a prompt whose prefix your local node hasn't prefilled yet. Iter 2/3 hit the local cache (populated by iter 1).
• Check the peer even announced the prefix: look for DIAG: PrefixCacheAnnounce indexed node_id=... blocks=N in your log. No announce → peer's gossip never reached you (check grep 'Published message to GossipSub' | grep 'swarm/models').
• Check the peer passes the trust gate: default cross_node_prefix_trust_min = 0.5 equals DEFAULT_TRUST, so a freshly-seen peer should just barely pass. Any misbehavior drops it below.

I see prefix-probe: fetch timed out:

• The peer didn't return a snapshot inside the worker-probe window (3000 ms by default). On a large model (7B+) with cold CPU this can happen if the snapshot is >100 MB. The path degrades to local prefill — no worse than not having the feature. The current 3000/2500/2000 ms chained timeouts are sized for 7B-class snapshots; the older 500/400/500 ms values were TinyLlama-sized and forced a fallback to local prefill on larger models.

I see rejected KV snapshot — penalizing peer trust:

• The returned snapshot failed BLAKE3 reverification or contained NaN/Inf. Three rejection reasons:
  • hash_chain_mismatch → prefix_cache_block_tokens differs between nodes (default 64, common alternatives 32/128)
  • non_finite_tensors → GPU overflow on the serving side
  • deserialize_failed → wire corruption — open an issue

Stop sharing your own prefix cache: Leave inference.share_prefix_cache_with_peers = false (the default).

Running the Test Suite

Building SwarmLLM and running its tests is covered in CONTRIBUTING.md. See Benchmarking for reproducing the performance benchmarks and Performance for which knobs turn each speedup on/off.

A model the swarm never picks up

Computers only download parts of a model on their own once it has earned some trust, so a model from an unknown source can't make everyone download gigabytes. The trust levels you may see in the app are "Manually approved by you", "Has received real inference requests" and "Widely hosted across the network".

• Download any part of it yourself (see First Model) — that marks it as approved by you on this computer.
• Popular models on HuggingFace are approved automatically once they are at least a day old and have enough downloads: 10,000 for well-known publishers (meta-llama, mistralai, Qwen, google, microsoft, deepseek-ai, bartowski, TheBloke, unsloth and others), 100,000 for everyone else.
• A model also earns trust when people send it requests, or when many computers host it.

Chat dropdown shows "No models available yet"

This is the cold-start state. Click Get shared test model in the Chat tab to download a small model everyone can use and start chatting straight away. Once your computer hears from others, the Chat tab also shows three rows:

• "Available right now on the swarm" — models the swarm can run for you today. Click one to select it and open a fresh chat.
• "The swarm is gathering these" — the swarm has some of the parts; they will be ready once the rest finish downloading.
• "Popular models the swarm could adopt" — popular models nobody is running yet. Clicking one opens the model search.

If none of these appear, your computer hasn't heard from any others yet. Check the Computers panel on the Dashboard, or run swarmllm peers, to see whether it found other computers. A computer with hf_watcher_enabled = false won't see the "could adopt" row by design.

Still Stuck?

• Run ./swarmllm diagnostics and paste its output into your report — it is safe to post publicly
• For more detail in the log: ./swarmllm run -vv 2>&1 | grep "DIAG:"
• See the Diagnostics Guide for detailed log instrumentation
• Check GitHub Issues
• Open a new issue with your OS, hardware, ./swarmllm version, and the swarmllm diagnostics output

How SwarmLLM Compares

SwarmLLM sits beside two kinds of project: apps that run AI on ONE machine (Ollama, LM Studio, Jan), and networks that split a model across MANY (Petals, exo, Bittensor). This page compares it with the second kind, where the differences are less obvious.

⚠ Splitting a model is only fast when the computers are close. Every network on this page passes your request through each computer for every word of the reply, so the round trip between them is paid once per word. Two computers on the same local network are fast; two on different continents are slow. SwarmLLM's plan for automatically forming teams of nearby computers is in the regional pipelines plan.

Details on the others change quickly — check each project's own page before relying on a row.

System Overview

SwarmLLM is a single Rust binary that simultaneously functions as:

1. A P2P network node — connects to peers over TCP (Noise+Yamux) and QUIC/UDP using libp2p
2. An HTTP API server — serves OpenAI + Anthropic-compatible endpoints, MCP server, and cloud provider proxy via Axum
3. A web dashboard — embedded frontend (component-based vanilla HTML/CSS/JS, 11 HTML templates in one page, no build step)

All three share a single port (default 8800) and a common Arc<SharedState>.

┌──────────────────────────────────────────────────────────┐
│                      swarmllm binary                      │
│                                                          │
│  ┌──────────┐  ┌──────────────┐  ┌──────────────┐       │
│  │  P2P     │  │  HTTP API    │  │  Admin UI    │       │
│  │  Node    │  │  Server      │  │  (embedded)  │       │
│  │(TCP+QUIC)│  │  (Axum)      │  │              │       │
│  └────┬─────┘  └──────┬───────┘  └──────┬───────┘       │
│       │               │                 │                │
│  ┌────┴───────────────┴─────────────────┴─────────────┐  │
│  │              Shared State (Arc)                     │  │
│  │  DashMap<NodeId, PeerInfo>      — peer registry     │  │
│  │  ModelRegistry                  — models + shards   │  │
│  │  state.events (EventBus)        — activity + dashboard│ │
│  │  state.credits (CreditPool)     — balance + pool     │  │
│  │  state.models (ModelMgmt)       — acquisition + trust │  │
│  │  state.metrics (MetricsProviders)— stats + providers │  │
│  └────────────────────────────────────────────────────┘  │
└──────────────────────────────────────────────────────────┘

Key Design Decisions

• Config priority: CLI flags > seven named SWARMLLM_* environment variables > config.toml > defaults
• Data directory: ~/.local/share/swarmllm/ (Linux), ~/Library/Application Support/swarmllm/ (macOS), %APPDATA%\swarmllm\ (Windows)
• Port layout: HTTP API on TCP:port, P2P TCP on port+10 (Noise+Yamux), P2P QUIC on UDP:port
• Shard-only: Nodes never need a full GGUF. Shards are downloaded individually.
• No blockchain: Credit system uses dual-signed transactions, not a token or chain

Technology Stack

Daemon & Subsystems

The daemon spawns 12 Tokio tasks wired together with mpsc channels:

                           ┌──────────────┐
                           │  daemon/     │
                           │  (bootstrap) │
                           └──────┬───────┘
                                  │ spawns tokio tasks
  ┌───────┬───────┬───────┬───────┼───────┬──────────┬──────────┬──────────┬──────────┬──────────┬─────┐
  ▼       ▼       ▼       ▼       ▼       ▼          ▼          ▼          ▼          ▼          ▼     ▼
Network  Infer   Credit  Health   API    Rebal-   Acquisi-   Message    Pool     AutoShrd   HfWat- Update
Manager  Router  Ledger  Monitor  Server ancer    tion Mgr   Dispatch   Manager  Manager   cher   Checker

Subsystem Responsibilities

Channel Layout

Broadcast Channels

Note: Former separate channels (prune_events_tx, models_changed_tx, lan_discovery_tx, system_notify_tx, peer_list_changed_tx, update_tx) were consolidated into these two in the event system unification.

Startup Sequence

1. Parse CLI args (clap)
2. Initialize tracing subscriber
3. Load/create config (TOML + env + defaults + CLI overrides)
4. Ensure data directory exists
5. Load/generate Ed25519 identity
6. Open redb database
7. Build Daemon { config, identity, db }
8. Initialize ModelExecutor (load GGUF if --model provided)
9. Build Arc<SharedState> (includes ModelRegistry from DB)
10. Scan local shards, register in registries
11. Create mpsc channels
12. Spawn all 12 tasks
13. Open browser if configured
14. tokio::select! on Ctrl+C or task exit
15. Graceful shutdown: save peer cache, flush database

Graceful Shutdown

Shutdown is triggered by Ctrl+C (SIGINT/SIGTERM) or any task exiting:

• A watch channel signals all subsystems
• Peer cache is saved to redb
• Database is flushed
• Open connections are drained

Networking & Discovery

Transport Stack

libp2p Swarm
├── Kademlia (DHT) — distributed hash table for peer/shard/model lookup
├── GossipSub — pub/sub for shard/health/credits/identity/pools/regions
├── request_response — unified protocol (/swarmllm/1.0.0, 600s timeout)
├── mDNS — optional LAN peer discovery
├── connection_limits — `network.max_connections_per_peer` (default 3; below 2 disables hole punching), `max_peers` total (150 / 300 / 500 by contribution level)
├── Identify — protocol identification
├── AutoNAT (v2 client + server) — NAT detection
├── DCUtR — hole punching
├── UPnP — optional port mapping on the home router (`network.enable_upnp`)
├── stream — `/swarmllm/pipeline/1.0.0`, one long-lived stream per split request (`persistent_pipeline_stream`, off by default)
├── relay::client — circuit relay (reaching others through a relay)
└── relay (server) — relaying connections for peers behind firewalls (`network.enable_relay`, its limits)

Protocol Format

The unified protocol uses a type-tag byte on every frame (src/network/protocol/mod.rs):

Receivers auto-dispatch on the leading byte; senders choose based on config + request kind. Only the 0x00 frame carries a JSON body; the rest use binary framing with length prefixes.

Discovery Stack

SwarmLLM uses 5 independent discovery layers:

1. mDNS — Discovers LAN peers in seconds. Config: enable_mdns = true
2. Persistent Peer Cache — Saves up to 200 peers every 5 min + on shutdown. Fastest reconnect.
3. Invite Codes — two formats, see below.
4. Peer Exchange (PEX) — On each connection, exchanges up to 20 known peers.
5. Kademlia DHT — Bootstrap flag + periodic re-bootstrap every 5 minutes (and quick retries at 10/30/60/120 s while the node has no peers).

Invite code formats

Two codes exist for two different jobs. Both are ChaCha20-Poly1305 sealed with a randomly generated key embedded in the blob itself — that is deliberately not confidentiality against someone holding the code, only a guard against casual harvesting of node addresses from screenshots and chat logs.

swarm://... — network invite. base64url(key ‖ nonce ‖ encrypted_multiaddr). Carries one reachable address for the node that minted it. Used to bring a machine onto the swarm.

swarmpool://... — pool invite (v2, R140). Wraps the 8-character pool code with everything a fresh node needs to find the inviter before any shared discovery exists. Inner payload, JSON-serialised then sealed then base64url'd:

{
  "version": 2,
  "pool_id": "...",
  "pool_name": "...",
  "multiaddrs": ["/ip4/…/tcp/8810/p2p/12D3KooW…"],
  "code": "A3F7K2M9",
  "expires_at_unix": 1750000000
}

Roughly 300–500 characters — long, but it fits a copy-paste, which the 8-char code could not do for this purpose. multiaddrs is the node's live listen_multiaddrs snapshot: bound sockets unioned with confirmed external addresses (UPnP-mapped, AutoNAT-confirmed, relay-circuit, or manually declared via network.external_addresses), each suffixed with /p2p/<peer_id> so the dialer can verify identity. Without that union a NAT'd node silently minted a LAN-only code that worked on the LAN and died over the internet.

The legacy bare 8-character code (A3F7K2M9) still works and still means "broadcast a join request over the existing swarm". It only ever worked when both nodes were already on the same swarm — which is exactly the situation an invite code is least needed for. pool::invite::looks_like_v2 is the prefix sniff that routes between the two paths; decode_invite_code normalises every decode failure to Validation, because the overwhelmingly likely cause is a truncated paste rather than a daemon bug.

Generation fails with ServiceUnavailable when listen_multiaddrs is empty (the daemon hasn't bound yet). When it has entries but none pass the stricter any_internet_reachable check — public IP, DNS name, or relay circuit; LAN and CGNAT ranges excluded — generation still succeeds but emits an invite_lan_only warning, so nobody is handed a code that cannot survive the trip it was made for.

GossipSub Topics

Six topics, all subscribed at startup in discovery::subscribe_topics:

The topic match in NetworkManager::handle_broadcast is contract-not-default: a SwarmMessage variant with no topic arm falls through _ => return and silently drops at the wire. Adding a new gossip variant requires updating the match — an early multi-node test caught PrefixCacheAnnounce missing from the TOPIC_MODELS arm, which had silently dropped every cross-node prefix-cache announce at the network layer until a two-daemon run flushed it out.

Messages older than 5 minutes are rejected (replay protection).

Cross-Node Prefix KV Sharing Dispatch

The cross-node prefix-cache fetch path uses the request_response protocol, not gossip. The gossip layer only broadcasts which blocks each peer holds (PrefixCacheAnnounce on swarm/models); the actual snapshot transfer is a direct bilateral exchange:

1. Requesting daemon sends SwarmRequest::PrefixKvFetch to the peer chosen by the probe resolver (trust-gated by cross_node_prefix_trust_min, default 0.5)
2. Serving daemon runs fetch_local_snapshot against its own worker over IPC (2000 ms timeout) and gets the serialized bytes or None
3. Serving daemon returns SwarmResponse::PrefixKvData { present, payload } with the bytes wrapped in the WIRE_TAG_PREFIX_KV frame on the binary payload slot (not in the JSON header — serde_json inflates Vec<u8> ~5× and blows past the 64 MiB IPC cap)
4. Requesting daemon BLAKE3-reverifies + NaN/Inf-scans, hands bytes to its worker to hydrate a KvCacheEntry

See Inference > Prefix-Cache KV Sharing for the full pipeline and measured numbers.

Anti-Gaming

• Subnet clustering (>5 nodes per /24) is tracked; the elevated 25% check rate applies only to credit transactions, which are dormant, so today it has no effect. The SubnetClustering trust penalty is defined but never applied
• Signed balance reports with timestamp freshness (5 min window)
• Gossip replay rejection (5 min window)
• cross_node_prefix_trust_min gates fetch peers at a minimum trust score (default 0.5, equal to DEFAULT_TRUST; set to 2.0 to disable cross-node fetch entirely)

Inference Pipeline

Subprocess-Per-Model Isolation

Each loaded model runs in its own swarmllm model-worker subprocess (Ollama-style). When a model is unloaded, the subprocess is killed and the OS + CUDA driver immediately reclaim all GPU memory — no daemon restart required.

Main daemon                          model-worker subprocess (one per model)
───────────────────────────────      ───────────────────────────────────────
ModelProcessPool.generate()  ─────►  loads shards from disk on first request
ModelProcessPool.forward()   ─────►  runs forward passes / full decode loop
                             ◄─────  streams WorkerMsg::Token / LayerResult
unload_model()               ─────►  kill process → OS frees all VRAM

IPC: a local socket (a Unix domain socket on Linux and macOS, a named pipe on Windows) with binary framing — [4B json_len][json header][4B payload_len][raw tensor bytes]. JSON carries message metadata; the payload carries raw activation bytes to avoid base64 overhead.

Message types (src/inference/worker_ipc.rs):

SplitModelEntry is metadata-only — it caches eos_tokens, vocab, chat_template, bos_token, and eos_token_str from the GGUF header without loading model weights. The weights live exclusively in the worker subprocess.

Worker granularity: one process per ModelId (not per shard). A single worker handles all layer ranges for a model and owns its own KvCacheStore. Individual shard unload uses DaemonMsg::Unload; the process exits only when all shards are released.

Split Inference Engine

The split inference engine (src/inference/split/) enables distributed inference using candle for direct tensor computation with quantized GGUF weights. Each node loads only its assigned transformer layers (in the worker subprocess), forwarding hidden-state activations between nodes. The module is split into: model.rs (SplitModel struct + accessors), loader/ (GGUF/shard load), executor.rs (forward pass + tensor-parallel), hybrid.rs (how many of a segment's layers go on the card), kv_cache.rs, kv_budget.rs (card-aware KV and RoPE sizing), kv_write.rs (the card's f16 cache), token_embedding.rs, draft.rs (the in-engine drafter for speculation across computers), entry.rs, gguf_meta.rs, shard_reader.rs, rope.rs, prefix_cache.rs.

Client → API Server → InferenceRouter → Pipeline Assembly
                                              │
                      ┌───────────────────────┘
                      ▼
          ┌──────────────────────┐
          │   Pipeline Segment   │     Token IDs (prefill)
          │ Node A: Layers 0-15  │──── LayerForward ──►
          └──────────────────────┘                      │
                                        ┌───────────────┘
                                        ▼
                            ┌──────────────────────┐
                            │   Pipeline Segment   │
                            │ Node B: Layers 16-27 │── sample token ──►
                            └──────────────────────┘

Pipeline Assembly

1. Fetch model manifest to determine layer ranges
2. Pipeline affinity check: if multi-turn session has a previous pipeline and all nodes are still connected, reuse it (KV cache locality)
3. Query model_registry.shard_holders for hosting nodes
4. Liveness filter: drop holders that aren't in connected_node_ids (the libp2p truth — DHT can re-inject providers for peers that just disconnected, and peer_registry is intentionally preserved across mid-pipeline disconnects for reconnect attempts)
5. Fetch node load/latency from peer_registry
6. Parallax scheduler: shortest-path dynamic programming over observed per-layer latencies (EMA over recent forwards), rather than a greedy latency-only sort. Cross-gossips top-32 observed latencies via NodeCapability.observed_latencies so every node has a current view of the network's compute profile. Since 0.3.230 a candidate that must first load the model is also charged for the load: each node times its own loads and advertises NodeCapability.model_load_ms_per_gib, and the layers a candidate does not hold are priced at that rate (10 s/GB for a node that has not reported one) — this node included
7. Start-and-finish check (encrypted_pipeline): if enabled for this model, force first and last segments to the local node (boomerang topology)
8. Assignment: widest contiguous layer range per node, merging on same-node
9. Identify standby nodes per segment (failover)
10. Send PipelineAssignment, wait for ACKs, begin forwarding

Failure Handling

The router applies a single retry on transient remote failures (silent rr drops, OutboundFailure, remote-generate timeouts). The retry passes preferred_pipeline = None so the scheduler re-runs and the dead/dropped peer is filtered out via the liveness oracle above. Failure of the second attempt propagates to the user with a "try again" hint.

One source of "silent rr drops" was found and removed in v0.3.219: the request-response layer kept connections the swarm had already refused (the per-peer limit of 3) and, whenever the real connections each had a request in flight, sent the next one down a refused connection, where it vanished until the 600 s request timeout. Bursts of streamed tokens lost their tail this way. A refused connection is now forgotten and anything sent to it reported failed (gotcha #774); the retry above still covers any other loss.

Independently, streaming-tracked SendDirectMessage sends carry a delivery_request_id; if the receiver doesn't ACK within RR_ACK_TIMEOUT_SECS (10s), the daemon closes the caller's streaming channel — converting a first-token-deadline hang (120 s plus 0.5 s per prompt token, at most 600 s, plus 240 s per computer that may first have to load the model) into a fast-fail in ~10–20s. This handles the rare case where libp2p request_response accepts a send_request call but never delivers it (no OutboundFailure event fires).

Concurrent Request Throttling

Every request gets the same middle tier while credits are dormant, and that tier runs at most HALF of max_concurrent_requests at once (5 with the default 10). Requests beyond the cap queue in the router. The queue is event-driven: every active_count.fetch_sub(1) on completion is paired with queue_notify.notify_one() so drain_queue wakes immediately. Without that pairing, queued requests would sit indefinitely until the next Submit arrived (a real bug found in stress testing — fix in commit da6f485).

Pipeline affinity means that multi-turn conversations (with session_id) prefer to route through the same nodes, preserving KV-cache state and avoiding cold restarts on every turn.

The Parallax allocator also runs offline in AutoShardManager (Phase C.2) with a soft acquire/prune bias driven by a per-shard stability counter (≥3 consistent ticks of "this shard wants to move here" before it acts). Hard constraints (pinning, trust gates, VRAM caps) always win.

Architecture Detection

The SplitModel loader reads general.architecture from GGUF metadata and applies per-architecture handling:

The RoPE column is llama.cpp's per-architecture choice (llama_model_rope_type), which ModelArch::use_rope_contiguous copies arch by arch — it is not inferred.

Mixture-of-experts layers keep every expert quantized, one matrix per expert cut from the GGUF's stacked tensor, the way llama.cpp runs them — so a MoE model needs about its file's size in memory, and it is planned at its full size (every expert is resident; nothing pages experts in and out). On a graphics card too small for all of it, the experts of as many layers as needed are kept in the computer's memory and everything else stays on the card; the card runs experts stored as Q2_K-Q6_K, Q8_0, Q4_0, Q4_1, Q5_0 or Q5_1 (the last four since 0.3.241, for Gemma 4's 26B-A4B). Models quantized with the IQ* family (IQ1_S, IQ3_XXS, …) cannot be read by this engine and are refused with a message naming the type; choose a K-quant.

KV-Cache Management

• Per-request isolation via DashMap<(ModelKey, RequestId), Cache>
• Multi-turn reuse: session_id tracks conversations, prefix matching skips redundant prefill
• Configurable TTL (default 10 min)
• VRAM-aware LRU eviction for split model cache
• Sliding-window layers (Gemma 2, 3 and 4) keep only the positions their window still reads, as llama.cpp does — a Gemma 4 12B conversation at 32,000 tokens holds about 1 GB of cache instead of 10.5 GB — and every memory estimate (this computer's admission and placement, and the planner's bound on a peer that advertises it) charges them that

Prefix-Cache KV Sharing (Cross-Node)

Each worker stores a local prefix-cache keyed by BLAKE3 chained hashes over fixed-size token blocks (prefix_cache_block_tokens, default 64). Blocks are announced to peers via SwarmMessage::PrefixCacheAnnounce on the swarm/models gossipsub topic and indexed in state.models.cross_node_prefix_index.

When a local worker sees a prompt whose prefix it hasn't prefilled, it emits WorkerMsg::PrefixFetchProbe; the daemon walks the index (longest-match first), trust-gates candidate peers by cross_node_prefix_trust_min (default 0.5), and issues a SendPrefixKvFetch request-response to the best holder. The serving daemon re-issues DaemonMsg::ExportPrefixSnapshot to its worker, which narrows a stored KvSnapshot to the requested block boundary and returns the serialized bytes in the IPC binary-payload slot. Back on the requesting side, the bytes are BLAKE3-reverified against the requested hash and NaN/Inf-scanned before hydrating a new KvCacheEntry for the in-flight request, which then only has to prefill the suffix beyond the cached block boundary.

Three chained timeouts (worker probe 3000 ms, daemon network 2500 ms, serving IPC 2000 ms — sized for 7B-class f32 snapshots) guarantee that a stuck peer degrades to a clean miss rather than blocking the request. See the Performance chapter for measured TTFT numbers on TinyLlama (GPU, corner case where fetch is slightly slower than prefill) vs Qwen2.5-7B (12.9× iter-1 TTFT speedup on CPU-CPU localhost).

Advanced Features

• Speculative Decoding — Draft model proposes K tokens, target verifies in one pass (flag-gated speculative_distributed)
• SWIFT self-speculative — Target model acts as its own draft by skipping a layer range (flag-gated swift_self_speculative)
• DSD (Decentralized Speculative Decoding) — Multi-segment pipeline with γ-token speculation woven in (decentralized_spec_decoding, on by default since v0.3.213)
• Chunked Prefill — Sarathi-style: each Prefilling slot advances by prefill_chunk_tokens (default 128) per decode tick so a long admission can't block decode
• Continuous Batching — default-on: concurrent Generate requests share one forward_batch per decode tick; GPU uses fused kernel, CPU falls through to sequential
• Batched Prefill Forward — default-on: fuses concurrent same-shape Prefilling chunks into one forward_batch call
• Remote-generate Fast Path — default-on: single-segment distributed inference runs the full decode loop on the remote worker instead of per-token coordinator round-trips (measured 1.93× decode speedup)
• Delegated split (v0.3.219) — default-on: a request whose plan is several segments, none of them this node's, is handed to the node holding the model's first layers, which leads it among its own peers and streams the reply back the same way. The requester no longer sits inside every token's round trip: measured on a rig with the requester 150 ms away, 1.4-1.7× faster with two or three holders. The delegate keeps the first segment itself (the requester chose it to read the prompt) and never hands the request on; it is not used in private mode or with "Start and finish on this computer" (encrypted_pipeline). SWARMLLM_DELEGATE_SPLIT=0 switches it off
• Chained checks (v0.3.219) — a speculative check travels a run of remote segments like an ordinary decode step, one trip around them instead of one per segment (+31% on a guessing request across three nodes, the leader 150 ms from the others). SWARMLLM_CHAIN_VERIFY=0 switches it off
• Cross-request Prefix Cache — default-on: see "Prefix-Cache KV Sharing" above for the cross-node extension; the local cache alone is a 29.4× wall-clock win on prompt re-submission
• Activation Compression (Q8_0) — Intermediate pipeline activations wire-quantized ~3.76× (activation_compression, on by default)
• Flash Attention — CPU and GPU fast paths (GQA-native, no repeat_kv)
• PagedAttention — Not implemented. A vendored kernel crate (vendor/candle-paged-attention) is in the workspace, but nothing references it and there is no feature flag for it (#257)
• Logprobs — NOT returned by local inference. The sampler can compute them (sample_token_with_logprobs) and the response type serializes them, but every local execution path pins token_logprobs: vec![], so nothing reaches the response. /v1/chat/completions therefore REFUSES logprobs for a locally-served model rather than answering 200 with the field absent, which is indistinguishable from a request that never asked. Cloud-routed models still return them. See docs/ARCHITECTURE.md § Deferred Items
• No failure broadcast — A failed request is not announced to the swarm. It used to be, but receivers only logged it, and its text named the model, which told every peer what had failed here, in private mode too (#251). Receivers still accept the message from older nodes
• Local Embedding Privacy — When local_embedding_privacy: true, the requesting node performs token→embedding locally (~1ms) and sends pre-embedded hidden-state activations instead of raw token IDs to the first pipeline segment. Remote nodes never receive the prompt as text or token IDs, but they still compute on hidden states that can be turned back into much of it. See Security > Local Embedding Privacy
• Start and finish on this computer (encrypted_pipeline, "boomerang") — When enabled (per-model or global), the requesting node runs both the first segment (embedding) and the last (token sampling), so no other computer is handed the prompt as text or picks the reply's words. This is structural, not cryptographic: the middle computers still compute on hidden states in plaintext, which published attacks turn back into much of the text. See Security > Start and Finish on This Computer

Vision Language Models (VLM)

Distributed mmproj

The mmproj (vision encoder) is modeled as a sentinel shard (index = u32::MAX) decoupled from the text pipeline. Any node with mmproj can encode images — the router selects local → first-segment → any holder.

Image → JPEG compress → VisionEncodeRequest (remote) or encode locally
    → zstd+FP16 compressed embeddings
    → attached to first LayerForward (vision_embeddings field)
    → text pipeline processes as normal

Key types: VisionEncodeRequest, VisionEncodeResponse, LayerForward.vision_embeddings.

If no node has mmproj loaded, the API returns HTTP 503 (VisionEncoderUnavailable).

Tensor Wire Format

[4B ndim][4B×ndim shape][4B dtype_tag][f32 data]

For a 7B model (hidden_dim=3584):

• Prefill (14 tokens): ~200 KB
• Decode (1 token): ~14 KB

Credit System

⚠ Credits are switched off, and gate nothing

Since 2026-08-17 the credit economy is dormant. MIN_BALANCE_FOR_INFERENCE is 0 and calculate_tier returns the same tier whatever balance it is given, so no balance affects who is served, how fast, or what any surface shows. Nothing displays a balance either — not the dashboard, not the leaderboard, not the command line. (/metrics still exports the raw ledger figure as swarmllm_credits_balance for anyone graphing it; it gates nothing.)

The reason is simple and worth stating: credit has never moved between two machines as payment for work. Each node mints its own figure, and the one real transfer — pool forwarding — just concentrates self-minted numbers. Showing someone a number like that as earnings is a claim the project cannot stand behind; acting on it is worse.

The accounting below still runs and the rates below are real — it costs nothing and the recorded figures are the only honest data about what traffic actually looks like, which is what a re-enabled economy would have to be designed from. But read every "ensures", "enforced" and "deprioritized" on this page as what the design does when switched on, not as what your node does today.

The full picture — what is actually true now, why the obvious fix is wrong, the bilateral-settlement design, and the exit criteria that must hold before any of it comes back — is in docs/CREDITS_DESIGN.md.

Credits are a dormant design for fairness — no blockchain, no token, just local accounting with dual-signed transactions. Switched on, it would reward contributors and deprioritize free-riders; today it gates nothing.

Earning & Spending

Balanced rates: Both earn and spend use rate × tokens — no layer multiplier. A 22-layer model serving 100 tokens earns the same as it costs to consume, preventing credit inflation.

All rates are configurable per pool via [pool.credit_rates] in config.

Minimum Balance Enforcement (switched OFF)

MIN_BALANCE_FOR_INFERENCE is 0, which the enforcement check reads as "no floor". No requester is refused over credits today. It was -1000, and the paragraph below describes what that did. It was switched off because refusing somebody service over a self-minted figure is not a small inaccuracy — it is denying them the product on the strength of a number nobody reconciled, and a node driven negative by a bug stayed refused with no recovery path.

Nodes with balance below -1000 credits had remote inference requests rejected. They receive a clear error message telling them to contribute (host shards, serve inference, seed data).

• Local API requests (from localhost) are always allowed regardless of balance
• This prevents free-riders from endlessly consuming without contributing
• The floor is configurable via MIN_BALANCE_FOR_INFERENCE constant

Priority Tiers (switched OFF)

calculate_tier returns the same tier for everyone whatever balance it is given, so the table below is what the design does when switched on, not what your node does. Per-requester concurrency isolation is preserved — one peer still cannot monopolise the queue — while the advantage from a minted number is gone.

Tiers are calculated from your credit balance relative to the network:

How it works: On each inference request, the router computes your network percentile from peer credit gossip data (deduplicated by NodeId to prevent Sybil stuffing) and calls calculate_tier(). Higher tiers dequeue first. Bronze nodes are never fully blocked — they get deprioritized but always get at least 1 concurrent slot.

Anti-Abuse Mechanisms

• Anti-Sybil deduplication: Peer balance gossip is deduplicated by NodeId — a single peer can't stuff the percentile distribution by re-gossiping
• Atomic accumulation: Forward participation credits use AtomicI64 accumulator, flushed every 60s — no credits lost under high concurrency
• AntiGaming rate limiter: Max 100 credit transactions per node per 5-minute window
• Self-dealing rejection: Transactions from/to same node are rejected
• Signed balance reports: Ed25519 signatures with 5-minute freshness window

Failure Penalties

When distributed inference fails:

• The requesting node is penalized (configurable penalty_serve_failure, default 50 credits)

Transaction Security

• Every transaction requires dual Ed25519 signatures (serving node + requesting node)
• UUID deduplication prevents replay attacks (checked against DB)
• Balance arithmetic uses saturating_add (no overflow panics)
• Peer balance gossip rejects implausible values (abs > 100M)

Escrow

For large requests (above configurable threshold), credits are held in escrow:

• create_escrow() → release_escrow() (success) or refund_escrow() (failure)
• Balance deducted BEFORE escrow persisted (crash-safe: lose credits > create free credits)
• Refunds are persisted to DB immediately
• Entries expire after 10 minutes with automatic refund
• Escrow and direct charge are mutually exclusive (no double-billing)

Device Pool Credit Forwarding

When devices are linked in a pool, member devices forward their earnings to the owner:

• Credit split configurable: 0-100% kept by member (default 0), the rest forwarded
• Dual-signed PoolCreditForward (member signature + owner co-signature)
• Forwarded amount deducted from member balance before persisting
• Owner's PoolManager validates and applies credits atomically

Security & Encryption

Two Operator-Facing Encryption Layers

SwarmLLM ships with two distinct encryption layers that solve different problems. The first is on by default; the second ("Start and finish on this computer") is opt-in, and also switches on by itself for any model whose first and last shard this node holds (inference.encrypted_pipeline_auto, default on).

Layer 1 is "encryption in transit." Layer 2 keeps the two ENDS of the pipeline on your machine — a structural guarantee, not a cryptographic one: the nodes doing the work in the middle can still recover much of your text from what they compute on (see Activation inversion). For anything sensitive, use Private Mode or run the model entirely on your own machine. Disabling Layer 1 is only sensible for local-loopback debugging — there is NO plaintext fallback on seal() failure (forwards are dropped).

R139 hardened Layer 1 by offloading the ChaCha20-Poly1305 seal/open operations from the NetworkManager event loop via tokio::spawn, so encryption no longer adds jitter to libp2p ping / gossip / connection handling under concurrent decode load.

Three Encryption Tiers (internal mechanisms)

Tier 1: Pairwise Sessions (Unicast)

Underlying mechanism for the operator-facing Layer 1 above. For direct peer-to-peer communication:

• Ed25519 → X25519 → ECDH → ChaCha20-Poly1305
• Forward secrecy via ephemeral X25519 re-keying every 10 minutes
• Nonce reuse prevented by session clearing on disconnect (remove_session())
• Replay protection: RFC 6479 sliding window (128-bit bitmap) — allows packet reordering within window while rejecting duplicates
• Nonce state updated only after successful decryption (prevents DoS)
• Pending ephemeral keys expire after 60 seconds (prevents memory exhaustion from unanswered re-keys)
• AAD covers the cleartext header AND every optional trailer it emits (tensor-parallel, spec, kv-truncate, chunk-meta, chain next-hop/reply-to, generated-ids, pre-embedded, sampling — markers 0x02–0x0A) via build_layer_forward_aad — flipping cleartext metadata on the wire fails Poly1305
• Wire tag: TENSOR_TAG_ENCRYPTED = 0x10 — a session-sealed LayerForward (activations sealed; header and trailers travel in cleartext, bound as AAD)

Tier 2: Pipeline Sealing (Inference)

Designed to seal the final segment's output token IDs to the requester's X25519 key (per-request ephemeral key, crypto::pipeline_seal), so that only the requester could read them.

Not active today. The final segment calls the seal without the requester's key (src/daemon/dispatch/layer_forward.rs, "Deliberately NOT fed the requester id"), so it never runs: every result path on the coordinator would first have to unseal. Returned results travel under libp2p's transport encryption (Noise on TCP, TLS 1.3 on QUIC), not under this seal. Either way the final-segment node sees the sampled tokens, because sampling happens on that node. See Pipeline Privacy Model for a full breakdown of what each node can see.

Tier 3: Sealed Gossip (Broadcasts)

For GossipSub messages:

• Epoch-based group key + mandatory Ed25519 origin signature
• All gossip messages MUST be seal_signed() — unsigned messages are rejected
• Verifies sender authenticity before processing
• 1-hour rotation cycle

Transport-Authenticated Dispatch

All inbound network messages carry transport-authenticated sender identity:

• libp2p Noise protocol authenticates peers at the transport layer
• AuthenticatedMessage wrapper carries the verified NodeId of the sender
• MessageDispatcher validates sender identity against message claims:
  • ShardAnnounce: sender must match announce.node_id
  • CreditTransaction: sender must be a party (from or to)
  • CreditGossip, NicknameGossip: sender must match claimed node_id
  • HealthPing/Pong: sender must match claimed node_id
  • EphemeralKeyExchange: sender must match exchange.node_id
• Mismatched messages are logged and dropped

DHT Provider Records

The Kademlia DHT holds only provider records (start_providing / get_providers) saying which peers hold which shard. They are not signed payloads:

• start_providing_shards() announces a shard key; it signs nothing
• A provider record is only as trustworthy as the peer that announced it — which is why every shard fetched from a provider is BLAKE3-verified on arrival, and a failed check costs that peer 0.20 trust (ShardVerificationFail)
• Provider records expire after 12 hours and are republished every 4 hours (since 0.3.243; they were 1 hour and 20 minutes)
• verify_dht_value() would check [32B pubkey][64B signature][payload] values on a GetRecord result, but this build issues no GetRecord queries, so it never runs

Identity

• Ed25519 keypair generated on first run, stored in identity.key
• Private key never leaves the machine
• Public key = Node ID (first 8 bytes hex for display)
• Nickname system: Ed25519-signed records with timestamp-wins conflict resolution
• Nickname registry capped at 10,000 entries (requires peer_registry membership)

Trust & Reputation

TrustManager tracks per-peer scores (0.0-1.0, default 0.5):

ValidTransaction, InvalidGossip, SignatureViolation and SubnetClustering are defined in src/credit/trust.rs but nothing applies them.

Scores decay toward 0.5 over time (1% per health cycle, default 30 seconds). Trust orders pipeline candidates and gates cross-node prefix-KV fetches (cross_node_prefix_trust_min).

Sybil Resistance

• Subnet clustering (>5 nodes per /24) is tracked; the elevated check rate it triggers applies only to inbound credit transactions, which no node sends while credits are dormant, so today it has no effect
• Signed-only balance reports
• Timestamp freshness checks on gossip (5 min window, rejects >5 min old)

API Authentication

• Auto-generated 32-byte hex Bearer token (constant-time comparison)
• Protected: every route not listed as exempt below — including all of /v1/*, /api/* (reads as well as writes) and /mcp
• Exempt: /, /health, /health/ready, /admin, /chat, /setup and their sub-paths, /static/* and /favicon.ico
• /metrics is exempt from loopback only — and not even then when api.metrics_auth_required = true
• /api/admin/ws is exempt from Bearer auth but needs a single-use, short-lived ticket from POST /api/admin/ws-ticket (which is Bearer-authed)
• GET /api/admin/api-key answers without a key only to a network this node trusts (see api::dashboard_trust) and only with the dashboard's one-time page nonce
• Request body limit: 32 MB (raised from 2 MB to support VLM image payloads)
• Content-Security-Policy: default-src 'self'; script-src 'self'; style-src 'self' 'unsafe-inline'; connect-src 'self'; img-src 'self' data: blob:; font-src 'self'; frame-ancestors 'none'; base-uri 'self'; form-action 'self'
• X-Content-Type-Options: nosniff
• X-Frame-Options: DENY
• Referrer-Policy: no-referrer
• WebSocket Origin validation (rejects cross-site WebSocket hijacking)

Input Validation

• Model field length: max 256 chars in OpenAI + Anthropic handlers
• Tools array: max 128 entries
• Stop sequences: max 16 entries
• HuggingFace repo_id: validated owner/repo format (alphanumeric, hyphens, dots, underscores, max 96 chars)
• HuggingFace filename: must end in .gguf, no .., no URL metacharacters
• Path traversal: sanitize_path_component() on all network-provided model IDs before filesystem operations
• Update URLs: only GitHub download URLs accepted
• Update binaries: SHA-256 checksum verification is mandatory, and the checksum file must carry a valid minisign signature from the offline release key compiled into the binary, with the asset name and version bound in the signature's trusted comment. An unsigned release is refused (src/update_signature.rs)

Rate Limiting & DoS Protection

• Per-IP rate limiter with periodic cleanup (5 min intervals)
• Work for peers: a node-wide and a per-requester concurrency cap (see Misuse of the Network)
• Inference queue depth cap: 512 requests
• HTTP timeout: 5 minutes (Slowloris protection via tower-http TimeoutLayer). Model-running routes sit outside it (generation is unbounded in time) but still inside authentication and rate limiting.
• Credit transaction signature verification before ledger apply

Pipeline Privacy Model

Distributed inference splits a model across multiple nodes. This creates inherent privacy trade-offs — each node in the pipeline must process data to do its job. This section documents exactly what each node can see.

What each node sees during inference

Consider a 3-node pipeline: Requester → Node A (layers 0-10) → Node B (layers 11-21) → Node C (layers 22-27, final):

*Node A's visibility depends on the local_embedding_privacy setting — see below.

Risk: First-segment node sees raw tokens (default)

Without local_embedding_privacy (default): The first-segment node (Node A) receives the raw prompt text or token IDs to perform the embedding lookup. This means Node A can read the user's prompt in plaintext.

With local_embedding_privacy: true: The requesting node performs the embedding lookup locally and sends pre-embedded activation tensors. Node A receives floating-point vectors instead of token IDs. This is a significant privacy improvement, but not absolute — see Activation Inversion Risk below.

Risk: Final-segment node sees generated output

The final-segment node (Node C) must sample tokens from the logit distribution. This is fundamental — sampling is the act of choosing the next word, and it can only happen where the final layer's output logits exist. Node C therefore sees every generated token. (Tier 2 pipeline sealing, which would seal them to the requester's key, is not active — see above.)

This cannot be mitigated architecturally. The node that runs the last transformer layer and samples tokens will always know what tokens were sampled. The results travel back under the connection's transport encryption, which stops eavesdroppers on the network — but not the final-segment node itself.

Risk: Activation inversion attacks

All intermediate nodes see hidden-state activation tensors (floating-point matrices), in plaintext: the wire encryption is removed on arrival, because a matrix multiply cannot run on ciphertext. And hidden states are not safe to treat as meaningless numbers:

• Embedding-layer activations (layer 0 output) are essentially a lookup table and can be reversed trivially.
• Deeper layers are harder, but still leak a great deal. Published inversion attacks (2025) recover roughly 81% of the input text even from final-layer hidden states. Keeping more layers on your own machine therefore does not make a remote node blind — which is why there is no "keep N layers local" privacy setting.

Treat a remote node doing work on your request as able to read it. Truly hiding a prompt from the computer that processes it needs encrypted computation (homomorphic encryption or multi-party computation), which is currently about three orders of magnitude too slow to use here.

What SwarmLLM does, and what each measure protects:

1. Encryption in transit — every peer connection is encrypted (Noise / TLS), and activation forwards are additionally sealed with ChaCha20-Poly1305 (Tier 1). This stops eavesdroppers on the network, not the nodes doing the work.
2. local_embedding_privacy: true — the first remote node never receives the raw tokens or the trivially reversible embedding output. It removes the easy case, not the risk.
3. Private Mode — the only setting that keeps a request away from strangers' machines: your requests are served by your own devices (and, optionally, your local network) only.

Risk: Byzantine tensor manipulation

A malicious node can send garbage activations instead of computing the actual transformer layers. Mitigation is limited: every distributed result gets a well-formedness check (tokens reported with no text, text with no tokens, or a reply that ran to the token cap repeating one token). A malformed result earns no participant any trust, and when a single peer served it, that peer loses 0.10 trust. Fluent but wrong output passes the check, because there is no replicated or known-answer verification (src/inference/router/spot_check.rs).

Summary of privacy guarantees

All rows below assume network.enable_encryption = true (the default) — every peer connection is encrypted in transport, and activation forwards are additionally ChaCha20-Poly1305 sealed, regardless of which row you're in. The columns describe what the endpoints see; the wire is always encrypted.

Bottom line: Layer 1 (basic wire encryption) is on by default — eavesdroppers on the network see ciphertext. The "boomerang" mode (encrypted_pipeline) keeps the two ENDS of the pipeline on your machine, so no remote node handles the prompt text or picks the reply's words. It is a structural guarantee, not a cryptographic one: the middle nodes still compute on your hidden states in plaintext, and those can be partly turned back into text. For anything sensitive, use Private Mode or run the model entirely on your own machine.

Local Embedding Privacy

When local_embedding_privacy: true is set in [inference] config, the requesting node performs token→embedding lookup locally before sending activations to the first pipeline segment. Remote nodes never see raw token IDs — only hidden-state activation tensors.

How it works:

1. On startup, LocalEmbedder loads token_embd.weight from shard_000.bin and dequantizes it to f32 in RAM (vocab × hidden × 4 bytes — far larger than the quantized table on disk)
2. The requesting node tokenizes the prompt and performs the embedding lookup locally (~1ms)
3. The resulting hidden-state tensor ([1, seq_len, hidden_dim], FP32) is sent as LayerForward.activations with pre_embedded: true
4. The receiving first-segment node skips its embedding lookup and processes the pre-embedded activations directly

Wire format: pre_embedded travels as its own LayerForward trailer (0x09), bound into the seal's AAD. It is sent only to a first-segment peer that advertises features::FORWARD_PRE_EMBEDDED (v0.3.195 and later). If the peer holding segment 0 does not advertise it, the request is refused (PromptPrivacyUnavailable) rather than sent as raw tokens.

Trade-off: Pre-embedded activations are larger than raw text (e.g., 512 tokens × 4096 hidden × 4 bytes = 8MB vs ~2KB text). This matches the existing inter-segment activation sizes, so it does not change the bandwidth profile of distributed inference.

Relevant code: src/inference/local_embedder.rs, src/inference/pipeline/, src/daemon/state/mod.rs (local_embedders DashMap).

Start and Finish on This Computer (boomerang)

When encrypted_pipeline: true is enabled (globally or per-model), the pipeline scheduler forces the requesting node to handle both the first and last segments. This creates a "boomerang" topology:

Requester (shard 0, embed) → Remote A (middle shards) → ... → Requester (final shard, decode)

No remote node handles the raw prompt tokens or samples the generated output — those two ends stay on your machine. Remote nodes process intermediate hidden-state activations, in plaintext and partially invertible back to your text (see Activation inversion), so this is a structural guarantee, not a cryptographic one.

Requirements:

• The requesting node must hold shard 0 (embedding table) AND the final shard (output head)
• local_embedding_privacy is auto-enabled when this setting is active
• Only useful for models with 3+ shards (2-shard models = fully local, no distribution)

Overhead:

• Adds ~1 extra network RTT per generated token (activations must return to the requester for final decoding)
• Latency increase depends on distance to the furthest remote segment
• No bandwidth overhead vs normal distributed inference (activation sizes are the same)

Per-model configuration:

• API: GET/PUT/DELETE /api/admin/models/{id}/encrypted-pipeline ({"enabled": null} or DELETE returns the model to the global and automatic rules)
• Dashboard: gear icon on model card → "Start and finish on this computer" checkbox
• Global fallback: encrypted_pipeline = true in [inference] config
• Automatic: encrypted_pipeline_auto = true (default) turns it on for any model whose first and last shard this node holds, unless a per-model setting says otherwise
• Fetch the two ends so it can engage: swarmllm privacy <model> or POST /api/admin/models/{id}/enable-privacy
• Per-model overrides are persisted to the database

Relevant code: src/inference/scheduler/mod.rs (greedy_assign), src/inference/pipeline/ (auto-enable local embedding), src/api/admin_models/lifecycle.rs (API endpoints), src/daemon/state/mod.rs (encrypted_pipeline_models DashMap).

Misuse of the Network

Everything above protects the people using the swarm: who can read a prompt, who can fake a result. This section is about the swarm itself as something a bad actor can use, and states what is defended today and what is not.

The test applied here: a limit counts only if someone other than the party it is meant to stop enforces it. A limit in a node's own code binds only honest operators, because a fork removes it. A limit that honest serving nodes apply to the nodes asking them for work survives any fork of the asking side.

A fork run as a private swarm

SwarmLLM is Apache-2.0: anyone can build it, change it and run it as a private network — on machines they control, or on machines they do not, the way legitimate mining software ends up on botnets. No code in the official build can prevent that. What the project can do is keep the official build easy to recognise and never quiet:

• Signed releases. Each release's SHA-256 checksum files are signed with the project's offline minisign key (release_pubkey.txt, also compiled into every binary). The executables themselves carry no Windows Authenticode or Apple signature, so a security tool that checks those sees an unsigned program.
• Nothing hidden. The process runs under its own name (swarmllm, or the release asset's name such as swarmllm-linux-x86_64-cuda), prints its dashboard address on start and opens the dashboard in a browser by default. The Linux package does not enable its service on install, the Windows installer script adds no autostart entry, and no code renames or hides the process. (That installer script uses Inno Setup, which accepts its standard /SILENT flags; it is not part of current releases.)
• A recognisable network footprint, for defenders:

A fork can change every one of these. They identify the official build and unmodified copies of it, which is what they are for.

A rogue consumer of the public swarm

Anyone can run a node and ask the swarm for inference. There is no account and, while credits are dormant, no price, so capacity other people contribute can be consumed by one party. What serving nodes enforce on requesters today:

• Authentication, not identity. Every peer request must come from a transport-authenticated peer this node knows. A node key costs nothing to make, so this names a requester; it does not limit one.
• A concurrency cap per requester on every kind of work a peer can ask for — tensor forwards, whole-model generation, image encoding — of half the node's total, floored at 4, inside a node-wide cap set by the owner's contribution level (8 / 24 / 64 at Minimal / Moderate / Maximum). A request that has not started yet may not take the last quarter of those slots — they are kept for the next steps of requests already running. A tensor forward or a whole-model request over any of these limits is refused with a reply at once, so the requester can go elsewhere (an image-encoding request cannot be told no yet; its requester waits out a timeout).
• The owner's contribution level bounds what the swarm can take: processor threads and memory are capped by it, so demand from the swarm can keep this node busy but cannot take more of the machine than its owner offered.

What is not enforced today:

• No rate or quota over time, only concurrency. A requester that stays inside the concurrency cap can keep a node busy indefinitely.
• No reputation check on the requester. Trust scores are read by a node choosing peers to SEND work to, never by a node deciding whether to SERVE.
• Per-requester limits key on the node key, which is free. A requester with many keys (a Sybil) gets many allowances. Only a cost attached to an identity — credits that actually move, or trust earned by serving — closes that, which is why limits enforced by serving nodes are a precondition for credits going live (docs/CREDITS_DESIGN.md § 6).
• The HTTP API's rate limiter (per client IP, 60 requests a minute by default) protects one node's own API. It does nothing for the swarm behind that node.

A rogue operator of many nodes

Many nodes under one operator can bias DHT routing (an eclipse) and return fluent but wrong output. Both are listed under Known Limitations below.

Nothing in this section is known to be happening. It is written down now because the levers that survive a fork — limits applied by serving nodes, and a cost on identity — are cheaper to design in than to retrofit.

Known Limitations

These are architectural properties that cannot be fully mitigated with code changes:

• Misuse of the network — anyone can ask the swarm for work, and every limit on a requester keys on an identity that costs nothing to make (see Misuse of the Network).

• Gossip epoch key is publicly derivable — derived from "swarmllm-mainnet-v1". Gossip encryption is defense-in-depth; Ed25519 signing is the primary security mechanism.

• Final-segment output visibility — the node running the last transformer layers sees all generated tokens. This is inherent to the architecture (see Pipeline Privacy Model).

• Activation inversion — each computing node decrypts the hidden states it works on, and published 2025 attacks recover roughly 81% of the input text even from final-layer hidden states (see Activation inversion). local_embedding_privacy removes only the trivial case (reversing the embedding lookup). Treat any remote node computing on your request as able to read it.

• Byzantine tensor manipulation — malicious peers can send garbage activations. Mitigation is limited to a well-formedness check on every distributed result; fluent but wrong output is not detected (see Byzantine tensor manipulation).

• Sybil credit farming — Ed25519 keys are free. Anti-gaming heuristics help but are not bulletproof.

• GGUF parser vulnerabilities — llama.cpp CVEs. BLAKE3 content hash gates shard loading but parser bugs remain upstream.

• Kademlia eclipse attacks — strategic Sybil node IDs can control DHT routing. K-bucket eviction policies help.

Storage & Data

Data Directory Layout

~/.local/share/swarmllm/
├── config.toml          # User configuration
├── identity.key         # Ed25519 secret key: 32 raw bytes, owner-only (mode 0600)
├── api_key              # Bearer token (auto-generated)
├── db.redb              # redb database (migrated from sled db/ directory)
├── canonical/           # the swarm's copy's header and side files, staged while
│   └── <model>/         # this computer checks its own copy against it
└── models/
    ├── qwen2.5-coder-7b/
    │   ├── manifest.json
    │   ├── gguf_header.bin
    │   ├── shard_000.bin
    │   └── shard_001.bin
    └── tinyllama-1.1b/
        └── ...

Database Tables (redb)

Every tree below lives in ONE redb table named data, keyed "{tree}\0{key}". The node's identity key is not in the database: it is the identity.key file above. Which parts a node holds is read from the models/ directory, not stored.

Model Acquisition Pipeline

Network Registry (GossipSub/DHT)
        │
        ▼
  Manifest Check ──► Reject if BLAKE3 mismatch
        │
        ▼
  Shard Selection ──► Rarest-first (BitTorrent-style)
        │
        ▼
  Download Loop ──► Atomic write to .tmp, rename to .bin
        │
        ▼
  Shard Verify ──► BLAKE3 vs manifest hash
        │
        ▼
  Model Ready

Integrity guarantees:

• Manifests verified via BLAKE3 self-hash
• Each shard verified against manifest hash
• Failed shards renamed .bin.quarantine, serving peer penalized
• Downloads retried (3 attempts, exponential backoff)
• Atomic writes prevent corrupt partial files
• Stale .tmp files cleaned on startup

One upload per model (v0.3.221)

A model's id comes from its file name, and several people publish their own copy of the same model and quantisation on HuggingFace — byte-different files under one id. Parts of two copies cannot be combined in a split, so every node uses the same upload of each model:

• One ranking everywhere. A pinned reference model first, then the publisher's place in the trusted list (the model's own author, then well-known curators, in a fixed order), then anyone else, ties by name. Every node uses the best upload anyone has claimed — after checking it on HuggingFace without a login token, so every node sees the same answer.
• Nothing is fetched until the choice is made, and a peer's manifest of another upload is ignored.
• Self-healing. A node holding another upload fetches the chosen upload's parts for the layers it holds (by byte range — never the whole file) into canonical/, keeps serving its old parts meanwhile, and swaps when every part is in and the model is idle. A node whose parts are right but whose header or manifest describe another upload gets the right ones and reloads.
• GET /api/admin/models reports it per model in shared_copy; SWARMLLM_CANONICAL_UPLOADS=0 turns the mechanism off (test rigs).

Details and evidence: docs/invariants/network.md § "One upload per model id".

OpenAI-Compatible API

SwarmLLM provides a drop-in replacement for the OpenAI API. All endpoints require Bearer token authentication.

POST /v1/chat/completions

Chat completions with streaming support.

curl http://localhost:8800/v1/chat/completions \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen2.5-coder-7b",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "What is Rust?"}
    ],
    "stream": true,
    "max_tokens": 512,
    "temperature": 0.7
  }'

Request Body

Response (non-streaming)

{
  "id": "chatcmpl-abc123",
  "object": "chat.completion",
  "model": "qwen2.5-coder-7b",
  "choices": [{
    "index": 0,
    "message": {"role": "assistant", "content": "Rust is a systems programming language..."},
    "finish_reason": "stop",
    "logprobs": null
  }],
  "usage": {
    "prompt_tokens": 15,
    "completion_tokens": 42,
    "total_tokens": 57
  }
}

Response with logprobs

When logprobs: true and top_logprobs: 3:

{
  "choices": [{
    "message": {"role": "assistant", "content": "Hello"},
    "finish_reason": "stop",
    "logprobs": {
      "content": [{
        "token": "Hello",
        "logprob": -0.234,
        "bytes": null,
        "top_logprobs": [
          {"token": "Hello", "logprob": -0.234, "bytes": null},
          {"token": "Hi", "logprob": -1.456, "bytes": null},
          {"token": "Hey", "logprob": -2.012, "bytes": null}
        ]
      }]
    }
  }]
}

Response with tool_calls

When the model calls a tool, finish_reason is "tool_calls" and content is null:

{
  "choices": [{
    "message": {
      "role": "assistant",
      "content": null,
      "tool_calls": [{
        "id": "call_abc123",
        "type": "function",
        "function": {
          "name": "get_weather",
          "arguments": "{\"location\":\"NYC\"}"
        }
      }]
    },
    "finish_reason": "tool_calls"
  }]
}

Streaming (SSE)

When stream: true, responses arrive as Server-Sent Events:

data: {"id":"chatcmpl-abc123","choices":[{"delta":{"content":"Rust"},"index":0}]}

data: {"id":"chatcmpl-abc123","choices":[{"delta":{"content":" is"},"index":0}]}

data: [DONE]

GET /v1/models

List available models.

curl http://localhost:8800/v1/models \
  -H "Authorization: Bearer YOUR_API_KEY"

{
  "object": "list",
  "data": [
    {
      "id": "qwen2.5-coder-7b-instruct-q4-k-m",
      "object": "model",
      "created": 1756000000,
      "owned_by": "local",
      "max_model_len": 8192,
      "context_length": 8192
    }
  ]
}

owned_by says where the model's parts sit: local (every part is on this computer), hybrid (some parts here, some on peers) or network. It does not say where a request runs. A local model that this computer's graphics card cannot hold whole may still be run with peers' help, and peers then compute on the model's working data for the layers they run. They are not handed the prompt as text, but that data can be turned back into much of it (Security > Start and finish on this computer). To keep one model's requests on this computer, turn on Keep on this computer in the model's ⚙ panel on the Dashboard (or PUT /api/admin/models/{id}/keep-on-this-computer with {"enabled": true}): no other computer, in the swarm or a cloud provider, is then handed anything of those requests, and one this computer cannot run alone is refused with kept_on_this_computer_error (503) instead of being sent out. To keep every request on your own computers, use a private pool with private_mode = true and private_mode_allow_lan = false (Joining the network > Private Networks).

Every reply says which computers answered it. A reply that is not streamed carries the x-swarm-route, x-swarm-segments, x-swarm-peers (how many OTHER computers took part), x-swarm-nodes and, where they declared one, x-swarm-regions headers. The same facts are in the reply itself, as "swarmllm": {"route": {"route": "distributed", "segments": 2, "peers": 1, "nodes": ["0718d8b9", "96842635"], "attempts": 1, "all_nodes": ["0718d8b9", "96842635"], "all_peers": 1}}: in the body of a chat completion or an Anthropic reply, in the last chunk of a streamed chat completion (the one with finish_reason), in a streamed Anthropic reply's message_delta, and in the response object of /v1/responses (also when it is streamed, run in the background or read back later). A streamed reply has no route headers — they are sent before the route exists — so read it there. An MCP tool that asks one model puts it in the result's _meta["com.swarmllm/route"]; compare, research and batch_prompts give each model's result its own route. A reply a cloud provider wrote carries none.

route, segments, peers and nodes describe the computers the reply finished on. When a computer failed or refused part-way and the request was planned again (or its reply carried on elsewhere), the computers it was sent to first saw it too — so all_nodes and all_peers name every computer any attempt reached, this one included, and attempts says how many plans it took (headers x-swarm-all-nodes, x-swarm-all-peers, x-swarm-attempts). A reply that says local with all_peers above 0 was answered here after another computer had been handed it. The dashboard's chat says so beside the route, as "tried 1 other computer first".

To keep a single request on this computer, whatever the model's setting, send the header X-SwarmLLM-Keep-On-This-Computer: true, or put "swarmllm": {"keep_on_this_computer": true} in the body of /v1/chat/completions, /v1/messages or /v1/responses. An MCP tool call takes "_meta": {"com.swarmllm/keep_on_this_computer": true} in its params (or the same header on /mcp), and every model call the tool makes is kept. The request is then treated like one for a kept model: nothing of it goes to another computer or to a cloud provider, and if this computer cannot run it alone it is refused with kept_on_this_computer_error (503). The swarmllm field itself is never passed on to a provider. Whichever instruction is strictest wins — false never undoes the model's own setting — and an instruction the node cannot read (a header value other than true, false, 1, 0, yes or no, or a misspelt field name) is refused with a 400 rather than ignored.

curl http://localhost:8800/v1/chat/completions \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -H "X-SwarmLLM-Keep-On-This-Computer: true" \
  -d '{"model": "auto", "messages": [{"role": "user", "content": "Hello"}]}'

max_model_len and context_length are the same number — the longest conversation this node will serve, after the shipped 8192-token default and any inference.max_seq_len_override. The first is the name vLLM clients read, the second the name OpenClaw's model discovery reads. Both are omitted when the model's declared context cannot be read, rather than guessed. Up to v0.3.224 that is every model no part of which is on this machine; from v0.3.225 such a model reports the context its holders declare, once they run v0.3.225 or later too.

It is THIS node's limit. When a model is split, each computer applies its own, and each advertises it: a prompt is not sent to a computer that has said it serves less, and one that refuses is failed over from. If no computer that could run some part of the model serves the length, the request fails with a 400 that says the limit is set on those computers — raising your own setting does not change it.

GET /v1/status

Node status (SwarmLLM extension).

curl http://localhost:8800/v1/status \
  -H "Authorization: Bearer YOUR_API_KEY"

Using with OpenAI Client Libraries

Python (openai)

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8800/v1",
    api_key="YOUR_API_KEY"
)

# Basic streaming
response = client.chat.completions.create(
    model="qwen2.5-coder-7b",
    messages=[{"role": "user", "content": "Hello!"}],
    stream=True
)

for chunk in response:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")

Python — Function calling

response = client.chat.completions.create(
    model="qwen2.5-coder-7b",
    messages=[{"role": "user", "content": "What's the weather in NYC?"}],
    tools=[{
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get current weather",
            "parameters": {
                "type": "object",
                "properties": {"location": {"type": "string"}},
                "required": ["location"]
            }
        }
    }],
    tool_choice="auto"
)

if response.choices[0].finish_reason == "tool_calls":
    for tc in response.choices[0].message.tool_calls:
        print(f"Call {tc.function.name}({tc.function.arguments})")

JavaScript (openai)

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "http://localhost:8800/v1",
  apiKey: "YOUR_API_KEY",
});

const stream = await client.chat.completions.create({
  model: "qwen2.5-coder-7b",
  messages: [{ role: "user", content: "Hello!" }],
  stream: true,
});

for await (const chunk of stream) {
  process.stdout.write(chunk.choices[0]?.delta?.content || "");
}

curl (streaming)

curl -N http://localhost:8800/v1/chat/completions \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen2.5-coder-7b","messages":[{"role":"user","content":"Hello!"}],"stream":true}'

OpenClaw

OpenClaw accepts any OpenAI-compatible server as a model provider. Export your key as SWARMLLM_API_KEY (the daemon reads the same variable) and add a swarmllm provider to ~/.openclaw/openclaw.json:

{
  models: {
    providers: {
      swarmllm: {
        baseUrl: "http://127.0.0.1:8800/v1",
        apiKey: "${SWARMLLM_API_KEY}",
        api: "openai-completions",
        timeoutSeconds: 300,
        models: [{ id: "meta-llama-3.1-8b-instruct-q4-k-m", name: "SwarmLLM Llama 3.1 8B",
                   contextWindow: 32768, maxTokens: 4096,
                   cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 } }],
      },
    },
  },
  agents: { defaults: { model: { primary: "swarmllm/meta-llama-3.1-8b-instruct-q4-k-m" } } },
}

Two requirements, both on the SwarmLLM side:

• A real key. OpenClaw accepts a non-secret marker such as "ollama-local" for loopback servers, but a marker sends no Authorization header at all, and every /v1 route here requires one.
• A context window above the 8192-token default. OpenClaw's system prompt alone is larger than that, and its context guard refuses a model below 4k tokens and warns below 8k. Set max_seq_len_override = 32768 under [inference] in config.toml (config file only — it is not a dashboard setting or an environment variable).

Model ids are whatever GET /v1/models lists, referenced as swarmllm/<id>. Each entry carries context_length, which is the field OpenClaw's self-hosted discovery reads; with it absent OpenClaw assumes 128,000 and sends prompts the node has to refuse. Embeddings are not served here (see below), so point OpenClaw's memory search at another embedding provider.

POST /v1/embeddings

Returns 501 Not Implemented. Text embeddings are not supported via the subprocess inference path. Use a dedicated embedding provider or the OpenAI embeddings API directly.

GET /v1/providers

List configured cloud providers and their available models.

curl http://localhost:8800/v1/providers \
  -H "Authorization: Bearer YOUR_API_KEY"

Returns an array of { name, models: [...] } objects for each configured provider.

Responses API

OpenAI's /v1/responses is the default API for o-series and gpt-5-series models in 2026 and the replacement for the sunsetting Assistants API (2026-08-26). SwarmLLM exposes the full v1 surface plus follow-on features such as resumable streams, async background runs, MCP tool integration, and conversation chaining via previous_response_id.

Endpoints

All endpoints accept the same Bearer-auth header as the rest of the API.

Routing

POST /v1/responses picks one of three execution paths in this order:

1. Cloud proxy — when the requested model resolves to an OpenAI-routed provider, the request is serialized verbatim and forwarded to the upstream /v1/responses endpoint. Built-in tools, streaming, background, reasoning effort, text.verbosity, include[], previous_response_id, and any future field round-trip via #[serde(flatten)] extras.
2. Anthropic-Messages bridge (V3) — when the model resolves to an Anthropic provider (or the local claude-subscription subprocess), the Responses request is translated to an Anthropic Messages request, forwarded, and translated back. This lets Claude Code clients drive /v1/responses end-to-end without losing tool-call or streaming semantics.
3. Local inference — translates to /v1/chat/completions and runs on the local model. Function tools and tool_choice translate through; built-in tools (web_search, file_search, computer_use_preview, code_interpreter, image_generation, mcp, custom) are rejected with HTTP 400 because they require backing infrastructure SwarmLLM does not run.

A response that ran locally or on the swarm names the computers that served it, as "swarmllm": {"route": {...}} on the response object, whether it was streamed, run in the background or read back later (OpenAI-compatible API describes its fields).

Capabilities

• Multimodal input (V2) — input_image and input_file (UTF-8 only) parts in the structured input array. Binary file payloads (PDF, docx, image bytes via file_data) are rejected with a clear hint pointing at input_image.

• Function tools — tools definitions and tool_choice translate to OpenAI Chat Completions tool semantics; assistant tool_calls map back to function_call output items.

• Streaming SSE (M6 + V1) — stream=true emits the full Responses event sequence (response.created → response.in_progress → response.output_item.added → response.content_part.added → per-delta response.output_text.delta → response.output_text.done → response.content_part.done → response.output_item.done → response.completed). The V1 fix shipped in 2026-04-25 cuts first-token latency by emitting created and in_progress before model warmup instead of after.

• Persistence (M7) — store=true (the OpenAI default) writes the full response object to redb with a 30-day TTL. previous_response_id (M8) chains follow-up requests by prepending the prior turn's messages before the new input.

• Background mode (M9 + V8) — background=true returns HTTP 202 with a Location: /v1/responses/{id} header; the client polls or, with background=true && stream=true, opens a resumable SSE connection at GET /v1/responses/{id}?stream=true that replays buffered events and then tails the live producer.

  A background run that fails stores status: "failed" with an error.code describing why, not merely that something went wrong — the same code the foreground request would have returned for the identical input (not_found_error for a model that does not exist, invalid_request_error for a prompt longer than the context, upstream_error for a provider failure, server_error for an actual fault). Since a background caller never sees a status code, that field is the only thing distinguishing "fix the request" from "retry later".

Validation (ingress)

The handler runs validate_responses_ingress BEFORE any routing decision so the cloud-proxy and Anthropic-bridge paths can't forward attacker-sized strings to upstream providers (where they'd burn quota or land in log lines). Caps:

Stop / temperature / top_p / max_tokens are validated as on /v1/chat/completions: a value out of range is refused (400), never clamped.

Dashboard

The admin dashboard exposes a Responses tab (/admin/responses) backed by GET /api/admin/responses. It shows the most-recent stored response records with status, model, input snippet, and per-record cancel/delete actions.

Deferred

• POST /v1/responses/compact (V9) — no concrete caller has asked for it.
• Server-side conversation resource CRUD — OpenAI's conversation parameter forwards through cloud proxy verbatim today; a local conversation type with its own endpoints is a separate design.
• Built-in tools on the local path — see "Local inference" above.
• custom tools with Lark / regex grammars — rejected on local, forwarded on cloud. Local grammar-constrained generation is a candle-side project.
• Audio input on /v1/responses — input_audio returns 400; needs a Whisper-class transcription model SwarmLLM doesn't currently expose.
• Binary file inputs in input_file{file_data} — UTF-8 only; PDF/docx/ image-bytes payloads are rejected with a clear hint pointing at input_image (for images) or server-side text extraction.

Anthropic Messages API

SwarmLLM provides a full Anthropic Messages API at POST /v1/messages, enabling it to serve as a drop-in backend for Claude Code and other Anthropic-compatible clients.

Claude Code Integration

Use SwarmLLM as your Claude Code backend to access all models (local, network, and cloud) through a single endpoint:

ANTHROPIC_BASE_URL=http://localhost:8800 ANTHROPIC_AUTH_TOKEN="$SWARMLLM_KEY" \
  claude --model qwen2.5-coder-7b-instruct-q4-k-m

Environment Variables

The node accepts its key as Authorization: Bearer <key> or as x-api-key: <key>, which is what the Anthropic SDKs send.

POST /v1/messages

Request Body

Content Block Types

Messages can contain these content block types:

// Text
{"type": "text", "text": "Hello, world!"}

// Image (base64)
{"type": "image", "source": {"type": "base64", "media_type": "image/png", "data": "..."}}

// Tool use (assistant response)
{"type": "tool_use", "id": "toolu_123", "name": "get_weather", "input": {"location": "NYC"}}

// Tool result (user message)
{"type": "tool_result", "tool_use_id": "toolu_123", "content": "72F, sunny"}

// Thinking (extended thinking)
{"type": "thinking", "thinking": "Let me reason about this..."}

// Redacted thinking
{"type": "redacted_thinking", "data": "..."}

Response

{
  "id": "msg_abc123",
  "type": "message",
  "role": "assistant",
  "model": "qwen2.5-coder-7b",
  "content": [
    {"type": "text", "text": "Here's my response..."}
  ],
  "stop_reason": "end_turn",
  "usage": {
    "input_tokens": 25,
    "output_tokens": 150
  }
}

A reply SwarmLLM ran also names the computers that served it, as "swarmllm": {"route": {...}} in the body, or in the message_delta event of a streamed reply (OpenAI-compatible API describes its fields). A reply a cloud provider wrote carries none.

Model Routing

A model this node or the swarm holds is always used first. Only a name that is neither goes to a cloud provider, chosen by its name:

All 12 cloud providers are supported. Configure API keys in the dashboard (the gear icon, Settings → Cloud Providers) or by placing a .env file in the data directory (~/.local/share/swarmllm/.env) with standard variable names (e.g., OPENAI_API_KEY, DEEPSEEK_API_KEY).

System Blocks with Cache Control

Anthropic-compatible prompt caching:

{
  "system": [
    {"type": "text", "text": "You are a helpful assistant.", "cache_control": {"type": "ephemeral"}}
  ]
}

Streaming (SSE)

When stream: true, responses arrive as Server-Sent Events following the Anthropic streaming format:

event: message_start
data: {"type":"message_start","message":{"id":"msg_123","type":"message","role":"assistant","model":"qwen2.5-coder-7b","content":[]}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"Hello"}}

event: message_stop
data: {"type":"message_stop"}

Errors during a stream

If the request fails after the stream has opened — the model is unavailable, the node refuses by policy, the prompt is too long — the failure arrives as an error event, in the same shape the Anthropic API uses:

event: error
data: {"type":"error","error":{"type":"invalid_request_error","message":"This conversation is too long for …"}}

The error event is terminal: the stream ends there, with no message_delta or message_stop after it. The error.type is one Anthropic defines (invalid_request_error, not_found_error, authentication_error, permission_error, rate_limit_error, request_too_large, overloaded_error, or api_error), so a client matching on it behaves the same as against the real API.

A failure is never delivered as assistant text, and stop_reason never carries a value the API does not define — so a reply your client receives as content is always something the model actually produced.

MCP Server

SwarmLLM includes a native Model Context Protocol (MCP) server at POST /mcp. This enables AI agents like Claude Code, Cursor, VS Code Copilot, and other MCP-compatible tools to use your SwarmLLM node as a tool provider.

Protocol version: 2025-11-25 (JSON-RPC 2.0 over HTTP); 2025-06-18, 2025-03-26 and 2024-11-05 are also accepted.

Endpoint

POST /mcp
Content-Type: application/json
Authorization: Bearer YOUR_API_KEY

All requests use JSON-RPC 2.0 format. All tools include tool annotations (readOnlyHint, destructiveHint, etc.).

Available Tools

chat

Send a message to any model available on the node (local, network, or cloud). The result's _meta["com.swarmllm/route"] names the computers that served it (as does delegate's; compare, research and batch_prompts give each model's result its own route). A cloud provider's reply carries none.

{
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "name": "chat",
    "arguments": {
      "model": "qwen2.5-coder-7b",
      "messages": [
        {"role": "system", "content": "You are a helpful coding assistant."},
        {"role": "user", "content": "Explain Rust's ownership model"}
      ],
      "temperature": 0.7,
      "max_tokens": 2048
    }
  },
  "id": 1
}

models

List all available models (local + network + cloud).

{
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": { "name": "models", "arguments": {} },
  "id": 2
}

compare

Send the same prompt to multiple models concurrently and get side-by-side results. Up to 10 models per comparison.

{
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "name": "compare",
    "arguments": {
      "prompt": "Write a function to check if a number is prime",
      "models": ["qwen2.5-coder-7b", "gpt-4o", "claude-sonnet-4-20250514"],
      "system": "Write clean, efficient code.",
      "max_tokens": 1024
    }
  },
  "id": 3
}

research

Fan out a research question to multiple models in parallel. Designed for knowledge gathering — offload questions to cheap/fast models to get diverse perspectives without using expensive model tokens. If models is omitted, it picks up to max_models (default 5) models on this node or the swarm. It never picks a paid cloud model by itself — name cloud models in models to use them.

{
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "name": "research",
    "arguments": {
      "question": "What are the tradeoffs between ring-allreduce and star topology for tensor parallelism?",
      "models": ["deepseek-chat", "gpt-4o-mini", "qwen2.5-coder-7b"],
      "system": "Be concise and technical.",
      "max_tokens": 2048
    }
  },
  "id": 4
}

Response:

{
  "question": "What are the tradeoffs...",
  "models_queried": 3,
  "successful_responses": 3,
  "total_tokens_used": 1847,
  "results": [
    {
      "model": "deepseek-chat",
      "response": "Ring-allreduce...",
      "input_tokens": 24,
      "output_tokens": 512,
      "latency_ms": 2100,
      "status": "ok"
    }
  ]
}

batch_prompts

Execute multiple independent prompts in parallel, each targeting a specific model. Ideal for offloading parallel subtasks — e.g., ask one model to summarize, another to translate, another to review code, all at once.

{
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "name": "batch_prompts",
    "arguments": {
      "tasks": [
        {
          "id": "summary",
          "model": "gpt-4o-mini",
          "prompt": "Summarize this error log: ...",
          "max_tokens": 512
        },
        {
          "id": "fix",
          "model": "qwen2.5-coder-7b",
          "prompt": "Write a fix for this bug: ...",
          "max_tokens": 1024
        },
        {
          "id": "translate",
          "model": "deepseek-chat",
          "prompt": "Translate to Japanese: ...",
          "max_tokens": 256
        }
      ]
    }
  },
  "id": 5
}

Response:

{
  "tasks_submitted": 3,
  "tasks_completed": 3,
  "results": [
    {
      "task_id": "summary",
      "model": "gpt-4o-mini",
      "content": "The error log shows...",
      "latency_ms": 890,
      "status": "ok"
    }
  ]
}

delegate

Offload a task to the most appropriate model based on a tier preference. Tiers: fast picks the lowest-latency local model, cheap picks a small/free model, smart picks the most capable available model (may use cloud). Saves subscription tokens by routing routine work to local/cheap models.

{
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "name": "delegate",
    "arguments": {
      "prompt": "Summarize this function in one sentence: ...",
      "tier": "fast",
      "max_tokens": 256
    }
  },
  "id": 6
}

Tiers (tier is required):

• fast — lowest-latency local model
• cheap — smallest/free model available
• smart — most capable model (may use cloud provider)

node_info

Get detailed information about the SwarmLLM node: loaded models, connected peers, the ledger credit figure (credits are dormant and gate nothing), available cloud providers, and network status.

{
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": { "name": "node_info", "arguments": {} },
  "id": 7
}

Available Resources

swarmllm://status

Returns node status information (version, model loaded, peer count).

swarmllm://models

All models available for inference — local, network and cloud providers — with capabilities and status.

swarmllm://peers

Currently connected computers, with latency, trust, load and shard information.

{
  "jsonrpc": "2.0",
  "method": "resources/read",
  "params": { "uri": "swarmllm://status" },
  "id": 8
}

IDE Integration

Claude Code

Option A: MCP tools — access SwarmLLM's tools (research, batch, compare) alongside your normal model:

claude mcp add --transport http swarmllm http://localhost:8800/mcp \
  --header "Authorization: Bearer YOUR_API_KEY"

Option B: Model backend — use SwarmLLM as your inference backend (routes all requests through the swarm):

ANTHROPIC_BASE_URL=http://localhost:8800 ANTHROPIC_AUTH_TOKEN=YOUR_API_KEY \
  claude --model qwen2.5-coder-7b

Option C: Both — use Claude for reasoning, SwarmLLM MCP for offloading research to cheap models:

# Add SwarmLLM as MCP server
claude mcp add --transport http swarmllm http://localhost:8800/mcp \
  --header "Authorization: Bearer YOUR_API_KEY"

# Then use Claude normally — it can call research/batch/compare tools via MCP
claude

VS Code (Copilot Chat)

Add to .vscode/mcp.json in your project:

{
  "servers": {
    "swarmllm": {
      "type": "http",
      "url": "http://localhost:8800/mcp",
      "headers": {
        "Authorization": "Bearer YOUR_API_KEY"
      }
    }
  }
}

Copilot Chat will discover SwarmLLM's tools automatically. Use them by asking Copilot to research, compare models, or batch prompts.

Cursor / Windsurf / Other MCP Clients

Any MCP-compatible client can connect via HTTP:

URL: http://localhost:8800/mcp
Transport: HTTP (Streamable HTTP)
Auth: Bearer token in Authorization header

Continue.dev (OpenAI API)

If your IDE extension supports the OpenAI API format, point it directly at SwarmLLM:

{
  "models": [{
    "title": "SwarmLLM Local",
    "provider": "openai",
    "model": "qwen2.5-coder-7b",
    "apiBase": "http://localhost:8800/v1",
    "apiKey": "YOUR_API_KEY"
  }]
}

Model Compare Dashboard

The compare functionality is also available in the web dashboard via the Compare tab. Select 2-10 models, enter a prompt, and view results side-by-side with latency, token counts, and response content.

Errors

A tool that ran and failed answers the way MCP (2025-11-25) asks: a normal result with isError: true and the reason in its text, so the model driving the tool reads it and can correct the call, choose another model or tell you. That covers a model name that does not exist, a prompt longer than the context window, a missing required field, no models loaded, and a request that timed out. The text carries the same advice the dashboard shows, and _meta["com.swarmllm/error"] carries the API's error object (type, message, hint, hint_key) for a program that wants to branch on it:

{"jsonrpc": "2.0", "id": 4, "result": {
  "content": [{"type": "text", "text": "Model not available: no-such-model\n\nThis model isn't available yet. …"}],
  "isError": true,
  "_meta": {"com.swarmllm/error": {"type": "not_found_error", "hint_key": "model_not_available", "…": "…"}}
}}

Only a call no tool could run is a JSON-RPC error: an unknown tool name (-32602), an unreadable keep-on-this-computer instruction (-32602), and a node whose inference service is not running (-32603). Fan-out tools (compare, research, batch_prompts) do not fail the whole call when one model fails — each result carries its own status and error, and the summary counts only the ones that actually answered.

Admin API

Every /api/admin/* endpoint needs Authorization: Bearer <key> — the key is the api_key file in the data directory — reads included. The one exception is the WebSocket upgrade /api/admin/ws, which takes a single-use ticket from POST /api/admin/ws-ticket instead. Admin endpoints are also CORS-protected.

Node Management

GET /api/admin/stats

Node statistics and hardware info.

Its network_traffic figure (also on /v1/status and the dashboard's stats) says what this node's upload and download are made of. Since 0.3.243, direct_by_kind lists this node's own messages by kind, and by_protocol lists every byte by the network protocol its connection used, with opened_by: us for what this node asked, peer for what it answered. The by_protocol rows add up to out_bytes and in_bytes. Since 0.3.244, kad_inbound has one row for each kind of Kademlia request other nodes send this one (find_node, get_provider, add_provider, get_record, put_record, zeros included): requests handled, and the peers its answers listed — closer_peers, and provider_peers for the holders it named.

GET /api/admin/peers

Connected peers with latency, trust scores, and hosted models.

GET /api/admin/credits

The internal ledger figures (balance, lifetime earned and spent, escrow) and the tier name. Credits are dormant: the balance is self-minted, gates nothing, and the tier is the same for everyone.

GET /api/admin/credits/transactions

A bounded log of recent balance movements (delta, kind, reason, resulting balance), oldest first. lifetime_refunded is partly synthetic: unexplained gaps are attributed to refunds so the books close.

GET /api/admin/diagnostics

Plain-text health report, meant to be read in a terminal or pasted into a bug report. Covers internet reachability and NAT status, hole-punching counts, the last 50 completed requests with their routes and per-segment timings, per-peer serving performance, what this node has served for others, and recent failures including which peer served each one. See Troubleshooting for how to read it.

Network addresses are redacted unless ?full=1 is passed — replaced by a placeholder naming their kind, with transport, port, peer id and /p2p-circuit structure left intact. The default is the safe one because this endpoint exists to be shared: the dashboard's Copy diagnostics button and swarmllm diagnostics both use it, and the report otherwise carries this machine's addresses plus every remembered peer address.

GET /api/admin/performance

The JSON sibling of diagnostics, used by the dashboard's Performance view.

Fetched on demand rather than pushed over the WebSocket. The per-peer and per-request detail here is high-cardinality by nature; it is served when asked for and never retained beyond the bounded rings above.

GET /api/admin/network-map

Geographic distribution of peers and shards across regions. Each entry includes the total peer count for that region, per-model shard-holder counts, per-model request demand rates, coverage gaps (models with zero holders in the region), and per-model replication targets derived from pool size and demand. Includes the local node in its auto-detected or configured region.

Response:

{
  "regions": {
    "US": {
      "total": 3,
      "models": { "tinyllama-1.1b-q4-k-m": 2 },
      "demand": { "tinyllama-1.1b-q4-k-m": 5 },
      "coverage_gaps": [],
      "replication_target": { "tinyllama-1.1b-q4-k-m": 2 }
    }
  },
  "recent_routes": [
    { "model": "qwen2.5-coder-7b-instruct-q4-k-m", "regions": ["TH", "IT"], "ok": true }
  ]
}

recent_routes lists this node's recent requests that left its region: the regions each one crossed, starting here, and whether it succeeded.

GET/PUT /api/admin/config

Read or update daemon configuration. Both need the API key.

PUT changes only the settings the dashboard exposes (contribution level, disk, bandwidth, concurrency, auto-manage, update mode, relaying, LAN trust, …). A key it does not apply is refused with a 400 naming it; for a config.toml setting such as network.enable_upnp or network.external_addresses, the message says which section of config.toml to put it in.

POST /api/admin/config/reload

Re-read config.toml and apply it to the running node. Bearer auth required.

The response separates what took effect from what did not:

• applied — in force now. Most settings are simply re-read by whatever acts on them; a few (concurrency limit, batch size and window, auto-manage interval, session timeout) are pushed to the subsystem that has to resize or retime something.
• restart_required — read once at startup and unchanged by this call. Only max_peers is in this group: connection limits are fixed when the peer-to-peer network is built.

POST /api/admin/shutdown

Gracefully shut down the node. Localhost only, Bearer auth required.

POST /api/admin/api-key/rotate

Issue a new API key and invalidate the old one. It takes effect when the node next starts — the running server keeps the key it started with, and logs that a restart is needed.

Model Management

GET /api/admin/models

List models with shard status, VRAM estimates, and acquisition state. Each model includes:

• mmproj field with available (bool), local (bool), and holders (count) for VLM vision encoder status
• trust_level field: one of "Discovered", "Pinned", "DemandVerified", or "NetworkPopular" indicating the model's trust status (auto-manage only downloads shards for DemandVerified+ or Pinned models, OR for any model where this node already hosts at least one shard — gap-filling exemption). R141: trusted curator publishers (meta-llama, Qwen, mistralai, bartowski, unsloth, etc.) promote at 10k HF downloads instead of 100k

POST /api/admin/models/{id}/add

Trigger model acquisition from the network.

GET /api/admin/models/{id}/status

Check model acquisition progress.

DELETE /api/admin/models/

Remove model (shards + manifest + state).

DELETE /api/admin/models/{id}/shards/

Delete a single shard.

GET/PUT /api/admin/models/{id}/auto-manage

Per-model auto-manage policy (including prune toggle).

GET/PUT/DELETE /api/admin/models/{id}/encrypted-pipeline

Per-model "Start and finish on this computer" setting (encrypted_pipeline). It is decided, first match wins, by: your choice for this model; an explicit global encrypted_pipeline = true; otherwise it is on exactly while this node holds both the first and last part of the model (encrypted_pipeline_auto, default on).

GET returns encrypted_pipeline (in force or not), decided_by (this_model, global_setting or automatic), per_model_override (your choice, or null), default_for_this_model (what it would be with no choice), global_default, automatic_when_both_ends_here, readiness (ready, has_first_shard, has_last_shard) and an overhead note.

PUT {"enabled": true} or {"enabled": false} stores a choice. Turning it on without both ends is recorded, and requests for the model are then refused until the parts are here (fails closed). A warning is logged for 2-part models (fully local, no distribution benefit). PUT {"enabled": null} or DELETE forgets the choice, so the global setting and the automatic rule decide again. A body without enabled is a 400. The choice is persisted and survives restarts.

PUT /api/admin/models/{id}/shards/{index}/lock

Lock/unlock a shard to prevent auto-pruning.

GET /api/admin/models/{id}/pipeline-plan

A preview of how a request for this model would be routed now: the ordered segments and the candidate holders of each range. "No route right now" is an answer, not a failure — 200 with routable: false and a reason carrying the same message / hint a real request would get. Only a model id this node has never heard of is a 404.

GET /api/admin/reference-models

The pinned reference models (docs/REFERENCE_MODELS.md) offered as opt-in downloads, so a speed figure from one machine means the same as one from another.

Storage & Shards

POST /api/admin/rescan-shards

Rescan local shard files on disk and update the model registry and network announcements without restarting the daemon. Useful after manually placing shard files in the data directory. Bearer auth required.

Response:

{ "status": "ok", "models_updated": ["model-id-1"], "count": 1 }

GET /api/admin/models/{id}/metadata

Read parsed GGUF metadata from a locally-stored model header (gguf_header.bin). Returns architecture parameters, tokenizer settings, quantization type, and all raw metadata key/value pairs (tokenizer vocabulary arrays are excluded). Returns 400 if no header file exists for the model.

Response shape:

{
  "model_id": "...",
  "general": { "name": "...", "architecture": "llama", "architecture_supported": true, "file_type": 11, "quantization": "Q4_K_M" },
  "model": { "context_length": 4096, "block_count": 32, "embedding_length": 4096, "head_count": 32, "head_count_kv": 8, "rope_dimension_count": 128, "rope_freq_base": 500000.0, "layer_norm_rms_epsilon": 1e-5, "vocab_size": 32000 },
  "tokenizer": { "model": "llama", "pre": "...", "eos_token_id": 2, "bos_token_id": 1, "padding_token_id": null },
  "tensors": { "count": 291, "data_offset": 131072 },
  "raw": [{ "key": "general.architecture", "value": "llama" }, ...]
}

POST /api/admin/models/{id}/shards/{index}/download

Trigger a P2P download of a specific shard that is not yet held locally. The daemon first checks for P2P peers that hold the shard (picking the best peer by LAN-proximity, latency, and trust), then falls back to returning HuggingFace source info if no peers are available. Bearer auth required.

Responses:

• { "status": "already_local", ... } — shard is already on disk
• { "status": "downloading", "source": "p2p", "peer": "...", ... } — P2P download started
• { "status": "use_hf", "source": "huggingface", "repo_id": "...", "filename": "...", ... } — no P2P peers, use hf/download-shards instead
• 400 if no peers and no HuggingFace source known

POST /api/admin/models/{id}/shards/{index}/unload

Unload a single shard from memory (VRAM/RAM) without deleting the file from disk. Narrows the model's shard window to exclude this shard and restarts the worker subprocess. If this is the last loaded shard, the model is fully unloaded. Bearer auth required.

Response:

{ "status": "unloaded", "model_id": "...", "shard_index": 0, "remaining_loaded": [1, 2] }

POST /api/admin/models/{id}/shards/{index}/load

Load a shard that is on disk into memory. The shard must already be present locally (use /download first if not). Expands the model's shard window to include the shard and restarts the worker subprocess. Bearer auth required.

Response:

{ "status": "loaded", "model_id": "...", "shard_index": 0, "loaded_shards": [0, 1, 2] }

POST /api/admin/models/{id}/unload

Unload an entire model from memory (VRAM/RAM) without deleting any files from disk. Evicts all split-model entries, kills the worker subprocess, clears GGUF metadata cache, and clears the loaded-model record. Bearer auth required.

Response:

{ "status": "unloaded", "model_id": "...", "model_name": "...", "segments_removed": 2, "estimated_freed_mb": 4096 }

GET /api/admin/shard-storage

Per-model storage breakdown, disk and VRAM usage.

GET /api/admin/prune-history

Recent auto-prune events.

GET /api/admin/storage/breakdown

The figures behind the storage bar, in MB: total_mb, used_mb, free_mb, auto_target_mb (the cap on what auto-manage may hold in total, not head-room on top of used_mb), auto_remaining_mb, auto_limited_by, plus held_shards, auto_manage_enabled and the contribution level.

GET/PUT /api/admin/schedule

Resource schedule management.

HuggingFace Integration

GET /api/admin/hf/search?q=...

Search HuggingFace for GGUF models. The parameter is q — query is ignored, and a missing or empty q returns []. Optional tasks=chat,code,vision,multilingual,reasoning narrows the results. Returns results grouped by repository with quantization variants, recommended variant, and VRAM fitness indicator.

Response format:

[{
  "repo_id": "TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF",
  "downloads": 50000,
  "likes": 120,
  "variants": [
    { "filename": "...Q4_K_M.gguf", "size_bytes": 668000000, "quant": "Q4_K_M" },
    { "filename": "...Q8_0.gguf", "size_bytes": 1100000000, "quant": "Q8_0" }
  ],
  "recommended_variant": "Q4_K_M",
  "fits_vram": true
}]

GET /api/admin/hf/probe?repo_id=...&filename=...

Probe a remote GGUF file (size, shard layout).

POST /api/admin/hf/download-shards

Download specific shard indices from HuggingFace. Bearer auth required.

Name exactly one of these, or the request is refused with a 400:

• "shards": [0, 1, 2] — these parts.
• "all_shards": true — every part of the file, worked out from the probe the endpoint runs anyway. What the Models tab's Download sends for a model that fits this computer.
• "peer_fair_share": true — one seed part, chosen by BLAKE3(node_id || model_id) so different computers seed different parts. Other computers with auto-manage on pick up the rest only once the model clears the auto-manage trust gate, so on its own this does not make a model usable.

curl -X POST http://localhost:8800/api/admin/hf/download-shards \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"repo_id": "Qwen/Qwen2.5-Coder-7B-Instruct-GGUF", "filename": "qwen2.5-coder-7b-instruct.Q4_K_M.gguf", "all_shards": true}'

GET /api/admin/hf/source/

Look up the HuggingFace source (repo + filename) for a locally-known model. First checks the in-memory source cache and the probe cache, then auto-discovers by searching HuggingFace if neither has an entry. If found via auto-discovery the result is cached to the database and hf_source.json in the model directory.

Response:

{ "model_id": "...", "repo_id": "TheBloke/TinyLlama-...-GGUF", "filename": "tinyllama-...Q4_K_M.gguf" }

GET /api/admin/downloads

List the download queue with per-shard progress, speed, and source.

POST /api/admin/downloads/{model_id}/cancel

Cancel an in-progress download.

LoRA Adapters

A registered adapter is applied to any chat request that names it in lora_adapter. It runs only on the computer it was registered on, and only for a model that computer holds whole — a request that cannot use it is refused with a 400 saying why, never answered by the base model.

Registration is recorded in <data_dir>/adapters/<id>.adapter.json and survives a restart. If you replace the adapter file, unregister and register it again: a file whose contents changed since registration is refused.

GET /api/admin/adapters

List all registered LoRA adapters with their metadata (id, name, base model, rank, alpha, path).

Response: { "adapters": [ { "id": "...", "name": "...", "base_model": "...", "rank": 16, "alpha": 32.0, "path": "..." } ] }

POST /api/admin/adapters

Register a LoRA adapter from a safetensors file (PEFT naming, …layers.N.self_attn.q_proj.lora_A.weight; F32, F16 or BF16). Bearer auth required. The file must be inside <data_dir>/adapters/. id must be a plain name (no /, \ or leading .); if omitted, a UUID is generated. rank must match the file's matrices.

Request body:

{ "id": "my-adapter", "name": "My Adapter", "base_model": "tinyllama-...", "rank": 16, "alpha": 32.0, "path": "my-adapter.safetensors" }

path may be absolute or relative to <data_dir>/adapters/.

Response: { "status": "ok", "adapter": { ... } }

DELETE /api/admin/adapters/

Unregister a LoRA adapter. Does not delete the file from disk. Bearer auth required. Returns 400 if the id is not found.

Response: { "status": "ok", "message": "Adapter 'my-adapter' removed" }

Cloud Providers

GET /api/admin/providers

List configured cloud providers (name + configured flag, no keys exposed).

PUT /api/admin/providers

Update cloud provider API keys. Bearer auth required. Keys are encrypted at rest.

GET /api/admin/provider-models

List available models from all configured cloud providers. Results are cached for 30 seconds; stale results are returned immediately and refreshed in the background. Includes models from OpenAI, Anthropic (static list), DeepSeek, Mistral, Groq, NVIDIA NIM, Cerebras, SambaNova, Fireworks, Together AI, DeepInfra, and Moonshot/Kimi.

Calling this endpoint also refreshes the routing table that decides whether a model whose id carries no recognisable prefix is proxied to a provider or refused locally. Since v0.3.174 that table is also refreshed on its own every 15 minutes, so a node whose dashboard is never opened still routes its cloud models; before that it was filled only here, and a node that had not been visited refused them outright. A provider that fails to answer a refresh keeps the models it had, rather than having them dropped — a failed fetch used to be indistinguishable from a provider with no models.

Response: { "models": [ { "id": "gpt-4o", "name": "GPT-4o", "provider": "openai" } ] }

GET /api/admin/provider-health

Probe each configured provider by sending a tiny max_tokens=1 inference request (using a suitable test model per provider). All probes run in parallel with a connect timeout.

Response:

{ "providers": [ { "provider": "openai", "status": "up", "latency_ms": 320, "detail": "" } ] }

Status values: up, rate_limited, overloaded, timeout, unreachable, error_<code>.

POST /api/admin/provider-model-status

Probe availability and latency for a list of specific cloud model IDs (up to 20 per request). Sends a max_tokens=1 request to each model's provider endpoint. Anthropic models are skipped (no cloud proxy probing). Needs the API key, like every admin endpoint.

Request body: { "models": ["gpt-5.4", "claude-sonnet-5", "deepseek-v4-flash"] }

Response:

{ "models": [ { "model": "gpt-4o", "status": "up", "latency_ms": 210 } ] }

Status values: up, rate_limited, not_found, unavailable, timeout, error.

Claude Subscription (feature-gated)

Requires building with --features claude-subscription. When the feature is not enabled, these endpoints return {"error": "claude-subscription feature not enabled"}.

GET /api/admin/claude-subscription/status

Detect whether the claude CLI is installed and authenticated on this machine. Reads version from claude --version and subscription info from ~/.claude/.credentials.json (read-only).

Response:

{
  "cli_installed": true,
  "cli_version": "2.1.92 (Claude Code)",
  "authenticated": true,
  "subscription_type": "max",
  "rate_limit_tier": "default_claude_max_5x"
}

PUT /api/admin/providers (claude_subscription_enabled field)

Enable or disable the Claude subscription provider. Pass claude_subscription_enabled alongside other provider key updates.

{ "claude_subscription_enabled": true }

When enabled, claude-* model requests are routed through the local CLI subprocess instead of the Anthropic API key. The Anthropic API key (if configured) is used as fallback when disabled.

Updates

GET /api/admin/version

Current binary version info.

POST /api/admin/update/check

Check for available updates. Returns version info and changelog if update available.

POST /api/admin/update/apply

Download and apply an update. Bearer auth required.

Discovery

GET /api/admin/network-code

Get an encrypted shareable invite code and network phase. The code embeds the node's TCP listening address encrypted with ChaCha20Poly1305 — the IP is not visible in the code.

POST /api/admin/join-network

Join the network via encrypted invite code (swarm://...) or raw multiaddr. Immediately dials the peer and saves the address to the peer cache.

GET /api/admin/foreign-pool-catalog

Models other pools advertise — discovery only, and only when pool.share_model_catalog is on. Using them is a separate opt-in, pool.allow_cross_pool_inference (settable live through PUT /api/admin/config).

Responses API listing

GET /api/admin/responses

List stored Responses-API records (backs the dashboard's Responses tab). Optional query params: ?limit=N (cap on returned records, default 100, max 500) and ?status=... (filter by completed / in_progress / cancelled / failed / queued). See Responses API for the user-facing surface.

Auto-Manage Surfaces

GET /api/admin/swarm/capacity

A snapshot of the swarm's collective capacity: online nodes, total graphics memory and disk, GPU nodes, regions represented, which models it can serve, which it could with more holders, which this node hosts, and their redundancy.

GET /api/admin/swarm/capacity-plan

What-if scenarios over that snapshot, with a headline target naming how many more contributors a model needs.

GET /api/admin/wishlist

R111 — ranked list of models the swarm wants. Triggers a wishlist refresh on every call so the response always reflects current state. Response shape:

{
  "computed_at": 1727712345,
  "entries": [
    {
      "model_id": "mistral-7b-instruct-v0.3-Q4_K_M",
      "display_name": "Mistral 7B Instruct v0.3",
      "status": "serveable",
      "score": 78,
      "swarm_replicas": 4,
      "target_replicas": 3,
      "size_mb": 4368,
      "vram_required_mb": 5023,
      "shards_covered": 8,
      "total_shards": 8,
      "hosted_by_us": false,
      "why_tags": ["wishlist.why.popular_on_swarm", "wishlist.why.fits_your_memory"]
    },
    {
      "model_id": "hf-candidate:bartowski/Llama-3.2-3B-Instruct-GGUF",
      "display_name": "Llama-3.2-3B-Instruct-GGUF",
      "status": "candidate",
      "score": 65,
      "hf_repo_id": "bartowski/Llama-3.2-3B-Instruct-GGUF",
      "task_tags": ["chat"],
      "why_tags": ["wishlist.why.popular_on_hf", "wishlist.why.trusted_publisher", "wishlist.why.candidate_one_click"]
    }
  ]
}

status is one of hosting / serveable / aspirational / candidate (R141) / unreachable / blocked. Candidate entries carry an hf_repo_id field and a synthetic model_id (prefix hf-candidate:) so the frontend can route the click to the HF browse without colliding with real model_ids. See Wishlist architecture for the score formula and trust-promotion path.

GET /api/admin/hf/trending

R112 — most recent HF trending GGUF snapshot from the background HfWatcher (refreshed hourly). Response shape mirrors HfTrendingSnapshot with entries: [{ repo_id, downloads, likes, task_tags, created_at_secs }] and fetched_at: <unix-secs>. Empty entries means the watcher hasn't completed its first poll yet (30s startup delay) OR the daemon is configured with auto_manage.hf_watcher_enabled = false.

GET /api/admin/quant-recommendations

R133 — per-family quant-choice recommendations. Returns the highest-quality variant that fits the swarm's aggregate VRAM with reasonable replication for each model family. R141 flipped auto_manage.auto_switch_quants to default true, so this surface now drives automatic trust promotion of the recommended variant — the UI uses it as a transparency view rather than a manual action surface.

Authentication

GET /api/admin/api-key

Retrieve the API key. Bearer auth required.

The dashboard is the exception: it has no key yet on page load, so it may fetch one here without a Bearer token, provided both of the following hold.

1. The request arrives over a network this node trusts — loopback always, a Tailscale-style overlay when the node is on one too, and a private/LAN address only with api.dashboard_trust_lan. See the [api] section of the configuration reference.
2. It carries the single-use X-Dashboard-Nonce that the daemon substituted into the served HTML for that page load (60s TTL).

A bare curl against this endpoint therefore returns 401 — it has neither. To read the key from a script, use the api_key file in the data directory.

WebSocket

GET /api/admin/ws

WebSocket for live updates. Pushes the following event types:

Claude Subscription Provider

Use your existing Claude Pro, Max, Team, or Enterprise subscription to access Claude models through SwarmLLM — no API key or per-token charges needed.

Feature-gated: Build with --features claude-subscription to enable. This feature is isolated behind a compile-time flag for easy removal.

How It Works

When enabled, SwarmLLM spawns the claude CLI as a subprocess for each Claude model request:

Client Request (OpenAI or Anthropic format)
  → SwarmLLM API (openai.rs / anthropic/mod.rs)
    → Provider resolution: model starts with "claude-"
      → Claude subscription enabled? → Spawn subprocess
      → Else: use Anthropic API key (existing behavior)
    → claude -p --output-format stream-json --model <model> "<prompt>"
    → Parse NDJSON → Translate to API format → Return response

Both the OpenAI (/v1/chat/completions) and Anthropic (/v1/messages) endpoints are supported, with streaming and non-streaming modes.

Setup

1. Install the Claude CLI

npm install -g @anthropic-ai/claude-code

2. Log in with your subscription

claude login

This opens a browser window. Sign in with your Claude Pro/Max/Team/Enterprise account.

3. Build SwarmLLM with the feature

cargo build --release --features claude-subscription

The official downloads do not include this feature, so it needs a build from source.

4. Enable via the dashboard

Open Settings → Cloud Providers → Claude Subscription, click "Check Status" to verify your CLI is detected, then enable the toggle.

Or via API:

curl -X PUT http://localhost:8800/api/admin/providers \
  -H "Authorization: Bearer <your-api-key>" \
  -H "Content-Type: application/json" \
  -d '{"claude_subscription_enabled": true}'

5. Send requests

# OpenAI format
curl http://localhost:8800/v1/chat/completions \
  -H "Authorization: Bearer <your-api-key>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-5",
    "messages": [{"role": "user", "content": "Hello!"}],
    "stream": true
  }'

# Anthropic format
curl http://localhost:8800/v1/messages \
  -H "x-api-key: <your-api-key>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-5",
    "max_tokens": 100,
    "messages": [{"role": "user", "content": "Hello!"}],
    "stream": true
  }'

Multi-Turn Conversations

Multi-turn conversations work by serializing the full message history into the prompt on each request. The format uses XML tags that Claude understands natively:

• System messages → <system>...</system>
• Assistant messages → <previous_response>...</previous_response>
• User messages → bare text

This is the same stateless approach used by OpenAI-compatible APIs — the client sends the full conversation every time, and the server doesn't maintain session state.

Long-lived sessions

For interactive use, where re-sending the whole conversation every turn would be wasteful, the same build also offers sessions backed by one persistent claude subprocess (--input-format stream-json). Like everything on this page, they exist only in a build with --features claude-subscription:

• GET /api/claude-code/sessions — list live sessions
• POST /api/claude-code/session — start one (spawns the subprocess)
• GET /api/claude-code/session/{id} — its state
• DELETE /api/claude-code/session/{id} — close it and reap the subprocess
• POST /api/claude-code/session/{id}/message — send a turn
• POST /api/claude-code/session/{id}/permission — answer a tool-permission prompt the CLI raised

Configuration

All configuration is in the providers.claude_subscription section, manageable via the admin API or dashboard:

Working Directory

By default, the subprocess runs in the system temp directory to avoid loading project-specific CLAUDE.md files, hooks, and MCP servers. Set working_dir to a project path if you want Claude to have project context for its responses.

Routing Priority

When a claude-* model is requested:

1. Claude subscription (if enabled and CLI detected) — subprocess path, uses subscription
2. Anthropic API key (if configured) — direct API proxy, pay-per-token
3. Error — no provider available

The subscription provider takes priority over the API key. Disable the subscription toggle to fall back to API key billing.

Rate Limits

Subscription rate limits are per rolling 5-hour window (not per-second RPM like API keys). The concurrency limiter (default 3) prevents spawning too many concurrent processes. Community reports suggest ~3-5 parallel Opus sessions before degradation.

Rate limit info is returned in the NDJSON output and logged. The GET /api/admin/claude-subscription/status endpoint shows the current rate limit tier.

Removal

If this feature needs to be removed:

git rm src/api/claude_sub.rs
# Remove "claude-subscription = []" from Cargo.toml
grep -rn 'claude.subscription\|claude_sub' src/ frontend/
# Remove the ~6 #[cfg] blocks found by grep

Single commit, clean removal. No deep dependencies on the rest of the codebase.

Identity & Device Pool API

Identity

GET /api/identity/nickname

Get the current node's nickname.

PUT /api/identity/nickname

Set a nickname. Body: {"nickname": "my-node"}

DELETE /api/identity/nickname

Clear the nickname.

GET /api/identity/leaderboard

Network-wide leaderboard. Ranks by shards hosted, not by credits — the credit economy is dormant and no surface publishes a balance (docs/CREDITS_DESIGN.md).

GET /api/identity/peers

Peer identity directory (nicknames, regions, tiers).

Device Pools ("My Devices")

Link multiple devices owned by the same user, so they serve each other privately and the owner can pin models to them. Credit forwarding to the owner's balance still runs underneath, but credits are dormant and nothing shows a balance (docs/CREDITS_DESIGN.md).

Terminology: "My Devices" in the UI (More → My Devices). This is different from connecting to the SwarmLLM network — linking devices groups your own hardware, while the network connects you with other people.

Quick Start (CLI)

# On your main device:
swarmllm pool create --name "My Devices"
swarmllm pool invite-code
# → swarmpool://… (a long code)

# On each other device:
swarmllm pool join 'swarmpool://…'

# Check status:
swarmllm pool status

Invite Code System

Instead of exchanging raw 64-character node IDs, device pools use an invite code: a swarmpool://… link that carries the main device's reachable addresses and a short join token, so the joining device can dial it directly. (A bare 8-character code such as A3F7K2M9 — the token on its own — is still accepted, but only joins when both devices already share a swarm.)

1. Owner generates a code → POST /api/pool/generate-code
2. Code shared verbally, via QR, or copy-paste
3. Member enters code → POST /api/pool/join → broadcasts join request over gossip
4. Owner's node auto-validates code and creates invitation
5. Member auto-accepts → pool established

Security: Codes use a 32-character alphabet (no 0/O/1/I), are one-time use, expire in 24h, and the code itself is never transmitted over the network — only its BLAKE3 hash.

API Endpoints

GET /api/pool/state

Current pool membership state. Returns in_pool and this node's own node_id; in a pool, also is_owner, the member list with device names, online status, per-device stats, and the credit split percentage.

POST /api/pool/create

Create a new device pool. Body: {"name": "My Devices"}

POST /api/pool/generate-code

Generate an invite code (owner only). Returns {"status": "ok", "code": "swarmpool://…"} — the whole link the other device pastes, not the short token inside it. Max 5 active codes.

POST /api/pool/join

Join a pool using an invite code. Body: {"code": "A3F7K2M9"}

POST /api/pool/invite

Invite a specific node by ID (advanced). Body: {"node_id": "abc123..."}

POST /api/pool/accept

Accept a pool invitation. Body: {"invitation_id": "..."}

POST /api/pool/remove

Remove a member (owner only). Body: {"node_id": "..."}

POST /api/pool/leave

Leave the current pool.

POST /api/pool/device-name

Set this device's nickname. Body: {"name": "Gaming PC"}

PUT /api/pool/credit-split

Set credit split percentage (owner only). Body: {"pct": 20} (0-100, the share the member keeps; default 0 = everything is forwarded)

PUT /api/pool/contribution

Set per-member contribution level override. Body: {"node_id": "...", "level": 75} (integer 0–100)

GET /api/pool/invitations

List pending invitations for this node.

GET /api/pool/leaderboard

Pool member contribution rankings.

GET/PUT /api/admin/pools/:id/rates

Per-pool credit rate overrides.

Private Mode

Keep your OWN requests on your device group. By default this also admits other SwarmLLM computers on your local network (pool.private_mode_allow_lan = true); set it to false to keep requests on your own devices only. It does not stop your node serving the swarm.

GET /api/pool/private-mode

Current state + coverage summary. Returns enabled, allow_lan, offline_mode, and coverage object.

PUT /api/pool/private-mode

Toggle private mode. Body: {"enabled": true} or {"enabled": true, "offline_mode": true}. Returns coverage summary so the UI can show trade-offs immediately.

GET /api/pool/coverage

Per-model coverage breakdown: total_shards, pool_shards, coverage_pct, missing indices, est_download_mb. Also returns disk_budget_mb and disk_used_mb.

Shard Pinning

GET /api/pool/pins

List current shard pins.

POST /api/pool/pin

Pin a model to a specific device (owner only). Body: {"model_id": "...", "target_node_id": "hex..."}. Optional shard_indices array for specific shards (empty = all shards).

DELETE /api/pool/pin

Remove a shard pin. Same body format as POST.

Pool Features

• Device nicknames: Name each device for easy identification
• Online/offline status: Tracked via health pings, displayed with last-seen timestamps
• Per-device stats: VRAM, shards hosted, forwards served, uptime, models hosted
• Combined VRAM: Aggregate GPU memory across all linked devices
• Credit split: Owner configures what percentage (0-100%, default 0) members keep vs forward
• Private Mode: Restrict inference to pool devices only. Toggle via UI or API
• Shard Pinning: Assign specific models to specific devices. Auto-manage respects pins
• Offline Mode: Air-gapped LAN operation with mDNS-only discovery
• Coverage Dashboard: Per-model availability bars showing pool shard coverage
• Max 10 devices per pool (configurable), 10 pool operations per hour rate limit

Pool Security

• Invite codes: 32^8 ≈ 1.1 trillion combos, one-time use, 24h expiry
• Join requests signed with Ed25519 (transport-layer sender authentication)
• Credit forwarding uses dual-signed PoolCreditForward (member + owner)
• Member removal requires Ed25519-signed removal notice with replay protection
• Pool state gossip verifies each member's acceptance signature
• Blinded invitation broadcast (SEC-M18): network observers can't see who's invited

Prometheus Metrics

SwarmLLM exposes a Prometheus-compatible metrics endpoint at GET /metrics. It answers without a key only to a request made on the node's own machine, and not even then when api.metrics_auth_required = true; from anywhere else send Authorization: Bearer <api key> (see Monitoring).

Available Metrics

Core Metrics

route is one of local, split, distributed, relayed, cloud; outcome is ok, error, cancelled or pending. Both are closed sets, so this metric is 20 series regardless of how large the swarm grows. Per-peer, per-model and per-shard breakdowns are deliberately not exported here — that label set grows with the swarm and would eventually break the scrape. Fetch them from GET /api/admin/performance instead, which is served on request and retains nothing.

OpenTelemetry GenAI Metrics

Named to the OpenTelemetry GenAI semantic conventions so an OTel collector and the community Grafana dashboards work without a translation layer.

These are the two figures that separate a backed-up queue from slow generation; end-to-end latency alone cannot. swarmllm_inference_latency_seconds is the same measurement as the conventions' gen_ai.server.request.duration under a local name, and both are exported while dashboards migrate.

Only requests that emitted an incremental token contribute to these histograms. A non-streaming path never stamps a first token, so there is no honest way to split decode out of its total and it is omitted rather than counted as zero.

Serving-Side Metrics

Work this node performed for other peers. Every metric above measures requests this node made; these measure what it gave.

rate(swarmllm_segment_serve_seconds_total[5m]) / rate(swarmllm_layers_served_total[5m]) gives seconds per layer served — the figure other peers' schedulers actually rank this node on.

Channel Metrics

Internal channel health metrics for monitoring backpressure:

Emitted for the channels whose send paths actually count: network_out, rebalance and pool_cmd. A channel absent from the output is one nothing increments, which is deliberate — a perpetual zero cannot be told apart from a healthy channel, and an alert on it can never fire.

swarmllm_channel_dropped_total rising is worth an alert, but the log is where the severity is: a drop line carries nothing_accepted_for_secs, and that is what separates a momentary burst from a consumer that has stopped. Anything above a few seconds on network_out means this node is not receiving from the swarm at all (docs/FUTURE_WORK.md #90). The line is rate-limited to one per 30 s per channel and reports how many it swallowed.

Histogram Buckets

swarmllm_inference_latency_seconds uses these bucket boundaries (in seconds): 0.01, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0, +Inf

The two gen_ai_server_* histograms use the boundaries published in the OpenTelemetry GenAI conventions rather than locally chosen ones, so buckets line up with every other GenAI server a collector scrapes: 0.001, 0.005, 0.01, 0.02, 0.04, 0.06, 0.08, 0.1, 0.25, 0.5, 0.75, 1.0, 2.5, 5.0, 7.5, 10.0, +Inf

For every histogram, _count and _sum come from monotonic counters rather than the in-memory sample ring. The ring is both size- and age-bounded, so its length falls when it wraps, which would break rate() and increase().

Scraping Configuration

Add to your prometheus.yml:

scrape_configs:
  - job_name: "swarmllm"
    static_configs:
      - targets: ["localhost:8800"]

Example Queries

# Request rate (requests per second over 5 minutes)
rate(swarmllm_inference_requests_total[5m])

# P50 latency
histogram_quantile(0.50, rate(swarmllm_inference_latency_seconds_bucket[5m]))

# P99 latency
histogram_quantile(0.99, rate(swarmllm_inference_latency_seconds_bucket[5m]))

# Average latency
rate(swarmllm_inference_latency_seconds_sum[5m]) / rate(swarmllm_inference_latency_seconds_count[5m])

# P95 time to first token — is the queue backed up, or is generation slow?
histogram_quantile(0.95, rate(gen_ai_server_time_to_first_token_seconds_bucket[5m]))

# P95 per-token decode cost
histogram_quantile(0.95, rate(gen_ai_server_time_per_output_token_seconds_bucket[5m]))

# Share of requests that left this machine
sum(rate(swarmllm_inference_requests_by_route_total{route=~"distributed|relayed"}[5m]))
  / sum(rate(swarmllm_inference_requests_by_route_total[5m]))

# Error rate by route
sum by (route) (rate(swarmllm_inference_requests_by_route_total{outcome="error"}[5m]))

# Seconds of compute contributed per layer served
rate(swarmllm_segment_serve_seconds_total[5m]) / rate(swarmllm_layers_served_total[5m])

Health Check

GET /health/ready

Readiness probe returning subsystem status. Returns 200 when ready, 503 otherwise. No auth required.

{
  "ready": true,
  "subsystems": {
    "network": true,
    "inference_router": true,
    "api_server": true,
    ...
  }
}

Deployment Guide

Single Node

The simplest deployment — just run the binary:

./swarmllm run

This starts the daemon on port 8800 with default settings.

Production Configuration

For production use, create config.toml in the node's data directory (/var/lib/swarmllm/config.toml for the systemd service below):

[node]
listen_port = 8800
contribution = "maximum"

[resources]
max_gpu_vram_mb = 0        # Auto-detect
max_disk_mb = 100000       # 100 GB

[inference]
gpu_layers = 99            # Offload all layers to GPU
max_concurrent_requests = 20
max_batch_size = 4
session_timeout_seconds = 600

[auto_manage]
enabled = true
max_storage_mb = 50000
max_concurrent_downloads = 5

[logging]
level = "info"             # logs go to the console — under systemd, `journalctl -u swarmllm` —
                           # and to logs/swarmllm.log in the data directory (`file = "off"` stops that;
                           # `format` is not applied yet — see the configuration reference)

[ui]
open_browser_on_start = false

[identity]
region = "US"

Systemd Service

The .deb installs a ready unit (packaging/swarmllm.service in the repository). For a binary you installed yourself, create /etc/systemd/system/swarmllm.service along the same lines:

[Unit]
Description=SwarmLLM P2P Inference Node
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=swarmllm
Group=swarmllm
# Reads /var/lib/swarmllm/config.toml — put the production config there. Dashboard
# saves always write that file, so a --config elsewhere would lose them at the next start.
ExecStart=/usr/local/bin/swarmllm run
Restart=on-failure
RestartSec=10
LimitNOFILE=65536

# Data directory — without these the node would look in the service user's
# home, which ProtectSystem=strict makes read-only
StateDirectory=swarmllm
Environment=SWARMLLM_NODE_DATA_DIR=/var/lib/swarmllm

# Security hardening
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/lib/swarmllm
PrivateTmp=true

[Install]
WantedBy=multi-user.target

# The service runs as its own user; create it once
sudo useradd --system --no-create-home --shell /usr/sbin/nologin swarmllm
sudo systemctl daemon-reload
sudo systemctl enable --now swarmllm

Docker

Quick Start (Recommended)

# Download compose file and env template
curl -LO https://raw.githubusercontent.com/enapt/SwarmLLM/main/docker-compose.yml
curl -LO https://raw.githubusercontent.com/enapt/SwarmLLM/main/.env.example
cp .env.example .env

# CPU
docker compose up -d

# GPU (requires NVIDIA Container Toolkit)
docker compose --profile gpu up -d

Pre-built Images

Versioned tags follow semver — e.g. 0.3.14-alpha, 0.3.14-alpha-cuda (pre-releases publish the full version; stable releases additionally get the 0.3 / 0.3-cuda minor tags).

Manual Docker Run

# CPU
docker run -d \
  --name swarmllm \
  --restart unless-stopped \
  -p 8800:8800/tcp \
  -p 8810:8810/tcp \
  -p 8800:8800/udp \
  -v swarmllm-data:/data \
  -v /path/to/models:/data/models \
  --env-file .env \
  ghcr.io/enapt/swarmllm:latest

# GPU
docker run -d \
  --gpus all \
  --name swarmllm \
  --restart unless-stopped \
  -p 8800:8800/tcp \
  -p 8810:8810/tcp \
  -p 8800:8800/udp \
  -v swarmllm-data:/data \
  -v /path/to/models:/data/models \
  --env-file .env \
  ghcr.io/enapt/swarmllm:latest-cuda

Build from Source

# CPU
docker build -t swarmllm .

# CUDA
docker build -f Dockerfile.cuda -t swarmllm:cuda .

Multi-Node Dev Cluster

For development and testing, a 3-node compose file is available:

docker compose -f docker-compose.dev.yml up

Nodes are at localhost:8800, localhost:8801, localhost:8802. Add GPU support:

docker compose -f docker-compose.dev.yml -f docker-compose.cuda.dev.yml up

Multi-Node Cluster

Same LAN

Nodes on the same network discover each other automatically via mDNS. Just start multiple instances on different ports:

# Node 1
./swarmllm run -p 8800

# Node 2
./swarmllm run -p 8801 -d ~/.local/share/swarmllm-node2

Across Networks

Use bootstrap peers or invite codes:

# Node 1 (get its address from the dashboard or logs)
./swarmllm run

# Node 2 (connect to Node 1)
./swarmllm run --bootstrap "/ip4/NODE1_IP/udp/8800/quic-v1/p2p/PEER_ID"

Split Inference Cluster

For a dedicated split-inference setup across multiple machines:

# Machine A: shards 0-3
./swarmllm run --shards "0-3" --bootstrap "/ip4/MACHINE_B/udp/8800/quic-v1/p2p/..."

# Machine B: shards 4-7
./swarmllm run --shards "4-7" --bootstrap "/ip4/MACHINE_A/udp/8800/quic-v1/p2p/..."

Firewall

Open TCP port 8800 (HTTP API), TCP port 8810 (P2P), and optionally UDP port 8800 (QUIC):

# Linux (ufw)
sudo ufw allow 8800/tcp    # HTTP API
sudo ufw allow 8810/tcp    # P2P (Noise+Yamux, primary transport)
sudo ufw allow 8800/udp    # P2P (QUIC, optional)

# Linux (iptables)
sudo iptables -A INPUT -p tcp --dport 8800 -j ACCEPT
sudo iptables -A INPUT -p tcp --dport 8810 -j ACCEPT
sudo iptables -A INPUT -p udp --dport 8800 -j ACCEPT

Reverse Proxy (Optional)

If you want to put the HTTP API behind nginx:

server {
    listen 443 ssl;
    server_name swarmllm.example.com;

    location / {
        # This machine's LAN address — NOT 127.0.0.1 (see below).
        proxy_pass http://192.168.1.10:8800;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_buffering off;       # stream tokens as they are generated
        proxy_read_timeout 1h;     # generation has no server-side deadline
    }
}

Do not proxy over 127.0.0.1. SwarmLLM treats someone at the machine itself as the owner: it hands the dashboard its API key automatically, serves /metrics without a key, and allows updating and shutting the node down. Behind a proxy on the same host, every visitor used to arrive from 127.0.0.1 — so anyone who could reach the proxy got admin access. Pointing proxy_pass at the machine's LAN address makes visitors remote, so they must paste the API key. Leave api.dashboard_trust_lan off for the same reason: with it on, LAN addresses — the proxy's included — are handed the key too.

Versions after v0.3.200 also refuse those privileges to any request that carries a forwarding header (X-Forwarded-For, X-Real-IP, Forwarded, Via, Tailscale-User-Login) or names a host other than localhost, so the example above is safe either way. v0.3.200 and earlier check only the connection's address, and a proxy can be set up to send neither — nginx's defaults do — so keep proxy_pass off loopback regardless.

Note: the proxy only carries HTTP. P2P must still be reachable directly: TCP 8810 and UDP 8800 (QUIC).

Cloud Provider API Keys

To use cloud model fallback, configure provider API keys via:

1. Dashboard: click the gear icon (Settings) and open Cloud Providers
2. Environment file: Place a .env file in the data directory with standard variable names:

# ~/.local/share/swarmllm/.env
OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...
DEEPSEEK_API_KEY=sk-...
MISTRAL_API_KEY=...
GROQ_API_KEY=gsk_...
NVIDIA_NIM_API_KEY=nvapi-...
CEREBRAS_API_KEY=...
SAMBANOVA_API_KEY=...
FIREWORKS_API_KEY=...
TOGETHER_API_KEY=...
DEEPINFRA_API_KEY=...
MOONSHOT_API_KEY=...

3. Shell environment: Export the same variables before starting the daemon

Performance & Inference Speedups

SwarmLLM's distributed inference path ships with a stack of optimisations that are on by default — you get them without touching a config. This chapter names each one, explains what it does, and shows the measured win so you can tell which levers matter for your workload.

A few are flag-gated because the win is workload-dependent or the path is still being hardened; those are documented at the bottom so you can turn them on intentionally.

The full design notes live in docs/plans/archive/distributed_inference_speedup.md with benchmark recipes in docs/plans/benchmarks/.

The default-on stack

Continuous batching

Concurrent /v1/chat/completions requests for the same model share one forward pass per decode tick instead of running serially. GPU builds use a fused forward_batch kernel; CPU workers fall through to sequential with no regression.

• Measured: 1.34–1.55× GPU throughput at batch 2–8 on RTX 3070 + TinyLlama Q4
• Config: inference.continuous_batching = true (default)

Remote-generate fast path

For single-segment distributed inference (the common case: one remote node owns the whole model, requester does embedding + sampling), skip the per-token coordinator round-trips and run the decode loop end-to-end on the remote worker. Tokens stream back as they're sampled.

• Measured: 1.93× decode speedup
• Config: default-on — no flag, triggered automatically on single-segment pipelines

Cross-request prefix cache

Each worker keeps an LRU cache of prefill KV snapshots keyed by the prompt's token prefix. A re-submission with the same system prompt (different user turn) skips prefill for the shared prefix and only forwards the suffix.

• Measured: 29.4× wall-clock speedup on re-submission of the same 513-token prompt (single node, TinyLlama)
• Config: inference.prefix_cache_enabled = true (default), inference.prefix_cache_block_tokens = 64 (default — block granularity), inference.prefix_cache_max_entries = 16 (default — per model)

Batched prefill + chunked prefill

Sarathi-style chunked prefill: a long admission advances by prefill_chunk_tokens (default 128) per decode tick, so new requests don't wait behind a full prior prefill. Phase 4 adds batched_prefill_forward = true (default), which fuses concurrent same-shape prefill chunks into one forward_batch call.

• Measured (Phases 1+2): 17–23× TTFT fairness at concurrency 2/4/8 on RTX 3070 + TinyLlama Q4 vs serial prefill
• Measured (Phase 4): 1.57× aggregate tok/s at c=4 with uniform 180/180/180 ms TTFT (vs pre-fix 52/235/447 ms spread)
• Config: inference.continuous_batching = true, inference.prefill_chunk_tokens = 128, inference.batched_prefill_forward = true (all default)

Cross-node prefix-KV sharing

Off by default — the one entry here that is: a computer shares its prefix cache only when its owner sets inference.share_prefix_cache_with_peers = true, because announcing it reveals hashes of the prompts it has cached. Your own local prefix cache works either way.

When node B receives a prompt whose prefix was already prefilled by peer A, B fetches A's KV snapshot over the wire instead of re-prefilling locally. The pipeline is:

A prefills → inserts prefix-cache block → gossips PrefixCacheAnnounce
B receives prompt → local cache miss → probe daemon → walk index
B sends SendPrefixKvFetch to A → A's worker exports snapshot
B verifies BLAKE3 + NaN/Inf → hydrates KV → prefill suffix only

• Measured (TinyLlama, GPU-GPU): fetched path is ~100 ms slower than local prefill — the 28 MB f32 snapshot takes ~260 ms to ship while the local prefill it replaces is only ~460 ms. TinyLlama is too small to demonstrate the win on localhost + fast GPU.
• Measured (Qwen2.5-Coder-7B, CPU-CPU): 12.9× TTFT speedup on iter 1 — control full-prefill = 151.7 s, fetched path = 11.8 s. The 73 MB f32 snapshot transfers in ~1 s while 640-token Qwen-7B CPU prefill runs ~150 s.
• Config: inference.cross_node_prefix_trust_min = 0.5 (default — gates peers by trust score; it accepts 0.0-1.0, and a value outside that range stops the node starting). The path runs only between peers that set share_prefix_cache_with_peers = true; leaving that off (the default) takes it out entirely.

The fetch path uses three chained timeouts (worker probe 3000 ms, daemon network 2500 ms, serving IPC 2000 ms) sized for 7B-class f32 snapshots. Missing the window degrades to a clean miss — no worse than not having the feature. See the two-daemon loopback bench recipe for reproduction details.

Parallax scheduler

Pipeline assignment uses shortest-path dynamic programming over observed per-layer latencies (EMA over recent forwards) rather than a greedy pick-the-closest-peer heuristic. Cross-gossip of top-32 observed latencies via NodeCapability.observed_latencies lets every node keep a current view of the network's compute profile. Since 0.3.230 a computer that would first have to load the model is also charged for the load: each node times its own model loads and advertises the rate (NodeCapability.model_load_ms_per_gib), so a warm computer can beat a faster one that is cold. A soft acquire/prune bias in AutoShardManager driven by a per-shard stability counter (≥3 consistent ticks before it acts) drifts shards toward where they're actually used without violating existing hard constraints.

• Measured: covered by routing, allocator and scheduler tests (19 when this was written in April 2026; many more since); real-world improvements depend on network heterogeneity. The biggest impact is in asymmetric setups where a cheap peer's low observed latency should beat a high-VRAM peer's big shard slot.
• Config: default-on. Multi-pipeline concurrency is deferred.

Processor attention kernels

On a computer without a graphics card, attention runs through two purpose-built kernels instead of generic matrix multiplies:

• One position (writing a reply): each cached key/value row is read once for every query head that shares it, in fixed 256-position chunks. Llama-3.2-3B at ~2,080 cached positions: 81.6 → 70.7 ms/token (v0.3.209); the slowdown from a short to a long conversation is close to llama.cpp's.
• Several positions (reading a prompt, a speculative check): tiled over the cached keys so the full score table is never written to memory (FlashAttention's approach). A 6,144-token prompt on Llama-3.2-3B: 43.7 → 51.1 tokens/s, level with llama.cpp on the same threads.

Both use fixed tile sizes, so a result does not depend on how many cores the machine has. SWARMLLM_DECODE_ATTN=standard and SWARMLLM_PREFILL_ATTN=standard switch back to the matrix-multiply path, for comparison.

Optional and tunable features

Some of these are on by default (each says so); turn the others on when you've measured that they match your workload.

Distributed speculative decoding (speculative_distributed)

Draft model proposes γ tokens locally; target verifies all γ in one remote forward pass.

• Status: End-to-end verified. 40–52% accept rate in a llama-cpp-draft / candle-target pairing (cross-backend numerical mismatch caps accept rate).
• Config: inference.speculative_distributed = true, inference.draft_model_path = "path/to/draft.gguf", inference.speculative_gamma = 4 (tokens per verify round)

SWIFT self-speculative decoding (swift_self_speculative)

The target model acts as its own draft by skipping a contiguous range of layers on the proposal pass. No external draft model needed.

• Status: Landed behind flag. Structurally slower than baseline on candle CPU until flash-attn-with-mask lands (attention kernel mismatch on multi-position verify). Shelved on CPU; may help on GPU.
• Config: inference.swift_self_speculative = true, inference.swift_skip_ratio = 0.45 (fraction of layers to skip on the draft pass)

DSD — decentralized speculative decoding (decentralized_spec_decoding)

Multi-segment distributed inference with speculative decoding woven in. A γ-token decode on the last-segment worker plus KV truncation primitives plus a coordinator loop in pipeline/dsd.rs.

• Status: On by default from the release after v0.3.212 (it was off in v0.3.212 while a failover question was open; that turned out to be the test rig, FUTURE_WORK #162, which absorbed #140). It does nothing on a computer that holds no suitable guessing model, and it stops guessing — and, for the next ten minutes, stays out of requests on the same computers — where guessing costs more than the round trips it saves: on two computers side by side it would otherwise have made replies six times slower. Measured across a real link: a model split between a graphics card in Thailand and one in Belgium, with a small model of the same family guessing (Qwen2.5-0.5B for Qwen2.5-Coder-7B), decoded at 4.63 tokens/s greedy and 5.55 at temperature 0.7, against 2.3-2.7 without guessing. Replies are the big model's own: the computer holding the last layers keeps a guess only when its own sample agrees. How many tokens to guess each round is chosen from what a check actually costs, including what each extra guess adds when that computer checks on its processor.
• Checks stream (from v0.3.216): the next few guesses are sent while the previous ones are still being checked, instead of waiting for each check to come back. Measured between Thailand and a computer in Italy holding the far half on its processor (~270 ms apart): 20-57% faster than checking one batch at a time, with replies that match the big model's own choices as closely as plain decoding. Used only when the far computer runs v0.3.216 or later — it treats a reply's stream of checks as one job; older ones keep the one-batch way. SWARMLLM_SPEC_STREAM=0 switches back to batches. ⚠ Only when this computer runs the start of the model and the far computer the end. A computer holding both ends keeps them itself ("Start and finish on this computer", on by default where possible) and one holding none runs no part; both check in batches, as before.
• The guesser is a model this computer already holds, run by SwarmLLM's own engine from its parts — chosen automatically (the largest held model that shares the big model's vocabulary and is at most a quarter of its size), or named with inference.draft_model. A whole model file named in draft_model_path is still used, by llama.cpp, where the build has it. It uses only graphics memory that is free — never the card of the model it guesses for — and runs on the processor otherwise.
• Failure behaviour: if the guesser fails, the reply finishes without guessing; if a check does not come back, the request fails and is retried like any other split request.
• Config: on by default (inference.speculative_decoding and inference.decentralized_spec_decoding); set either to false to switch it off. Optionally inference.draft_model = "<model id>".

Activation compression Q8_0 (activation_compression)

Intermediate pipeline hidden-state activations are quantized from f16 to Q8_0 before going over the wire. Receivers auto-dispatch on the dtype tag.

• Status: On by default. Codec verified: ~3.76× wire compression, RMS error <0.005. End-to-end multi-segment benchmark pending.
• Config: inference.activation_compression = false switches it off

Persistent pipeline stream (persistent_pipeline_stream)

Replace per-token request/response with one long-lived libp2p bidirectional stream per pipeline session.

• Status: Off by default, and no longer faster. Since v0.3.211 an ordinary split negotiates each message without an extra round trip, and with both computers on it request-response decodes as fast as the stream: 2.86 against 2.81 tokens/s on the same Thailand-Belgium split (2026-09-28). The earlier measurement below was taken before that. The first measurement, on loopback, showed no win — a round trip is free there. Across a real link it is the difference that matters: a model split between a graphics card in Thailand and one in Belgium decoded at 1.19 tokens/s over request-response and 2.85 over the stream (same binary, only this setting changed, back and forth). Each request-response message opens a new substream; the stream is opened once per request and peer.
• Any failure on the stream falls back to request-response for that forward, and only the machine coordinating the request needs the setting.
• Why it is still off: in a four-computer failover test a healthy computer never finished reading one prompt sent on the stream, with no error on either side, and the request waited out its full deadline before moving on. Until that is explained, request-response stays the default.
• Config: inference.persistent_pipeline_stream = true turns it on.

Daemon-side STREAM-chunked activation send (R139, streaming_chunked_send)

Split large activation tensors into K = ceil(size / chunk_size) chunks at the wire boundary and ship them sequentially on a single libp2p stream. Each chunk is sealed independently with ChaCha20-Poly1305; chunk_idx + total_chunks are bound into the AAD so reorder / wrong-total / cross-transfer-substitution fail Poly1305 before reaching dispatch. Receiver reassembles via SharedState.pending_activation_chunks before dispatching a single LayerForward to the worker.

• Status: Wire-format, AAD binding, receiver assembly, and sender wiring (persistent-stream path only) shipped in R139. Default off — needs WAN-bench evidence before the default flips. On LAN/loopback the activation send is already sub-millisecond and chunking adds per-chunk fixed cost with near-zero overlap win.
• Requires: persistent_pipeline_stream = true (RR fallback path not yet wired — needs per-chunk Acks).
• Config:
  • inference.streaming_chunked_send = false — master switch
  • inference.streaming_chunk_size_bytes = 262144 — 256 KiB default (age STREAM construction + TokenWeave MLSys 2026 K=2-4 sweet spot)
  • inference.streaming_min_activation_bytes = 65536 — 64 KiB floor; activations below this ship as one frame regardless of the flag
  • inference.streaming_chunk_assembly_ttl_secs = 30 — receiver eviction TTL for stuck assemblies

Encrypt/decrypt offload from event loop (R139 Phase C)

Unconditional benefit (no flag). ChaCha20-Poly1305 sealing in handle_send_tensor and the open in handle_tensor_payload (TENSOR_TAG_ENCRYPTED arm) are offloaded to tokio::spawn tasks so the NetworkManager event loop stays responsive under concurrent decode load. Saves ~50–200µs/forward of event-loop block time on the default RR encrypted path; multiplied across concurrent traffic this is the difference between smooth libp2p ping / gossip / connection handling and observable jitter.

Debugging slow inference

The default log level is info, which already prints the DIAG: request complete summary for every request. -v adds the debug-level DIAG: lines, which include the per-feature speedup signals; -vv adds libp2p internals:

./swarmllm run -v 2>&1 | grep "DIAG:"

Key DIAG kinds:

• DIAG: prefix-cache HIT — local prefix cache hit
• DIAG: cross-node prefix HIT — cross-node prefix-KV fetch succeeded
• DIAG: prefix-probe: fetch timed out — cross-node fetch missed the window (see Troubleshooting for timeout sizing on 7B+ models)
• DIAG: served PrefixKvFetch ... hit=true — this node served a cross-node fetch
• DIAG: BatchGenerate — batched-prefill slot table activity
• DIAG: chunk fused batch_size=N — fused prefill chunks (Phase 4)
• DIAG: Parallax — Parallax scheduler decisions

For the full DIAG taxonomy and what each line means, see docs/DIAGNOSTICS.md.

When should I turn a speedup off?

Almost never. The default-on features degrade cleanly under edge cases — the prefix cache falls through to full prefill on a miss, cross-node fetch falls through to local prefill on a timeout, batched prefill falls back to sequential when concurrency is 1. If you suspect one is the cause of a regression:

• Prefix cache off: inference.prefix_cache_enabled = false
• Cross-node fetch off: inference.share_prefix_cache_with_peers = false on the peers (the default — nothing is offered to fetch). Not cross_node_prefix_trust_min = 2.0: a node refuses to start with a value outside 0.0-1.0
• Continuous batching off: inference.continuous_batching = false (also disables Phase 4 fusion)
• Phase 4 fusion off, keep continuous batching: inference.batched_prefill_forward = false

Please open an issue if a speedup is costing you — the benchmarks above are RTX 3070 + WSL2 + a specific set of models, so real-world workloads will surface corners the benches miss.

Benchmarking

SwarmLLM ships with a built-in bench command and a set of reproducible recipes under docs/plans/benchmarks/. This chapter covers both.

Quick: swarmllm bench

The bench subcommand runs a real /v1/chat/completions workload against a daemon and reports latency + throughput.

./swarmllm bench \
    --max-tokens 100 \
    --iterations 5 \
    --concurrency 1 \
    --stream \
    --model-id tinyllama-1.1b-chat-v1.0.q4-k-m \
    --json

Key flags:

• --max-tokens — tokens to generate per request (default 100)
• --iterations — sequential iterations per concurrency level (default 5)
• --concurrency — concurrent requests for throughput tests (default 1)
• --stream — use streaming chat completions and report TTFT (time-to-first-token) per request. TTFT is the signal that captures the batched-prefill and cross-node-fetch wins; non-streaming bench rolls prefill + decode into one total time and hides the difference.
• --prompt — custom prompt; default is a short prompt about relativity that won't stress prefix caching. Pass a longer prompt (≥500 tokens) to exercise prefix cache paths.
• --model-id — target a specific model when several are registered; otherwise uses the first one from /v1/models.
• --json — machine-readable output

The bench reads the API key from the daemon's data dir, so run it with the same SWARMLLM_NODE_DATA_DIR or -d as the daemon.

Single-node baselines

Reference numbers on an AMD Ryzen 7 5800H + RTX 3070 Laptop 8 GB VRAM (WSL2, release build), measured 2026-04-20 and not re-taken since. Treat them as history: card decode has roughly doubled since (Qwen2.5-Coder 7B whole on this card read 46-49 tok/s in late September 2026, CUDA-graph decode came on by default after that), and processor prompt reading is now level with llama.cpp. Re-take with the method below before quoting any of them.

Single-node numbers are largely about your hardware. The interesting benchmarks are distributed.

Reproducing the performance benchmarks

Each performance optimization has a written benchmark recipe in docs/plans/benchmarks/. Most require two local daemons on loopback; a couple need three.

Batched prefill — TTFT fairness

docs/plans/benchmarks/round4.md

Measures TTFT at concurrency 2/4/8 with Phases 1+2 on vs off. The win is fairness, not aggregate throughput: Sarathi chunked prefill prevents new admits from waiting behind the full prior prefill.

Batched chunked prefill (Phase 4)

docs/plans/benchmarks/round5.md

Measures aggregate tok/s and per-request TTFT spread with batched_prefill_forward on vs off. The on-config fuses concurrent same-shape prefill chunks so TTFT lands tightly clustered instead of spreading.

Cross-node prefix-KV sharing

docs/plans/benchmarks/round6.md

Two-daemon loopback TCP. Measures iter-1 TTFT with the cross-node fetch path enabled vs gated off. Round 6 gated it with cross_node_prefix_trust_min = 2.0, which current builds refuse at startup (the accepted range is 0.0-1.0); gate it off by leaving share_prefix_cache_with_peers = false on node A instead. Sharing is OFF by default, so both nodes need share_prefix_cache_with_peers = true first. Same recipe runs against TinyLlama (fast-GPU corner case: fetch is slightly slower than prefill) and Qwen-7B (12.9× TTFT speedup on CPU-CPU because 7B CPU prefill is slow enough that the ~1 s fetch + verify + hydrate buys back ~150 s of local prefill).

Sketch of the recipe:

# Both nodes must opt in to sharing their prefix cache (default off)
for d in /tmp/swarm_a /tmp/swarm_b; do
  mkdir -p $d && printf '[inference]\nshare_prefix_cache_with_peers = true\n' > $d/config.toml
done

# Node A on 8800 (its log is where B finds A's identity)
SWARMLLM_NODE_DATA_DIR=/tmp/swarm_a ./target/release/swarmllm run -p 8800 -v \
    > /tmp/swarm_a.log 2>&1 &

# Node B on 8900, bootstrapped off A — wait for A to log its own peer id first
until grep -q "Initializing network peer_id=" /tmp/swarm_a.log; do sleep 1; done
A_MADDR=$(grep -oE "Initializing network peer_id=12D3KooW[A-Za-z0-9]+" /tmp/swarm_a.log | \
    head -1 | sed 's/.*peer_id=/\/ip4\/127.0.0.1\/tcp\/8810\/p2p\//')
SWARMLLM_NODE_DATA_DIR=/tmp/swarm_b ./target/release/swarmllm run \
    -p 8900 -v --bootstrap "$A_MADDR" > /tmp/swarm_b.log 2>&1 &

# Copy shards into both data dirs (or download via /api/admin/hf/download-shards)
cp -r ~/.local/share/swarmllm/models/<model-id> /tmp/swarm_a/models/
cp -r ~/.local/share/swarmllm/models/<model-id> /tmp/swarm_b/models/

# Warm A with the long prompt (populates A's prefix cache, announces to B)
./swarmllm bench -p 8800 --stream --iterations 3 --max-tokens 100 \
    --prompt "$(cat long-prompt.txt)" --model-id <model-id>

# Measure B TTFT — iter 1 should fire the cross-node fetch
./swarmllm bench -p 8900 --stream --iterations 3 --max-tokens 100 \
    --prompt "$(cat long-prompt.txt)" --model-id <model-id> --json

Check B's log for DIAG: cross-node prefix HIT — hydrated KV matched_tokens=... bytes=... to confirm the fetch path fired.

Caveats

• WSL2 localhost bandwidth is much higher than any real network — localhost benches are the best case for compute-bound paths and the worst case for fetch paths. WAN numbers will be different.
• TinyLlama is too small to show some speedups — cross-node prefix-KV sharing in particular needs a larger model (Phi-3.5, Qwen-7B) to flip the sign between fetch-cost and prefill-cost. See the round6 benchmark notes for the cross-over math.
• VRAM fit matters — Qwen-7B Q4 weights fit in 8 GB but batched attention kernel scratch does not. CPU-mode works but the baseline numbers above change.
• Pre-warm before measuring TTFT — iter 1 of a model includes disk read + weight load + first CUDA context init; exclude this by pre-warming with a short unrelated prompt before the real measurement.

Standard pre-push gate is cargo fmt && cargo lint && cargo dev-test (the aliases in .cargo/config.toml, all on the development feature set — see CONTRIBUTING.md). If you add a benchmark, add it under docs/plans/benchmarks/roundN.md with the recipe + results + interpretation, and link it from here.

Tailscale & WAN Access

SwarmLLM works over any IP-routable network, including VPN overlays like Tailscale, WireGuard, and ZeroTier. This guide covers how to access your node remotely and connect peers across the internet.

Use Cases

• Remote access — Chat with your home GPU from your laptop at a coffee shop
• Multi-site cluster — Connect nodes at home and work into one swarm
• Team deployment — Share a private swarm across your team without exposing ports to the internet
• Cloud + local hybrid — Connect a cloud GPU instance to your local network

Quick Setup with Tailscale

1. Install Tailscale on all machines

# Linux
curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up

# macOS
brew install tailscale
tailscale up

# Windows — download from https://tailscale.com/download

Each machine gets a stable 100.x.x.x IP address on the Tailscale network.

2. Start SwarmLLM normally

# On each machine — no special flags needed
./swarmllm run

SwarmLLM binds to 0.0.0.0 by default, which includes the Tailscale interface.

3. Connect peers via bootstrap

Since mDNS doesn't work across Tailscale (it's link-local only), use one of these methods:

Option A: Invite code (easiest)

On Node A, open the dashboard (http://localhost:8800) → Computers → Connect a computer and copy Your Swarm Address. On Node B, paste it under Another User's Swarm Address and connect. The invite code contains the node's addresses — including the Tailscale IP if it's listening on 0.0.0.0.

Option B: Bootstrap peers in config

# ~/.local/share/swarmllm/config.toml on Node B
[network]
bootstrap_peers = [
  "/ip4/100.64.0.5/tcp/8810",    # Node A's Tailscale IP
]

Option C: CLI flag

./swarmllm run --bootstrap /ip4/100.64.0.5/tcp/8810

4. Access the dashboard remotely

Once connected via Tailscale, open the dashboard from any machine:

http://100.64.0.5:8800

The API is also accessible at that address:

curl http://100.64.0.5:8800/v1/chat/completions \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "tinyllama", "messages": [{"role": "user", "content": "Hello!"}]}'

Recommended Config for WAN / Tailscale

[network]
enable_mdns = false           # mDNS is LAN-only, won't work through Tailscale
enable_autonat = false        # Tailscale handles NAT, disable noisy probes
enable_dcutr = false          # Hole punching unnecessary on Tailscale
enable_relay = true           # Keep as fallback for robustness
enable_quic = true            # QUIC works well on Tailscale (low-latency UDP)
bootstrap_peers = [
  "/ip4/100.64.0.5/tcp/8810", # Replace with your peer's Tailscale IP
]

For higher latency links (cross-continent), you may also want:

[inference]
tp_max_latency_ms = 50        # Relax tensor parallelism round-trip threshold (default: 5ms)

Binding to a Specific Interface

network.listen_address controls only the P2P listeners (TCP 8810, UDP 8800). The HTTP API and dashboard on TCP 8800 always listen on all interfaces (only --anchor limits them to loopback). To keep the API off your LAN, allow TCP 8800 only on the Tailscale interface with your firewall:

sudo ufw allow in on tailscale0 to any port 8800 proto tcp
sudo ufw deny 8800/tcp

Do not put tailscale serve or Funnel in front of the node. They connect to it over loopback, and SwarmLLM treats a loopback connection as the person at the machine: v0.3.200 and earlier hand that connection the dashboard's API key — with Funnel, that is anyone on the internet. Later versions refuse it to any request carrying Tailscale's Tailscale-User-Login or X-Forwarded-* headers, but reaching the node directly over the tailnet needs no proxy at all.

WireGuard / ZeroTier / Other VPNs

The same approach works with any VPN overlay:

1. Install the VPN on all machines
2. Start SwarmLLM with default config (listen_address = "0.0.0.0")
3. Use the VPN IP as a bootstrap peer address
4. Disable mDNS if peers aren't on the same physical LAN

Security Notes

• API key still required — remote access to inference endpoints requires Bearer token auth, even over Tailscale
• Transport encryption is independent of the VPN — every P2P connection is encrypted (Noise over TCP, TLS 1.3 over QUIC), and activations forwarded between computers are also sealed per hop with X25519 + ChaCha20-Poly1305 (network.enable_encryption, on by default). This protects data in transit only: a computer that works on part of your request decrypts it and sees those numbers, which can be turned back into much of your text. The VPN adds another layer of transport encryption.
• Dashboard requires the API key — admin endpoints need Bearer token auth. The dashboard fetches that key for itself on page load, but only over networks the node trusts (see below)

Opening the dashboard over Tailscale

Admin endpoints need a Bearer token. The dashboard normally obtains one for itself on page load, and the daemon decides whether to hand it over based on the source address of the request:

So if SwarmLLM runs on a machine that has joined your tailnet, browsing to http://100.x.x.x:8800 from another device on the tailnet just works, with no configuration. Set api.dashboard_trust_overlay = false to turn that off — worth doing if you share the tailnet with people you would not give admin access to.

The IPv4 range is shared CGNAT space that some ISPs also use, so it is not on its own proof of a tailnet. Trust is only extended across it when this node holds an overlay address too, which an ISP's CGNAT segment does not give it.

If you reach the node through a subnet router

A subnet router masquerades by default. If Tailscale runs on a Proxmox host (or any gateway) and advertises the subnet a SwarmLLM container sits on, the container does not see your device's 100.x address — it sees the router's private address, which is indistinguishable from any other LAN client. The same is true of a container port publish or any NAT hop.

Two ways through, in order of preference:

1. Unlock the dashboard once with the key. The page will say it wasn't given a key and offer a box to paste one. The key is printed when SwarmLLM starts and stored in the api_key file in its data directory — read it from inside the container. It's remembered per node, so this is a one-time step per browser.
2. Turn on "Allow access from my local network" in Settings → Identity & Access (or api.dashboard_trust_lan = true). This admits any private address, including the router's, and applies immediately without restarting the node. Only do this on a network whose devices you trust — it is a weaker boundary than the tailnet, which at least authenticates its members.

Alternatively, stop the router masquerading so the original address survives, and the tailnet rule above applies unchanged:

tailscale up --advertise-routes=<subnet> --snat-subnet-routes=false

That requires the destination to route back over the tailnet, and is Linux-only.

Inference and the OpenAI/Anthropic APIs are unaffected by any of this — they accept the API key as a Bearer token from any address.

Troubleshooting

Dashboard loads but nothing saves (setup wizard's "Start SwarmLLM" appears to do nothing): every admin call is returning 401 because the page was never given a key. The banner at the top of the page names the address the daemon actually saw for you — which behind a NAT or subnet router is not the address in your browser's address bar — and offers the paste box. See the section above.

Peers don't connect:

• Verify Tailscale is running: tailscale status
• Check that port 8810 (TCP) and 8800 (UDP/QUIC) are reachable: tailscale ping 100.64.0.5
• Try with --bootstrap /ip4/<TAILSCALE_IP>/tcp/8810 explicitly
• Check logs with -vv for connection errors

Slow inference across WAN:

• Pipeline parallelism (splitting layers across nodes) works best on low-latency links (<50ms)
• Tensor parallelism requires LAN-like latency (<5ms round trip) — increase tp_max_latency_ms or let SwarmLLM use pipeline mode instead
• Consider having each site run its own models for local inference, with the swarm as fallback

Stale peer cache after IP change:

• If your Tailscale IP changes, the old cached address just fails to dial; the node keeps saving the addresses of peers it is connected to, so once the two find each other again (through the swarm, or Connect a computer with the new address) the new one is cached too. Do not delete db.redb for this: it also resets your Access Token and unlinks the device from My Devices.

Monitoring with Grafana

SwarmLLM ships with a pre-built Grafana dashboard and Prometheus configuration in the monitoring/ directory.

Quick Start

cd monitoring/
cp .env.example .env   # required: sets the Grafana admin login
docker compose up -d

This starts:

• Prometheus at http://localhost:9090 — scrapes SwarmLLM metrics
• Grafana at http://localhost:3000 — visualizes metrics (log in with the user and password in .env; compose refuses to start without them)

⚠ Known issue: the bundled prometheus.yml scrapes localhost:8800, which inside the Prometheus container is the container itself, so this stack currently shows empty panels. Until that is fixed, run Prometheus directly on the machine that runs the node (see Manual Setup below).

The SwarmLLM dashboard is auto-provisioned on first start.

Dashboard Panels

The Grafana dashboard includes:

Node Overview

• Connected Peers (stat)
• Total Inference Requests (stat)
• Credit Balance (stat — the dormant ledger figure, which gates nothing)
• Shards Hosted (stat)

Inference

• Request Rate (req/s over time)
• Latency Percentiles (p50, p90, p99)
• Latency Distribution (histogram)
• Average Inference Latency (gauge)

Network & Peers

• Connected Peers Over Time

Storage & Shards

• Hosted Shards Over Time

Credits (dormant)

• Credit Balance Over Time — a self-minted ledger figure that gates nothing

Manual Setup

If you already have Prometheus and Grafana running:

1. Configure Prometheus

Add to prometheus.yml:

scrape_configs:
  - job_name: "swarmllm"
    static_configs:
      - targets: ["localhost:8800"]

/metrics answers without a key only to a request made on the node's own machine (and not even then if api.metrics_auth_required = true). A Prometheus running anywhere else — another computer, or a container — must send the node's API key, or every scrape is refused with 401:

scrape_configs:
  - job_name: "swarmllm"
    authorization:
      type: Bearer
      credentials_file: /path/to/api_key   # a copy of the node's `api_key` file
    static_configs:
      - targets: ["node1:8800"]

2. Import Dashboard

1. Open Grafana → Dashboards → Import
2. Upload monitoring/grafana-dashboard.json
3. Select your Prometheus data source
4. Click Import

Multi-Node Monitoring

For monitoring multiple SwarmLLM nodes, add all targets. Every node has its own API key, so a Prometheus scraping several nodes from elsewhere needs one scrape job per node, each with that node's credentials_file (see above):

scrape_configs:
  - job_name: "swarmllm"
    static_configs:
      - targets:
          - "node1:8800"
          - "node2:8800"
          - "node3:8800"

Or use file-based service discovery:

scrape_configs:
  - job_name: "swarmllm"
    file_sd_configs:
      - files: ["swarmllm-targets.json"]
        refresh_interval: 30s

Alerting

Example alert rules for Prometheus:

groups:
  - name: swarmllm
    rules:
      - alert: NoPeersConnected
        expr: swarmllm_peers_connected == 0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "SwarmLLM node has no connected peers"

      # More than 1% of requests slower than 10 s, i.e. p99 > 10 s.
      - alert: HighInferenceLatency
        expr: |
          (
            sum(rate(swarmllm_inference_latency_seconds_bucket{le="+Inf"}[5m]))
            - sum(rate(swarmllm_inference_latency_seconds_bucket{le=~"10|10.0"}[5m]))
          ) / sum(rate(swarmllm_inference_latency_seconds_bucket{le="+Inf"}[5m])) > 0.01
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "More than 1% of inference requests take longer than 10 seconds"

Two traps this rule avoids:

• histogram_quantile(...) > 10 can never fire. The histogram's highest finite bucket is 10 s, and when a quantile lands in the +Inf bucket Prometheus returns that bucket's lower edge — 10 — however slow requests really are. Counting the requests above the bucket is the reliable form.
• Prometheus 3 rewrites le="10" as le="10.0" on ingestion, so a rule matching the integer spelling silently matches nothing there. The regex matches both.