Shard-Only Mode

SwarmLLM supports shard-only operation — a node only needs individual shard files (~512 MB each) plus a small GGUF header (~6 MB), not the full model file.

How It Works

A model directory in shard-only mode:

~/.local/share/swarmllm/models/qwen2.5-coder-7b/
├── manifest.json        # Model metadata + shard layout
├── gguf_header.bin      # First ~6MB of GGUF (metadata + tensor index)
├── shard_000.bin        # 512MB shard
├── shard_001.bin
├── shard_002.bin
└── ...

SwarmLLM automatically extracts gguf_header.bin from shard_000.bin when first needed. The ShardReader constructs a virtual GGUF from header + shard files, so the model parser works exactly as if the full GGUF were present.

Why This Matters

  • A 7B model is ~4.5 GB as a full GGUF, but a single shard is only ~512 MB
  • Nodes only load the layers they're assigned — no wasted disk or VRAM
  • You can help serve a model bigger than your graphics card by hosting just a few of its shards

Manual Shard Assignment (--shards)

For multi-node split inference, assign each node a subset of shards:

./swarmllm run --shards "0-3"    # This node handles shards 0, 1, 2, 3

The range is saved and restored every time the node starts — even if you later start it without --shards. To go back to holding every shard, start once with --shards all.

Behavior when --shards is set:

  • The node only advertises the specified shard indices
  • Auto-manage prioritizes downloading missing shards in the range (100x scoring bonus)
  • Smart pruning never removes shards in the configured range

Multi-Node Example

Run a 7B model across two machines:

# Machine A (shards 0-3, layers 0-13):
./swarmllm run --shards "0-3" --bootstrap "/ip4/MACHINE_B_IP/udp/8800/quic-v1/p2p/PEER_ID"

# Machine B (shards 4-7, layers 14-27):
./swarmllm run --shards "4-7" --bootstrap "/ip4/MACHINE_A_IP/udp/8800/quic-v1/p2p/PEER_ID"

Both nodes discover each other, assemble a distributed pipeline, and forward hidden-state activations between them. The pipeline is assembled automatically by the InferenceRouter.

Without --shards

If you have never used --shards (or cleared it with --shards all), the node auto-detects and advertises all local shards. This is the normal mode for most users — --shards is only needed when you want explicit control over which layers a node handles.