Sharding the matching engine

One matching engine is one process, and one process is one CPU core for the part that matters: the matching walk runs under a lock, so it cannot use a second core no matter how many the box has. The shard tier splits the symbols across several such processes, each with its own book, its own lease and its own log, and puts a door in front that sends every order to the process that owns its symbol.

When this is worth doing, and when it is not

Shard when one engine's cycle is the bottleneck — the matching walk, not the database. The signals are on the engine health route: a lastCycleMs that grows with resident orders, an event-loop delay that tracks it, and placement latency that falls when you cancel resting orders. On the reference box a single engine sustains roughly 70 placements and 70 cancels a second, and the placement rate halves once a single market holds about 5,000 resting orders.

Do not shard because the site feels slow. Almost every "the exchange is slow" report is one of the things on Operations instead: the engine sharing a process with web traffic (fix that first — it is one variable, ECO_TRADING_ENABLED, and it costs nothing), a database without enough redo log, or wallet writes queuing behind a row lock. Sharding a deployment that is actually database-bound moves the bottleneck nowhere and gives you N times the processes to operate.

A useful order of escalation:

1. Split web from trading — ECO_TRADING_ENABLED. One variable, one proxy change, no new failure modes. See Two backend processes.
2. Tune the database — redo log, buffer pool, innodb_io_capacity.
3. Batch the ledger — ECO_LEDGER_BATCHER, which turns a commit per hold into a commit per tick.
4. Then shard, if the cycle is still the wall.

The pieces

A shard takes no HTTP traffic. It listens on a loopback port (ECO_SHARD_PORT_BASE + id) and the door reaches it there. Everything that arrives has already been through the door's session, API-key and permission checks, which is why that port carries no authentication of its own — and why it must never face a network.

How a symbol finds its shard

By hash: the symbol's FNV-1a modulo ECO_SHARDS. Every process computes it the same way from the same number, which is why every process must carry the same ECO_SHARDS. A signed override file (ECO_SHARD_MAP_FILE) pins named symbols to named shards; it is how you drain a market before changing the count, and it is signed because anyone who can rewrite it can point a live symbol at a shard that is not matching it.

What makes a crash survivable

Each shard writes a write-ahead log under ECO_WAL_DIR/shard-<id>, and the rule it never breaks is that a record is on disk before the money it describes moves. An intent is durable before a hold is submitted; a hold is committed before its order can match; a fill is durable before its ledger legs are submitted. On restart the shard replays the log, re-verifies against the transaction table what the log could not confirm, and re-submits exactly what the ledger does not already hold.

That re-verification is unconditional — there is no switch for it — which is what makes innodb_flush_log_at_trx_commit=2 survivable on a sharded install: MariaDB may lose up to a second of commits on a power cut, and the replay finds out which ones by asking the ledger rather than trusting the log.

One lease per shard

Every shard takes its own lease, keyed ecosystem-matching-shard-<id>, so shards never contend with each other or with the single matcher of an unsharded install. On every arm it bumps that row's epoch and installs it as a fence on its own ledger batcher, so a shard that lost its lease and has not noticed cannot commit past the epoch: its next tick aborts, and it refuses everything it was holding until it re-arms.

Turning it on

Do this on a maintenance window, in this order. Steps 1 to 3 change nothing a customer can see — the shards are running and idle, and every order still goes through the existing engine — so you can stop after any of them.

1. Give the shards a log directory.

ECO_WAL_DIR="/var/lib/bicrypto/wal"

Create it, and make it writable by the user PM2 runs as. Check that it is on real storage.

2. Declare the count and start the shard apps.

ECO_SHARDS=2

production.config.js reads it and adds one shard-<id> app per id, between the backends and the frontend. pm2 start production.config.js brings them up; pm2 logs shard-0 should show it take its lease, bump its epoch, open its log and start listening. It is idle: nothing is sending it orders yet.

3. Confirm every process agrees. ECO_SHARDS must be identical in .env and in every app's environment. The shard apps get it from the config file; the door reads it from the environment. A mismatch here is the one failure mode that is not caught for you at boot, because each process on its own is internally consistent.

4. Open the door.

ECO_DOOR=true

This is the step customers can see. From here, placements, cancels, cancel-all and the Hummingbot order reads go to the shard that owns the symbol. A shard that does not answer is a 503 with Retry-After: 1, and the correlation id the door mints makes that retry safe — the same request cannot hold twice.

5. Turn the projector on so what customers read stays current:

ECO_PROJECTOR=true

Without it the shard matches correctly and ScyllaDB goes stale, so the order list and the book lag behind reality. The door reads through to the shard for a single order, which covers the window but not the lists.

Turning it off

In reverse, and the first step is the whole rollback: unset ECO_DOOR and every route places locally again through the ordinary engine. Do that before stopping any shard app, or in-flight orders meet a door pointing at a process that is going away. Let the shards drain, stop them, then unset ECO_SHARDS.

A shard's log is not disposable while it holds unconfirmed work. Stop a shard cleanly (pm2 stop shard-1) rather than killing it, and keep the directory until you have confirmed it restarted clean at least once.

What to watch

GET /api/admin/ecosystem/engine/health on a door process now carries a shards array — one entry per shard, asked over the same transport the door places orders on, in parallel, so a dead shard costs one deadline and not N. On a process that is not a door it is null, which means "this process is not a door", not "the shards are down".

When something is wrong

A symbol stops matching, fillsRefusedAtApply went up. The shard decided a fill its own counters said was funded and the ledger refused it. The shard halts that symbol deliberately rather than guess. The refusal is in the shard's log with the group id; resolve the underlying wallet or pool state, then restart that shard, which replays and re-submits.

Every order returns 503 with Retry-After. The door cannot reach a shard. Check pm2 list for a shard-<id> that stopped — exit code 78 means it refused to start, and the reason is the first line in its log (almost always ECO_WAL_DIR or a shard id outside the count). Clients retrying with the same correlation id are safe.

A shard exits 78 in a loop. It does not loop: 78 is in stop_exit_codes, so PM2 stops it and leaves it stopped. That is deliberate — a shard that cannot start correctly must be visible, not quietly restarting.

Two shards claim one symbol. They cannot both hold the same lease, so this shows up as a symbol whose orders land on a shard that is not the one holding the book: a placement succeeds and nothing matches. The cause is always a disagreement about ECO_SHARDS or a stale map file. Stop the door first, get the numbers to agree, then restart the shards.

The order list is stale but the book is right. The projector is off or behind. Check ECO_PROJECTOR and the projector block in the shard's health.

What this does not change

• The rows written to the ledger are the same rows, with the same idempotency keys, amounts and descriptions, whether an order was matched by a shard or by the single engine.
• The HTTP replies are the same replies. A shard's refusal reaches the caller with the shard's own status code and words.
• The repair scripts on Operations still apply, and still work against ScyllaDB and the ledger rather than against a shard's memory.

Related

• Operations — where the engine runs, the two-process split, the cron jobs and the repair scripts.
• Environment variables — every variable on this page, with its default and range.
• The order desk — reading and repairing individual orders.