Operations

Running Ecosystem is running an exchange. The parts that need ongoing attention are the engine's placement, five scheduled jobs, gas balances, and backups the platform does not take for you.

Where the matching engine runs

The engine is not a separate service. It boots inside the backend process, and exactly one process in the deployment may own it.

Ownership is decided by a lease keyed ecosystem-matching, held in the engine_lease table and renewed through Redis. A process that loses the race runs a read-only view of the engine — it can price and value, it cannot match.

The placement rule is structural rather than a race, and for a good reason. In the default production layout, PM2 starts backend with CRON_MODE=off alongside a separate cron app with CRON_MODE=only. Order placement is served over HTTP, and placing an order puts it in the serving process's in-memory queue. If the cron process won the lease, the web process could no longer match anything and every order users placed would rest forever. So a cron-only process is refused the lease before any store is consulted — which also means the split stays safe when Redis or MySQL blink at exactly the wrong moment.

Two consequences worth internalising:

• The AI Market Maker must run in the same process as the engine. Its bots enqueue into the matcher's in-memory queue, and an order enqueued anywhere else is simply refused. Its placement is derived from the matcher's, not configured separately.
• Worker threads of one process share a lease by design. A threaded backend is one holder, not several, so a threaded install does not stop matching the moment the database hiccups.

If the ecosystem extension is disabled while ai_market_maker or trading_bot is enabled, the scheduler logs a refusal banner naming the jobs that will decline to run and why. That combination cannot work in any cron layout, because both engines trade ecosystem markets.

Two backend processes

The layout above puts the engine and every other route in one process, so a burst of orders, or a burst of refusals, is served on the same event loop as the login form, the admin screens and every unrelated WebSocket. Measured on the reference box, a ten-times trade burst on that single process moved an unrelated GET's p99 by 5.8x to 32.7x. The fix is a deployment change, not a code path: give the engine a process of its own.

Set ECO_TRADING_ENABLED=1 in .env and restart with pnpm stop && pnpm start. production.config.js then starts four apps:

That one variable is the whole of it. The config file gives each app the scheduling and engine-hosting settings its role needs, and the scheduler goes on running every job exactly as it did before — there is nothing to turn on or off yourself, and a value of your own in .env is what the boot refusals below exist to catch.

production.backend.config.js, the backend-only layout, becomes three apps under the same opt-in: backend (web), trading and cron. Its single all-in-one app cannot simply gain a sibling, because that app is itself a lease candidate and would race the trading app for the engine, so the scheduler needs a process of its own on that host too — again, arranged for you. production.thread.config.js refuses the opt-in outright: worker threads run the matcher without the lease arbitration, and a threaded trading tier would be several matchers with one lease.

Which routes move

The split is a proxy property. Nothing in the code forwards a request from one process to the other, so the reverse proxy config is part of enabling the feature: four prefixes go to the trading port, everything else stays on 4000. The exact blocks are in the nginx and Apache install pages.

/api/ecosystem/wallet, deposit, withdraw, token and chart, everything under /api/admin/ (the Ecosystem and Hummingbot admin screens and the engine health route included), and every non-Ecosystem route stay on the web process. A trading path that reaches the web process by mistake does not fail loudly, which is the trap. A placement written there is still handed to the engine: the order row is the request, and the web process publishes a nudge that makes the leaseholder re-read that symbol's open orders, the same path a follower uses. A cancel is deferred the same way (the drain the leaseholder runs for it is bounded by ECO_CANCEL_DRAIN_PER_TICK). What breaks is the streams: the order-book, ticker and order frames are pushed by the engine to the sockets of its own process, so a socket opened on the web process stays silent. That is why the proxy blocks are not optional.

The lease rule

The ecosystem-matching lease keeps its table, its Redis renewal and its 20-second TTL. What changes is who may ask for it. A process that declares ECO_PROCESS_ROLE=web is refused the lease before any store is consulted, exactly as a CRON_MODE=only process is today; the engine health route reports it as candidate: false, blockedBy: "policy". A process that declares trading is a candidate. A process that declares nothing keeps today's rule, so the default layout without the opt-in still matches on the web tier, and unsetting ECO_TRADING_ENABLED and restarting is the whole rollback.

The AI Market Maker and the trading bots follow the lease candidacy rule already, so they move to the trading process with no setting of their own. On a web-role process their refusal notices still name "the dedicated cron process" as the reason; the behaviour is right, the wording predates the trading role. Copy trading's live path is different: its in-memory queue is started in every process that is not CRON_MODE=only, so it exists in both the web and the trading process, but it is fed by order placements, which the proxy sends only to the trading process, so the copies are produced there and the web process's queue stays empty. The copy-trading backstop job that replicates a leader trade the queue missed stays on the cron process, where it always was.

Failover

The lease decides it, and the lease is unchanged, so the three shapes below are the ones it already had. During any of these gaps the four prefixes answer 502 at the proxy while every other route keeps working; that is the isolation the split exists for.

• The trading app dies and PM2 restarts it on the same host. This is the common path: every deploy and every pm2 restart trading ends the holder without a graceful release. The replacement finds the lease row held by a pid that no longer exists on this machine and reclaims it on its way up, without waiting for the TTL, then hydrates its books from Scylla. The outage is PM2's restart delay plus the boot, about 6.5 seconds to ready on the reference box, plus the hydrate. The reclaim is logged at debug, not as a warning, because it is routine.
• A second trading process sits as a follower. Started by hand with ECO_PROCESS_ROLE=trading, CRON_MODE=off, ECO_TRADING_PORT and NEXT_PUBLIC_BACKEND_PORT all set, it is refused the lease at boot and polls for it once per TTL, running a read-only engine meanwhile. When the leader dies the follower is promoted on its first poll after the row is free. Plan for 25 to 45 seconds at today's 20-second TTL: what remains of the dead leader's TTL (13 to 20 seconds, since the holder renews at a third of it), plus up to one poll interval (the poll is phased from the follower's own boot, so 0 to 20 seconds), plus the hydrate. On the same host the dead-pid reclaim removes the TTL term and the poll phase alone decides. The drill of record, two same-host handovers on the reference box, landed at the fast end: promoted 19.3 and 19.4 seconds after the kill, the inherited book resident 0.3 to 0.5 seconds later, the first crossing order accepted at 19.7 and 20.1 seconds, every fill exactly once. Another run can legitimately land near 40 seconds without anything having changed, so the number to plan with is the band, not the drill's figure.
• Every trading process is down. Nothing matches until one is back. The web process is never promoted, even then; the split would be pointless if it could be. A placement that reaches the web process in that state (it cannot, through a proxy configured as above) is accepted and its hold taken, and the next trading process ingests it at boot.

Orders placed on a follower trading process are handed to the leader through the cross-process path this addon already has (ECO_CANCEL_DRAIN_PER_TICK bounds the cancel half of it); a placement on the web process cannot happen, because the proxy never sends one there.

What to watch

• pm2 list shows trading online. Its exit code 78 means the role refused to boot: ECO_PROCESS_ROLE was found with CRON_MODE not off, or with an unrecognised value, which almost always means the variable leaked into .env. The PM2 configs set it per app; .env must not.
• GET /api/admin/ecosystem/engine/health asked on port 4010 directly (the proxy sends it to 4000) reports role: "trading" and the engine lease as held; asked on 4000 it reports role: "web", candidate: false and blockedBy: "policy".
• The log prefix: every line from the trading process carries a TRAD tag and the process title reads TRAD :4010.
• pnpm stop and pnpm restart own backend, frontend and cron in this release and do not stop trading; run pm2 stop trading alongside, and after turning the opt-in off run pm2 delete trading so a stale trading process does not keep claiming the lease from a layout that no longer expects it.
• The engine flags of the Core environment reference each announce themselves once in this process's log when they reach the engine (Resident book v2 ON for ..., Dirty-symbol cycles ON ..., Cancel cycles SCHEDULED ..., Reconciler from the aggregate ON ...), and scaleFlags on the same health route shows each as configured and effective. A flag set in .env with no announcement and effective: false is a process that cannot hear a kill over the settings bus and has turned its money-affecting flags off; the Core monitoring guide's engine flags section reads the rest of the payload.
• Ledger fence installed at epoch N is what this process prints when it arms with the ledger batcher on; Failed to bump the engine lease epoch; refusing to arm as leader on every attempt means the epoch column is missing from engine_lease (an install that boots with DB_SYNC=none; see the Core 6.7.6 upgrade notes), and no process is matching until it is added.

Scheduled jobs

Enabling the extension registers five jobs, visible and controllable at Admin → System → Cron.

A sixth recovery runs once at boot, before the scheduler starts taking new work: it sweeps every PENDING withdrawal with no age filter, so rows orphaned by a previous process lifetime are picked up in order.

The two withdrawal jobs coalesce onto one in-flight pass, because on every 30-minute boundary both fire at once and would otherwise issue duplicate explorer probes against the same stale rows.

Repair scripts

Six scripts ship for the situations the product cannot fix from a screen. All of them are dry-run by default; each needs an explicit flag to write.

# Order book divergence — ghost, missing and mismatched price levels
pnpm rebuild:eco-orderbook                 # report
node backend/scripts/rebuild-eco-orderbook.mjs ETH/USDT --execute

# Funds locked in `inOrder` with no matching open order
pnpm reconcile:eco-inorder                 # report
node backend/scripts/reconcile-eco-inorder.mjs --apply

# Orders whose funds were never properly locked
pnpm fix:eco-orders

# Duplicate or discontinuous candles, ecosystem and futures
pnpm fix:eco-candles

# The open-orders index against the order ledger
pnpm eco:index:check                       # verify, non-zero exit on drift
pnpm eco:index:repair                      # backfill and prune

# Accumulated AI market-maker orders resting in the book
pnpm eco:mm:orders                         # survey
node backend/scripts/eco-mm-orders.mjs --apply --keep=200

Three rules that come from the scripts themselves.

Restart the backend after any of them apply changes. The engine holds the order book and the open-order queue in process memory; a repaired table and a stale process disagree immediately.

Run eco-mm-orders with the backend stopped. Cancelling underneath a live engine races its settlement — the engine can be filling an order between the script reading it and cancelling it.

reconcile-eco-inorder only ever releases. It never debits a balance and never raises a hold, so a user with genuinely open orders cannot be over-credited. An under-locked wallet is reported, not silently fixed — that is a different script's job.

eco:index:check is designed to be used as a deployment gate: it exits non-zero when the index and the ledger disagree, which answers "is it right?" with a number rather than an opinion.

Backups

What each store actually holds, so you can size the risk:

The .env row is the one people get wrong. A database backup without the file that decrypts it is not a backup of a single wallet. Store them separately, and store the passphrase separately again.

Restart semantics

There is no configuration reload. These all require pm2 restart backend (and pm2 restart trading on an install running the split above):

• any <CHAIN>_* variable — provider instances are constructed at module load;
• SCYLLA_*, REDIS_*, and the vault variables;
• permission grants, because the route gate is held in memory;
• anything a repair script changed underneath a running engine.

Custom EVM chains are the exception: their variables are hydrated from the database into process.env at boot and re-applied when the chain registry reloads, so creating or editing one takes effect immediately.

Disabling the extension

Turning Ecosystem off in Admin → System → Extensions is a real stop, not a UI toggle. The matching engine does not boot, the five cron jobs deregister, the deposit scanners tear down, and the addon's routes and admin nav disappear.

What it does not do is move money. Balances stay, addresses stay, resting orders stay in ScyllaDB, and pending withdrawals stay PENDING — they resume when the extension is enabled again, through the same boot-time recovery.

If you are turning it off to stop a runaway, stop the process rather than only disabling the extension, and confirm the logs go quiet before you walk away.

A routine worth having

• Daily — open /admin/ecosystem. Look at the coverage counts, the stuck withdrawal count, and any chain flagged degraded.
• Weekly — check master wallet gas balances against the chains that actually see withdrawals. Legacy custodial contracts need no gas of their own: the master wallet pays for every token moved out of them. While the custodial mode is drain, sweep any contract still holding tokens to the master wallet. Run pnpm eco:index:check.
• After every deploy — confirm the vault is unlocked if you do not set the passphrase, and confirm the engine took the lease rather than falling back to read-only.
• Before adding a chain — run its diagnostics, and read the readiness rows rather than the green ticks.

Related

• Admin console — the screens these numbers appear on
• Sharding the matching engine — the next step up from the two-process split, when one engine's cycle is the wall
• Deposit wallets — the flows these jobs serve
• Troubleshooting — symptom-first diagnosis