Troubleshooting

This is the designed behaviour when ScyllaDB is unavailable. Wallets, deposits and withdrawals do not depend on Scylla; orders, the book and candles do.

Check three things in order. Is SCYLLA_ENABLED set to "false"? Is the node running (systemctl status scylla-server)? Does the backend log show ScyllaDB unreachable at … at boot?

The backend probes the first contact point with a short TCP check before it retries, so an absent node costs a warning rather than a minute of exponential backoff. If the port is open but CQL is not yet serving, you will instead see connection retries with doubling delays and then "Max retries reached".

Also check SCYLLA_DATACENTER. It must match the datacenter name the node reports, not a name you chose, and a mismatch fails the connection rather than degrading it.

The vault is locked. Either ENCRYPTION_KEY_PASSPHRASE is not set and nobody has unlocked it since the last restart, or ENCRYPTED_ENCRYPTION_KEY is missing or malformed.

Open /admin/ecosystem. The masthead shows Initiate Vault when locked — enter the passphrase there. If the badge is green and you still see this error, the process serving your request is not the one that was unlocked: on a multi-process deployment each process holds its own decrypted key in memory.

Check the format too. ENCRYPTED_ENCRYPTION_KEY must be exactly four colon-separated hex parts. Three parts, or a truncated copy-paste, will not decrypt.

Crediting is done by one job — verifyPendingEcoDeposits, every 60 seconds. Check Admin → System → Cron that it is registered and running. If the whole addon's job group is absent, the extension is disabled or the cron process did not pick up the change.

If the job is running, the transaction has not reached confirmation depth. Required depth is 3 for Bitcoin, 6 for Litecoin, Dogecoin and Dash, and 12 by default for anything that does not specify its own.

If neither, look for the pending record. Between detection and crediting the transaction lives in Redis, not MySQL. A Redis flush in that window loses the pending record — the coins are still at the user's address and will be found again on the next scan of it, but nothing will happen until that scan runs.

Finally, check that the token is active and its network matches the chain's configured network. A token whose network no longer matches is skipped when addresses are issued.

Deposit addresses are generated for the network configured at the time. If BTC_NETWORK was flipped after users had addresses, those addresses belong to a different chain and are invalid — they must be regenerated, and anything sent to them is on the old network.

The same applies less dramatically to Solana: SOL_NETWORK set to anything other than mainnet or testnet silently falls through to devnet, and SOLANA_NETWORK is a separate key that only stamps metadata. If the two disagree, the stored record lies about which cluster the address is on.

Older Tron wallets are a special case that self-heals: a Tron entry holding a 0x… EVM address predates proper Tron derivation and is regenerated automatically on the next fetch. Seeing that in the logs is the fix working, not a fault.

Work through these in order.

Is the vault unlocked? Signing needs key material.

Does the chain have a master wallet, and is it enabled? The diagnostics downgrade withdrawal readiness to failed for either condition, and a disabled wallet behaves exactly like a missing one.

Did the queue lose the row? The queue is in-memory and the debit is durable, so a restart between the two strands the row. Three recovery passes exist — one at boot with no age filter, a watchdog every 5 minutes for rows older than 3 minutes, and a legacy 30-minute pass. If a row is older than that and still PENDING, it is in the set the recovery deliberately refuses to touch, which is exactly the count the admin overview breaks out.

Is it a token withdrawal on a chain with no gas? The master wallet pays the gas on-chain, in the chain's native asset, for every token mover — and for a token moved out of a draining legacy custodial contract too. The customer repays it in the withdrawn token afterwards, but the coin has to be there first. Native withdrawals still succeed in this state, which is why the failure looks selective.

A PROCESSING row that already carries a transaction hash is never re-broadcast — recovery promotes it to COMPLETED, because the funds have irreversibly left. That is correct, not a bug.

The full message names the amount, the token and the chain, then lists what was checked: the customer's own address, other customers' addresses, the master wallet, and — while the custodial mode is drain — the legacy custodial contracts. Every candidate is checked on-chain before it is chosen, so this is the engine saying that nothing it is allowed to pay from actually holds the token, whatever the platform's own records say. The customer is not debited.

Two causes account for nearly all of them.

Legacy credit sitting in a custodial contract, with the mode set to off. A wallet whose balance was earned under the old shared-deposit model has its coins in a contract, not at any address. With ecosystemCustodialMode on off those contracts are not a source. Open Admin → Ecosystem → Settings → Custodial wallets: if contracts still exist, either switch back to drain or sweep each contract to the master wallet, after which the master wallet is a source in either mode.

A recorded balance the chain does not back. wallet_data.balance says an address holds the token and the explorer says it does not — a deposit credited from a transaction that was later reorganised, or an address that was paid away from outside the platform. Compare the address on the explorer, then check the private ledger at Admin → Ecosystem → Wallets → Ledgers for that wallet. That is a coverage problem, and the overview will be reporting the asset as short of cover.

Admin → Ecosystem → Settings → Diagnostics → Chains refusing EIP-7702 lists every chain whose node rejected a type-4 transaction, with the time it happened. The log carries the same event under ECOSYSTEM: <CHAIN>:<network> refused an EIP-7702 transaction; using the fallback mover for 24h.

This is not a fault. The chain is either not past the Pectra upgrade, or its RPC provider does not relay type-4 transactions yet. For 24 hours every token withdrawal on that chain uses the fallback — permit for a PERMIT token, otherwise just-in-time gas, which is two transactions and costs the master wallet more — and then the next withdrawal probes again. The entry clears itself the first time a type-4 transaction is accepted.

Only the vocabulary of a "transaction type not supported" refusal is treated this way. A revert, an insufficient-funds error or a nonce clash is reported as the failure it is and never puts a chain on this list — so a chain that keeps failing withdrawals without appearing here has a different problem, and the log line under ECO_WITHDRAW names it.

If you know a chain will never accept type-4 transactions and want to skip the daily probe, set ecosystemEvmMover to jit_gas under Settings → Withdrawals. That applies to every chain, so it is the right answer only for an install whose chains are all in that position.

With ecosystemChargeNetworkFee on, a token withdrawal's estimated gas is converted into the withdrawn token through the platform's USD rates and charged to the customer. When either side of that conversion has no price, the engine logs under ECO_WITHDRAW:

No USD price for <symbol> on <chain>; network fee skipped — set fee.network on the <currency> token or list <native> on a market

and sends the withdrawal anyway, with the master wallet absorbing the gas. A token that cannot be withdrawn is a worse outcome than one unrecovered fee, so this is a warning rather than a refusal — but every such withdrawal costs you the gas.

Two fixes, and the message names both. Set the token's fee.network on its edit screen at Admin → Ecosystem → Trading → Tokens — a flat amount in token units that is charged whenever the priced estimate is unavailable. Or give the unpriced side a price: list the chain's native coin, or the token, on a market. A Gas price unavailable on <chain> line in the same place is the RPC failing to quote a fee at all; that is the chain's RPC, not pricing, and the diagnostics for that chain will show it.

Three known shapes of this.

The Arbitrum typo. ARBIRUM_MAINNET_RPC, missing the second T, is read by the admin balance endpoint and the legacy health check but never by the real provider path. Set only that and health says Up while deposits and withdrawals are broken. Set ARBITRUM_MAINNET_RPC.

Keys that used to be display-only. SOL_<NETWORK>_RPC and XMR_<NETWORK>_RPC (each with a _FALLBACK sibling) are read by the services now — a comma-separated list forming a failover pool, replacing the public cluster endpoint and XMR_DAEMON_RPC_URL respectively. The pool does not walk the list in order: it prefers an endpoint it has never tried, then the fastest one measured, so any entry can serve any query and one bad daemon in the list is not harmless. The Monero diagnostics therefore probe every entry. The overview badge still only checks that the key is present, which proves nothing about reachability; the Solana test probes the public cluster endpoint.

Explorer coverage gaps. Etherscan's free tier is paid-only for BSC, OP Mainnet, Base and Avalanche, and Fantom, Cronos and HECO are not on V2 at all. Only three of those have a keyless instance the provider chain falls back to on its own — OP Mainnet and Base on hosted Blockscout, Avalanche on Routescan. BSC, Fantom, Cronos, HECO and Polygon Amoy have neither a hosted Blockscout instance nor Routescan coverage, so on those chains no keyless provider is appended to the order. BSC is the one where a key is the normal answer for a production install: set NODEREAL_API_KEY for mainnet, which is free for BSC, or MORALIS_API_KEY / COVALENT_API_KEY for testnet, the only two that index chain 97. Without one, transaction history and native-deposit detection fall back to an RPC block scan, which is slower and only walks forward from now.

<CHAIN>_NETWORK holds a value that is not one of that chain's valid network names, so the <CHAIN>_<NETWORK>_RPC key it derives does not exist.

The classic is Polygon: its mainnet is called matic, not mainnet. Also check that the RPC you configured actually serves the chain ID the network expects — the diagnostics report a mismatch explicitly.

Remember that a fresh value needs a restart. Provider instances are constructed at module load and nothing re-reads .env.

The matching engine boots inside the backend process, and exactly one process may own it. Order placement puts the order in the serving process's in-memory queue, so if the wrong process holds the lease, orders rest forever.

A dedicated cron process is refused the lease structurally, precisely to prevent this. If you are running a custom process layout, check the engine_lease table and the boot logs for a message about another process holding the lease.

The same rule is why the AI Market Maker must run in the engine's process: its bots enqueue into that same in-memory queue, and an order enqueued anywhere else is refused.

The market has no precision.amount and precision.price configured, and order placement refuses rather than guessing. Edit the market and set both, along with the amount, price and cost limits.

This is the usual cause of a newly created market that accepts nothing — the market row is created before its metadata is filled in.

The symptom is specific: the book displays depth at a price, the customer's Open Orders tab does not list it, and both cancelling it individually and Cancel All leave it exactly where it was. It usually appears after a restart.

Check the order index first — not the book. Every list in the UI reads the orders ledger, so an order the ledger has already cancelled or deleted is correctly absent from all of them. The matching engine reads a different table: open_orders_by_market, the book-ordered index. A row left there with no OPEN order behind it is loaded at every boot as a live resting order, and the five-minute reconciler then rebuilds the aggregated level to match it. The depth is real to the engine and invisible to every cancel door the customer has.

pnpm eco:index:check                       # read-only; names the markets that disagree

A market reported with STALE=n is this case. The repair:

pnpm eco:index:repair                      # removes the stale rows

Then restart the backend. The engine is still holding those orders in memory, and until it reloads it will keep the level standing.

From 6.6.4 the server does this for itself: it compares both counts at boot, and one market an hour while it runs, and rebuilds any market whose index holds more rows than the ledger has open orders — before a single order is loaded, so one restart clears both the index and the depth it was producing. eco:index:check stays the way to confirm it, and the only way to see the state on a version that does not yet self-heal.

Two things this is NOT, and both look identical from the trade page: a genuine ghost level (below), and funds still held for an order that no longer exists (further down).

Three ways the aggregated book can diverge from the orders table: a level with no backing open order (ghost), an open order with no level (missing), and a level whose amount does not equal the sum of remaining at that price (mismatch).

pnpm rebuild:eco-orderbook                                 # report
node backend/scripts/rebuild-eco-orderbook.mjs ETH/USDT --execute

Run pnpm eco:index:check before this one. This script rebuilds the book from the orders ledger, which is the right authority — but the running engine rebuilds it from what it loaded out of open_orders_by_market, so if that index is holding rows the ledger disowns, the level you delete here comes straight back. Repair the index, restart, then rebuild the book if anything is still wrong.

It is non-destructive: orders, trades, candles and user funds are untouched. It also reports ecosystem levels stranded in the futures keyspace by an older defect where unqualified writes landed in the wrong keyspace.

Restart the backend afterwards. The engine holds the book in process memory and a repaired table plus a stale process disagree immediately.

Symbols with an active AI market maker are skipped by default — that book is maintained without real order rows, so rebuilding from orders would wipe its quotes. Pass --include-ai to override, having stopped the market maker first.

The backend's own five-minute sweep is more discriminating than this script, and from 6.6.4 more so again. It identifies the market maker's own levels individually and reconciles only the rest, so an AI market is normally swept without touching its quotes. Where it cannot tell them apart — a market whose maker is configured ACTIVE while none of the levels are recognisably its own — it now waits for two consecutive quiet sweeps and then treats the book as customer depth, on the grounds that a maker which is quoting rewrites its quotes continuously. Before 6.6.4 it gave up on such a market permanently, which is how an unbacked level there became impossible to remove.

If you keep a maker configured ACTIVE while deliberately not quoting, its old ladder is now cleaned up like any other unbacked depth. Stop the market maker for that market if you want the ladder left alone. The transition is logged once per market rather than on every sweep, so Reconciling <symbol>: in the backend log is where to look.

Hold residue. Report it, then release it:

pnpm reconcile:eco-inorder                 # report
node backend/scripts/reconcile-eco-inorder.mjs --apply
node backend/scripts/reconcile-eco-inorder.mjs --user <userId> --apply

It recomputes the correct hold from the user's open orders and releases only the positive difference. It never debits a balance and never raises a hold, so it cannot over-credit. An under-locked wallet is reported rather than fixed — that is pnpm fix:eco-orders.

Restart the backend after applying, so cached wallet rows refresh.

The row prints the socket error and the address the backend actually used — connect ECONNREFUSED 127.0.0.1:8332, a 401, a 403 — and that address is the whole story. bitcoin-cli reads bitcoin.conf: the network, rpcport= and the .cookie file. The backend reads .env: BTC_NODE_HOST, BTC_NODE_PORT, BTC_NODE_USER, BTC_NODE_PASSWORD. A CLI that answers proves the node is up, not that .env points at it.

The fallback to mempool.space is decided once and cached, so a red row means one of two different situations and the page cannot tell you which.

• The node was already failing when this backend process first needed a BTC provider. The factory fell back to mempool.space, cached that choice for the life of the process, and deposits and withdrawals ride Esplora — which is why nothing else looks broken. What you lose is ZMQ realtime detection and node-side address tracking.
• The node failed after that. The factory never re-evaluates: the runtime is still on the cached node provider, whose calls now fail silently (they return empty results rather than raising), so deposits stop being detected.

Either way the repair ends the same: fix the cause, then restart the backend — nothing returns to the node, or moves off it, without a restart. The estimatesmartfee and ZMQ rows are skipped rather than failed whenever getblockchaininfo itself fails — transport, auth or an RPC error — so one cause shows as one red row. A node that answers with the wrong chain is a different case: that row fails while the other two still run.

The available unspent outputs would cost more in network fees to spend than the amount being withdrawn — usually many small inputs from repeated small deposits.

Check Admin → Ecosystem → Wallets → UTXO for that wallet. If the inputs are genuinely fragmented, the withdrawal needs to be larger or the wallet needs consolidating. If inputs show as LOCKED against a transaction that never completed, that is a stuck payout rather than a fee problem — check the withdrawal queue.

The driver caps in-flight requests per connection and refuses beyond it. The write path that used to cause this is bounded now, so seeing it today usually means genuine concurrency on an under-provisioned pool.

SCYLLA_LOCAL_CONNECTIONS tunes it, accepting 1 to 32 and defaulting to 4. A shared two-core VPS wants fewer; a dedicated cluster more. A bigger pool is headroom, not a fix — if it recurs immediately after raising it, something is issuing unbounded writes and the pool size is not the problem.

Two causes, usually both.

Permission gates ship strict and fail closed, so a newly enabled addon is reachable by Super Admin only until an operator grants its keys per role in Admin → Roles. The addon enforces twenty-eight keys on its routes; access.ecosystem is the one that opens the console.

And the route gate is held in memory, so a grant does not take effect until the API process restarts.

If the menu is missing for a Super Admin too, the extension is not actually enabled — check Admin → System → Extensions and confirm the licence validated.