Broker Configuration & Operation

Overview

The Broker is a service that runs within the Bento proving stack. It is responsible for market interactions including bidding on jobs, locking them, issuing job requests to the Bento proving cluster, and submitting proof fulfillments on-chain.

Broker Configuration

Broker configuration is primarily managed through the broker.toml file in the Boundless directory. This file is mounted into the Broker container and it is used to configure the Broker daemon.

Hitpoints (HP) Tokens

In the current phase, the Boundless market will only accept hitpoint (HP) tokens as stake for proving lock-ins. Provers stake HP tokens to participate in proof generation, and failure to meet performance standards results in slashing of your provers HP. These HP tokens can only be used within the Boundless market and as such, they are non-transferrable and hold no monetary value.

I got slashed, pls help?

If a slashing event occurs against your broker, your stake account will enter a frozen state. This means that your prover will not be able to lock-in any proof requests until you unfreeze your account manually.

To unfreeze your account after slashing, run:

Terminal

export RPC_URL=<TARGET_CHAIN_RPC_URL>
export PRIVATE_KEY=<BROKER_PRIVATE_KEY>
export BOUNDLESS_MARKET_ADDRESS=<BOUNDLESS_MARKET_ADDR>
 
boundless-cli unfreeze

Please refer to the Deployments page for the correct Boundless market contract address.

Deposit / Balance

The Boundless market requires staking funds to be deposited to bid on requests. Brokers must first deposit some HP into the market contract to fund their account. These funds cover staking during lock-in. It is recommend that a broker keep a balance on the market >= max_stake (configured via broker.toml).

Deposit Stake to the Market

Terminal

export RPC_URL=<TARGET_CHAIN_RPC_URL>
export PRIVATE_KEY=<BROKER_PRIVATE_KEY>
export BOUNDLESS_MARKET_ADDRESS=<BOUNDLESS_MARKET_ADDR>
 
# Example: 'deposit-stake 100'
boundless-cli deposit-stake <HP_TO_DEPOSIT>

Check Current Stake Balance

Terminal

export RPC_URL=<TARGET_CHAIN_RPC_URL>
export PRIVATE_KEY=<BROKER_PRIVATE_KEY>
export BOUNDLESS_MARKET_ADDRESS=<BOUNDLESS_MARKET_ADDR>
 
boundless-cli balance-of-stake [wallet_address]

You can omit the PRIVATE_KEY environment variable here and specify your wallet_address as a optional parameter to the balance command, i.e., balance 0x000....

Settings in Broker.toml

broker.toml contains the following settings for the market:

setting	initial value	description
mcycle_price	`".001"`	The price (in native token of target market) of proving 1M cycles.
assumption_price	`"0.1"`	Currently unused.
peak_prove_khz	`500`	This should correspond to the maximum number of cycles per second (in kHz) your proving backend can operate.
min_deadline	`150`	This is a minimum number of blocks before the requested job expiration that Broker will attempt to lock a job.
lookback_blocks	`100`	This is used on Broker initialization, and sets the number of blocks to look back for candidate proofs.
max_stake	`"0.5"`	The maximum amount used to lock in a job for any single order.
skip_preflight_ids	`[]`	A list of `imageID`s that the Broker should skip preflight checks in format `["0xID1","0xID2"]`.
max_file_size	`50_000_000`	The maximum guest image size in bytes that the Broker will accept.
allow_client_addresses	`[]`	When defined, this acts as a firewall to limit proving only to specific client addresses.
lockin_priority_gas	`100`	Additional gas to add to the base price when locking in stake on a contract to increase priority.

Broker Operation

Make sure Bento is running

To check Bento is running correctly, you can send a sample proof workload:

Terminal

# In the bento directory
RUST_LOG=info cargo run --bin bento_cli -- -c 32

Terminal

2024-10-23T14:37:37.364844Z  INFO bento_cli: image_id: a0dfc25e54ebde808e4fd8c34b6549bbb91b4928edeea90ceb7d1d8e7e9096c7 | input_id: eccc8f06-488a-426c-ae3d-e5acada9ae22
2024-10-23T14:37:37.368613Z  INFO bento_cli: STARK job_id: 0d89e2ca-a1e3-478f-b89d-8ab23b89f51e
2024-10-23T14:37:37.369346Z  INFO bento_cli: STARK Job running....
2024-10-23T14:37:39.371331Z  INFO bento_cli: STARK Job running....
2024-10-23T14:37:41.373508Z  INFO bento_cli: STARK Job running....
2024-10-23T14:37:43.375780Z  INFO bento_cli: Job done!

Running the Broker service

Terminal

docker compose --profile broker --env-file ./.env-compose up --build

Stopping The Broker Service

Terminal

docker compose --profile broker down

Broker Optimization

Increasing Lock-in Rate

Once your broker is running, there are a few methods to optimize the lock-in rate. These methods are aimed at making your broker service more competitive in the market through different means:

Decreasing the mcycle_price would tune your Broker to bid at lower prices for proofs.
Increasing lockin_priority_gas expedites your market operations by consuming more gas which could help outrun other bidders.
Adding known a imageID to skip_preflight_ids would reduce the delay of preflight/execution on a binary. This would allow you to beat other Brokers by submitting bids quicker.

Tuning Service Settings

The [prover] settings in broker.toml are used to configure the prover service and significantly impact the operation of the service. The most important configuration variable to monitor and iteratively tune is txn_timeout. This is the number of seconds to wait for a transaction to be confirmed before timing out. Therefore, if you see timeouts in your logs, txn_timeout can be increased to wait longer for transaction confirmations on-chain.

Debugging

Orders Stuck in 'Lockin' or `submit_merkle` Confirmation Timeouts

You may notice on the indexer that the broker has a high number of orders locked-in, but the fulfillment rate is low or even zero. This could be due to transaction confirmation timeouts. A good place to start would be to increase the txn_timeout in the broker.toml file iteratively, and see how that affects the broker's fulfillment rate.

If you need a manual way to restart orders that are "stuck", please follow:

First, you'll need to manually connect to the SQLite database for the broker. This can be done inside the broker container via `sqlite3 /db/broker.db` or by mounting the `broker-data` Docker volume

Next, you'll need to find the batch that contains the order:

Terminal

SELECT id FROM batches WHERE data->>'orders' LIKE '%"TARGET_ORDER_ID"%';
-- Example: SELECT id FROM batches WHERE data->>'orders' LIKE '%"0x466acfc0f27bba9fbb7a8508f576527e81e83bd00000caa"%';

Finally, you can trigger a rerun of the submitter task:

Terminal

UPDATE batches SET data = json_set(data, '$.status', 'Complete') WHERE id = YOUR_BATCH_ID_FROM_STEP_2;
-- Example: UPDATE batches SET data = json_set(data, '$.status', 'Complete') WHERE id = 1;

Overview