Broker Configuration & Operation

Overview

The Broker is a service that runs within the Bento proving stack. It is responsible for market interactions including bidding on jobs, locking them, issuing job requests to the Bento proving cluster, and submitting proof fulfillments onchain.

Broker Configuration

Broker configuration is primarily managed through the broker.toml file in the Boundless directory. This file is mounted into the Broker container and it is used to configure the Broker daemon.

Deposit / Balance

The Boundless market requires funds (USDC) deposited as stake before a prover can bid on requests. Brokers must first deposit some USDC into the market contract to fund their account. These funds cover staking during lock-in. It is recommend that a broker keep a balance on the market >= max_stake (configured via broker.toml).

Deposit Stake to the Market

Terminal

export RPC_URL=<TARGET_CHAIN_RPC_URL>
export PRIVATE_KEY=<BROKER_PRIVATE_KEY>
# Use the env file based on the network you are connecting to
source .env.eth-sepolia
 
# Example: 'account deposit-stake 100'
boundless account deposit-stake <USDC_TO_DEPOSIT>

Check Current Stake Balance

Terminal

export RPC_URL=<TARGET_CHAIN_RPC_URL>
export PRIVATE_KEY=<BROKER_PRIVATE_KEY>
# Use the env file based on the network you are connecting to
source .env.eth-sepolia
 
boundless account stake-balance [wallet_address]

You can omit the PRIVATE_KEY environment variable here and specify your wallet_address as a optional parameter to the balance command, i.e., account balance 0x000....

Settings in Broker.toml

broker.toml contains the following settings for the market:

setting	initial value	description
mcycle_price	`".001"`	The price (in native token of target market) of proving 1M cycles.
assumption_price	`"0.1"`	Currently unused.
peak_prove_khz	`500`	This should correspond to the maximum number of cycles per second (in kHz) your proving backend can operate.
min_deadline	`150`	This is a minimum number of blocks before the requested job expiration that Broker will attempt to lock a job.
lookback_blocks	`100`	This is used on Broker initialization, and sets the number of blocks to look back for candidate proofs.
max_stake	`"0.5"`	The maximum amount used to lock in a job for any single order.
skip_preflight_ids	`[]`	A list of `imageID`s that the Broker should skip preflight checks in format `["0xID1","0xID2"]`.
max_file_size	`50_000_000`	The maximum guest image size in bytes that the Broker will accept.
allow_client_addresses	`[]`	When defined, this acts as a firewall to limit proving only to specific client addresses.
lockin_priority_gas	`100`	Additional gas to add to the base price when locking in stake on a contract to increase priority.

Broker Operation

Terminal

2024-10-23T14:37:37.364844Z  INFO bento_cli: image_id: a0dfc25e54ebde808e4fd8c34b6549bbb91b4928edeea90ceb7d1d8e7e9096c7 | input_id: eccc8f06-488a-426c-ae3d-e5acada9ae22
2024-10-23T14:37:37.368613Z  INFO bento_cli: STARK job_id: 0d89e2ca-a1e3-478f-b89d-8ab23b89f51e
2024-10-23T14:37:37.369346Z  INFO bento_cli: STARK Job running....
2024-10-23T14:37:39.371331Z  INFO bento_cli: STARK Job running....
2024-10-23T14:37:41.373508Z  INFO bento_cli: STARK Job running....
2024-10-23T14:37:43.375780Z  INFO bento_cli: Job done!

Benchmarking Bento

Load environment variables for the target network:

Terminal

# For example, to benchmark an order on Ethereum Sepolia
source .env.eth-sepolia
 
# Load any other relevant env variables here, specifically `RPC_URL` and postgres env (if not default)

Start a bento cluster:

Terminal

just bento

Then, run the benchmark:

Terminal

boundless proving benchmark --request-ids <IDS>

where IDS is a comma-separated list of request IDs from the network or order stream configured.

It is recommended to pick a few requests of varying sizes and programs, biased towards larger proofs for a more representative benchmark.

To run programs manually, and for performance optimizations, see performance optimizations.

Running the Broker service with bento

Running a broker with just will also start the Bento cluster through docker compose.

Terminal

just broker

Make sure Bento is running

To check Bento is running correctly, you can send a sample proof workload:

Before running this, install Bento CLI

Terminal

# In the bento directory
RUST_LOG=info bento_cli -c 32

Running a standalone broker

To run broker with an already initialized Bento cluster or with a different prover, you can build and run a broker directly with the following:

Terminal

cargo build --bin broker --release
# Run with flags or environment variables based on network/configuration
./target/release/broker

Stopping The Broker Service

Terminal

just broker down

Safe Upgrade Steps

When upgrading your Boundless broker to a new version, follow these steps to ensure a safe migration:

Stop the broker and optionally clean the database

Terminal

just broker clean
 
# Or stop the broker without clearing volumes
just broker down

This will wait for any committed orders to finalize before shutting down. Avoid sending kill signals to the broker process and ensure either through the broker logs or through indexer that your broker does not have any incomplete locked orders before proceeding.

Update to the new version

See releases for latest tag to use.

Terminal

git checkout <new_version_tag>
# Example: git checkout v0.9.0

Start the broker with the new version

Terminal

just broker

Running Multiple Brokers

You can run multiple broker instances simultaneously to serve different networks at the same time while sharing the same Bento cluster. The Docker compose setup supports this through the broker2 service example.

Multi-Broker Configuration

Each broker instance requires:

Separate configuration file: Create different broker.toml files (e.g., broker.toml, broker2.toml, etc.)
Different RPC URL: Use different chain endpoints via setting respective RPC_URL environment variables, or modifying the compose.yml manually.
Optional separate private key: Use different PRIVATE_KEY variables if desired for different accounts on different networks.

Environment Variables for Multi-Broker Setup

If using the default compose.yml file and uncommenting the second broker config:

.env

# Export environment variables for the first broker
export RPC_URL=<URL for network 1>
export PRIVATE_KEY=0x...
 
# Export environment variables for the second broker
export RPC_URL_2=<URL for network 2>

Then, create the new broker config file that the second broker will use:

Terminal

# Copy from an existing broker config file
cp broker.toml broker2.toml
 
# Or creating one from a fresh template
cp broker-template.toml broker2.toml

Then, modify configuation values for each network, keeping the following in mind:

The peak_prove_khz setting is shared across all brokers
- For example, if you have benchmarked your broker to be able to prove at 500kHz, the values in each config should not sum up to be more than 500kHz.
max_concurrent_preflights is set to a value that the bento cluster can keep up with
- It is recommended that the max concurrent preflights across all networks is less than the number of exec agents you have specified in your compose.yml.
max_concurrent_proofs is a per-broker configuration, and is not shared across brokers

Then, just start the cluster as you normally would with:

Terminal

just broker

Broker Optimization

Increasing Lock-in Rate

Once your broker is running, there are a few methods to optimize the lock-in rate. These methods are aimed at making your broker service more competitive in the market through different means:

Decreasing the mcycle_price would tune your Broker to bid at lower prices for proofs.
Increasing lockin_priority_gas expedites your market operations by consuming more gas which could help outrun other bidders.

Tuning Service Settings

The [prover] settings in broker.toml are used to configure the prover service and significantly impact the operation of the service. The most important configuration variable to monitor and iteratively tune is txn_timeout. This is the number of seconds to wait for a transaction to be confirmed before timing out. Therefore, if you see timeouts in your logs, txn_timeout can be increased to wait longer for transaction confirmations onchain.