Broker Configuration & Operation

Overview

The Broker is a service that runs within the Bento proving stack. It is responsible for market interactions including bidding on jobs, locking them, issuing job requests to the Bento proving cluster, and submitting proof fulfillments onchain.

Broker Configuration

Broker configuration is primarily managed through the broker.toml file in the Boundless directory. This file is mounted into the Broker container and it is used to configure the Broker daemon.

Deposit / Balance

The Boundless market requires funds (USDC) deposited as stake before a prover can bid on requests. Brokers must first deposit some USDC into the market contract to fund their account. These funds cover staking during lock-in. It is recommend that a broker keep a balance on the market >= max_stake (configured via broker.toml).

Deposit Stake to the Market

export RPC_URL=<TARGET_CHAIN_RPC_URL>
export PRIVATE_KEY=<BROKER_PRIVATE_KEY>
source <(just env testnet)
 
# Example: 'account deposit-stake 100'
boundless account deposit-stake <USDC_TO_DEPOSIT>

Check Current Stake Balance

export RPC_URL=<TARGET_CHAIN_RPC_URL>
export PRIVATE_KEY=<BROKER_PRIVATE_KEY>
source <(just env testnet)
 
boundless account stake-balance [wallet_address]

You can omit the PRIVATE_KEY environment variable here and specify your wallet_address as a optional parameter to the balance command, i.e., account balance 0x000....

Settings in Broker.toml

broker.toml contains the following settings for the market:

Broker Operation

2024-10-23T14:37:37.364844Z  INFO bento_cli: image_id: a0dfc25e54ebde808e4fd8c34b6549bbb91b4928edeea90ceb7d1d8e7e9096c7 | input_id: eccc8f06-488a-426c-ae3d-e5acada9ae22
2024-10-23T14:37:37.368613Z  INFO bento_cli: STARK job_id: 0d89e2ca-a1e3-478f-b89d-8ab23b89f51e
2024-10-23T14:37:37.369346Z  INFO bento_cli: STARK Job running....
2024-10-23T14:37:39.371331Z  INFO bento_cli: STARK Job running....
2024-10-23T14:37:41.373508Z  INFO bento_cli: STARK Job running....
2024-10-23T14:37:43.375780Z  INFO bento_cli: Job done!

Benchmarking Bento

First, start a bento cluster:

just bento
 
# Or with an env file for configuration
just bento .env.broker

Load environment variables for the target network:

# For example, if using an env file
source <(just env broker)

Then, run the benchmark:

boundless proving benchmark --request-ids <IDS>

where IDS is a comma-separated list of request IDs from the network or order stream configured.

It is recommended to pick a few requests of varying sizes and programs, biased towards larger proofs for a more representative benchmark.

To run programs manually, and for performance optimizations, see performance optimizations.

Running the Broker service with bento

Running a broker with just will also start the Bento cluster through docker compose.

just broker

Make sure Bento is running

To check Bento is running correctly, you can send a sample proof workload:

Before running this, install Bento CLI

# In the bento directory
RUST_LOG=info bento_cli -c 32

Running a standalone broker

To run broker with an already initialized Bento cluster or with a different prover, you can build and run a broker directly with the following:

cargo build --bin broker --release
# Run with flags or environment variables based on network/configuration
./target/release/broker

Stopping The Broker Service

just broker down

Safe Upgrade Steps

When upgrading your Boundless broker to a new version, follow these steps to ensure a safe migration:

max_concurrent_proofs = 0

just broker clean
 
# Or stop the broker without clearing volumes
just broker down

git checkout <new_version_tag>
# Example: git checkout v0.9.0

just broker

max_concurrent_proofs = <your_previous_value>
# Example: max_concurrent_proofs = 4

Broker Optimization

Increasing Lock-in Rate

Once your broker is running, there are a few methods to optimize the lock-in rate. These methods are aimed at making your broker service more competitive in the market through different means:

1. Decreasing the mcycle_price would tune your Broker to bid at lower prices for proofs.
2. Increasing lockin_priority_gas expedites your market operations by consuming more gas which could help outrun other bidders.

Tuning Service Settings

The [prover] settings in broker.toml are used to configure the prover service and significantly impact the operation of the service. The most important configuration variable to monitor and iteratively tune is txn_timeout. This is the number of seconds to wait for a transaction to be confirmed before timing out. Therefore, if you see timeouts in your logs, txn_timeout can be increased to wait longer for transaction confirmations onchain.

Debugging

Orders Stuck in 'Lockin' or submit_merkle Confirmation Timeouts

You may notice on the explorer that the broker has a high number of orders locked-in, but the fulfillment rate is low or even zero. This could be due to transaction confirmation timeouts. A good place to start would be to increase the txn_timeout in the broker.toml file iteratively, and see how that affects the broker's fulfillment rate.

If you need a manual way to restart orders that are "stuck", please follow:

SELECT id FROM batches WHERE data->>'orders' LIKE '%"TARGET_ORDER_ID"%';
-- Example: SELECT id FROM batches WHERE data->>'orders' LIKE '%"0x466acfc0f27bba9fbb7a8508f576527e81e83bd00000caa"%';

UPDATE batches SET data = json_set(data, '$.status', 'Complete') WHERE id = YOUR_BATCH_ID_FROM_STEP_2;
-- Example: UPDATE batches SET data = json_set(data, '$.status', 'Complete') WHERE id = 1;