VeBench evaluation architecture

Versioned tasks · controlled scheduling · isolated execution

01Tasks & images

Benchmark

Original tasks

Harbor Task

Reuse directly

Convert / reuse

Normalize as a Harbor Task

Instructions: instruction.md
Environment: task.toml / environment
Verification: tests

One task contract; existing tasks stay reusable.

OCI image

Build or synchronize

Prewarm

Validate the exact image

TOS manifest · admission

Task identity + digest → image + task configuration

Task = definition. Image = environment.

02Jobs & scheduling

vebench run

Select case · agent · model

Evaluation configuration

Evaluation scheduler

Resolve task → lock versions

Trial queue · concurrency · retry

Runtime policy

Execution configuration

Execute trial

VeBench Environment

Resolve the exact image and task version

E2B: connect · execute commands · transfer files

Scheduling stays separate from sandbox execution.

03veFaaS execution

Provision through veFaaS

Prewarmed image + resources + mounts + gateway

One trial · one Agent Sandbox

Versioned agent bundle

Read-only mount from TOS

Upgrade the harness without rebuilding task images.

Agent runtime

Task environment + agent runtime

Read task → call model → run tools

localhost

Model gateway

Provider access + model requests

Collect raw execution traces

Shared loopback · separate filesystems

Model API

Model service

Ark / other providers

04Scoring & results

Independent verifier

Run the task tests

Separate scoring sandbox

Reward + verification output

Evaluation artifacts

Reward · logs · outputs · traces

Result tools

Reports · leaderboard · trace analysis

Scoring images are prepared and prewarmed too.

Adaptive concurrency · optional control loop

01 Observe

Model capacity, usage and rate-limit feedback

02 Decide

Estimate headroom and workload demand

03 Control

Adjust admission for new trials

Running trials continue; control changes apply to new admissions.

Model API