Policy adherence
In-car assistants must follow domain policies even when compliance conflicts with completing the user request.
Technical Report · CAR-bench Challenge
TRACE
A Self-Evolving Skill Bank for Consistent, Limit-Aware LLM Agents
On the official hidden-set evaluation, TRACE (Trajectory-Contrastive Evolution; team Darwin Agent) ranks 1st overall in Track 1 (Open Track), where any LLM from any provider is permitted. TRACE leads on strict reliability, consistency, and latency while keeping cost competitive. The top entries are summarized below.
| Rank | Team | LLM used | Pass^3 ↓ | Pass@3 | Pass@1 | Successful trials | Consistency | Latency | Tokens / trial | Cost / trial |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Darwin Agent | GPT-5.6 Sol | 70.00% | 83.33% | 70.00% | 69/90 | 88.00% | 25.87 s | 141,684 | $0.27 |
| 2 | Thylinao | Gemini 3.5 Flash | 66.67% | 86.67% | 70.00% | 67/90 | 78.85% | 52.85 s | 263,939 | $0.24 |
| 3 | Xzlon | Gemini 3.5 Flash | 63.33% | 80.00% | 80.00% | 66/90 | 87.50% | 41.00 s | 216,642 | $0.18 |
| 4–6 | 10cars | Claude Opus 4.6 | 60.00% | 80.00% | 76.67% | 65/90 | 85.42% | 104.78 s | 294,281 | $0.56 |
| — | Organizer Baseline | GPT-5.6 Sol | 50.00% | 66.67% | 60.00% | 54/90 | 85.00% | 21.21 s | 82,179 | $0.17 |
Pass^3 requires a task to be solved in all three repeated trials and measures reliability (consistency across runs); Pass@3 requires it to be solved in at least one of the three and measures potential (best-case capability). Bold highlights the best result among the four competing teams.
Reliable agents need more than raw task-solving ability.
CAR-bench tests agents in the safety-critical reality of in-car assistants, where a user may issue incomplete, ambiguous, or unsatisfiable requests, and the agent must respond through multi-turn dialogue, tool use, and strict policy adherence. The benchmark targets two properties: consistency, producing stable behavior across repeated trials, and limit-awareness, recognizing when a request cannot, or cannot yet, be safely fulfilled instead of claiming unsupported success. Even frontier models show a substantial gap between what they can solve at least once and what they solve on every trial. TRACE is built to close that gap.
Limit-awareness
Agents should recognize missing tools, parameters, results, or unsafe conditions instead of claiming unsupported success.
Clarification
Agents should resolve ambiguity internally where possible and ask the user for clarification only when a request remains genuinely underspecified.
Consistency
Agents should behave stably across repeated trials rather than succeed once and fail on the next run.
Can You Beat the Frontier?
Baseline Pass^3 (%) across task types, using the default agent scaffold with no optimization.| Model | Provider | Base | Hallucination | Disambiguation | Average |
|---|---|---|---|---|---|
| Claude Opus 4.6 | Anthropic | 80 | 48 | 46 | 58 |
| GPT-5 | OpenAI | 66 | 60 | 36 | 54 |
| Gemini 2.5 Pro | 53 | 34 | 28 | 38 |
|
| Qwen3-32B | Alibaba | 45 | 27 | 22 | 31 |
| xLAM-2-32B | Salesforce | 26 | 11 | 12 | 16 |
TRACE turns existing model capability into consistent, limit-aware behavior.
Overview of TRACE: the evolution loop refines the Skill Bank; the deployed Actor performs state-conditioned skill orchestration at every turn.
Rather than modifying model weights, TRACE improves the behavioral knowledge that an agent acts on. It couples an evolution loop — where the Curator bootstraps and iteratively refines a Skill Bank from the Actor's evaluation trajectories — with lightweight per-turn state-conditioned skill orchestration at deployment, where the Actor activates only the relevant skills to guide each turn.
InitializationBootstrap an initial bank from the Actor's evaluation trajectories.
A bottom-up pipeline abstracts task-specific experience into reusable, precisely retrievable competencies, \(\mathcal{B}^{\text{task}} \to \mathcal{B}^{\text{type}} \to \mathcal{B}^{\text{op}} \to \mathcal{B}^{(0)}\):
- Task-level distillation
- For each task, compare its trajectories across rounds and distill a skill capturing both successful behaviors and common failure patterns.
- Type-level aggregation
- Group tasks by type and merge their skills, stripping task-specific wording so only behavior shared within each type remains.
- Operation-level abstraction
- Merge skills across types by their underlying operation, unifying shared behavior even from tasks with different surface goals.
- Skill decomposition
- Split broad skills into single-competency ones, so the Actor can activate precisely the knowledge a turn needs rather than overlapping guidance.
Evolution loopRefine the bank each round by contrasting successful and failed trajectories.
Each round applies one update \(\mathcal{B}^{(r+1)} = \Phi(\mathcal{B}^{(r)}, \mathcal{T}^{(r)})\) in three steps, corresponding to the four steps below:
- Skill-aware grouping
- Partition trajectories by the skill each turn selected; set aside those with an unrecognized or no skill to mine missing competencies.
- Deployment-faithful reconstruction
- Render each trajectory as structured text that separates deployment-visible information from evidence available only to the Curator during evolution, so a skill is never rewritten to depend on knowledge the Actor will not have during deployment.
- Contrastive refinement
- Edit existing skills from paired success and failure trajectories, split overly broad ones, and mine new skills from the set-aside cases when a reusable pattern recurs.
DeploymentOrchestrate the relevant skills per turn — relevance, count, and order — and recompute on the next.
Orchestration goes beyond static retrieval: conditioned on the current dialogue state, the Actor jointly determines which skills are relevant, how many to activate, and in what order to compose them, then grounds the sequence in context and recomputes it after every turn.
- State-conditioned orchestration
- Each turn, the Actor evaluates the skill descriptions against the current dialogue history to build an ordered sequence, conditioned on the user's request, unresolved constraints, and observations rather than surface similarity alone.
- Context grounding
- The selected bodies — tool-use rules, behavioral guidelines, and mistakes to avoid — are arranged in context in the chosen order, shaping both what the Actor should do and what it should avoid.
- Per-turn re-orchestration
- Orchestration is recomputed each turn, independent of the previous one; injected bodies are not carried over, so the active competencies track the conversation as it evolves.
EvaluateRun the Actor with the current Skill Bank and collect multi-turn trajectories with rewards, tool calls, and errors.
GroupPartition trajectories by the skill invoked at each turn, while setting aside no-skill cases to mine missing competencies.
ContrastCompare successes and failures under deployment-faithful evidence to identify decisions that separate reliable runs.
RewriteEdit skills from paired success and failure trajectories, split overly broad ones, and mine missing competencies.
Citation.
If this technical report is useful for your research, please cite TRACE.
@techreport{wu2026trace,
title = {TRACE: A Self-Evolving Skill Bank for Consistent, Limit-Aware LLM Agents: A Technical Report on the CAR-bench Challenge},
author = {Wenhao Wu and Menghao Zhang and Xin Wang and Zhi Wang and Kun Shao and Jian Luan},
institution = {Xiaomi Inc. and Nanjing University and Beijing University of Posts and Telecommunications and Tsinghua University},
year = {2026}
}