TECHNICAL EVALUATION

Compare VTWM
with the baseline.

VTWM vs. a language-model agent with a policy monitor: hidden-effect recall, delayed-risk ranking and silent policy violations.

PRIMARY RESULTS

VTWM and the baseline.

Baseline: a language-model agent with a policy monitor.

MeasureVTWMBaselineDirection
Hidden-effect recall84.13%61.94%Higher is better
Delayed-risk AUROC0.89570.8496Higher is better
Silent policy-violation rate1.31%21.76%Lower is better
POLICY-INVISIBLE TEST SET

2 violations left silent.
Compared with 167.

Both systems were tested on the same 2,400 cases, including 648 policy violations.

MeasureVTWMBaseline
Cases2,4002,400
Policy violations648648
Hidden effects found356 / 480257 / 480
Delayed-risk AUPRC86.81%76.46%
Reviews1,0711,389
Violations left silent2167
TRANSFER EVALUATION

New structures, environments
and policy families.

Evaluation sliceTransition F1Hidden-effect recallDelayed-risk AUPRC
New workflow structures94.2184.04%86.09%
New enterprise environment94.1586.98%85.31%
New policy family94.2884.10%85.64%
METRIC DEFINITIONS

What each metric measures.

01

Hidden-effect recall

Of the relevant hidden effects in the evaluation, how many did the system identify?

02

Delayed-risk AUROC and AUPRC

AUROC and AUPRC measure delayed-risk ranking in their respective test sets.

03

Silent policy failure

Policy violations that pass without detection or review, measured within each test set.

Technical review

Discuss the evaluation protocol, test populations, data splits and supporting artifacts.

Discuss the evaluation
PUT SUPRAOS TO WORK

Inspect the evaluation.

Discuss model versions, baselines, data splits, uncertainty estimates and ablations.