Hidden-effect recall
Of the relevant hidden effects in the evaluation, how many did the system identify?
VTWM vs. a language-model agent with a policy monitor: hidden-effect recall, delayed-risk ranking and silent policy violations.
Baseline: a language-model agent with a policy monitor.
| Measure | VTWM | Baseline | Direction |
|---|---|---|---|
| Hidden-effect recall | 84.13% | 61.94% | Higher is better |
| Delayed-risk AUROC | 0.8957 | 0.8496 | Higher is better |
| Silent policy-violation rate | 1.31% | 21.76% | Lower is better |
Both systems were tested on the same 2,400 cases, including 648 policy violations.
| Measure | VTWM | Baseline |
|---|---|---|
| Cases | 2,400 | 2,400 |
| Policy violations | 648 | 648 |
| Hidden effects found | 356 / 480 | 257 / 480 |
| Delayed-risk AUPRC | 86.81% | 76.46% |
| Reviews | 1,071 | 1,389 |
| Violations left silent | 2 | 167 |
| Evaluation slice | Transition F1 | Hidden-effect recall | Delayed-risk AUPRC |
|---|---|---|---|
| New workflow structures | 94.21 | 84.04% | 86.09% |
| New enterprise environment | 94.15 | 86.98% | 85.31% |
| New policy family | 94.28 | 84.10% | 85.64% |
Of the relevant hidden effects in the evaluation, how many did the system identify?
AUROC and AUPRC measure delayed-risk ranking in their respective test sets.
Policy violations that pass without detection or review, measured within each test set.
Discuss the evaluation protocol, test populations, data splits and supporting artifacts.
Discuss the evaluationDiscuss model versions, baselines, data splits, uncertainty estimates and ablations.