Leaderboard

Frozen suites · measured latency · public splits

System One Bench gives every backend the same typed questions. Choice picks one of a fixed set of options. Score places a state on an ordered rubric. Noul returns the probability of yes. The rows below were produced by this repository. A dash is a missing measurement, not a zero, and a pending row is not a result.

Generated 2026-10-11 23:10 UTC. Protocol 1.0.0. Hardware for the published local runs: platform Linux-6.12.94+-x86_64-with-glibc2.41, python 3.12.14, cpu_count 8. Batch size 1, one request at a time. Client-observed latency, not a number copied from a model card.

How to read the floor. TF-IDF and majority are fit on each suite's train split. Random and Strands Decider are not. Strands Decider, hosted Jev, Kev, Laya, and the LLM baseline are zero-shot on the same questions; the hosted and server systems are pending until someone runs them here. Rows marked partial were run with --limit on the first N test items and cover fewer suites, so compare them suite by suite, not by the overall mean. A supervised floor can beat a zero-shot system without that meaning the floor is the better decision model. It means the decision was learnable from the labels.

Mean across suites, weighted by test size

A 3,000-row suite counts more than a 40-row suite. Intervals are omitted here because they belong to single suites; the table under this one has the 95% bootstrap interval on every suite. Flip is the size-weighted mean of per-suite mean probe flip rates. Lower ECE, Brier, flip, latency, and cost are better. Higher S1, accuracy, macro-F1, and schema validity are better.


TF-IDF + logistic regression	Supervised baseline	10 suites	mean	84.5	0.826	0.820	0.321	0.400	1.000	0.050	0.42	0.50	0.0000
Strands Decider 2B	System One (local default)	6 suites, partial (6 limited runs, 1116 test items)	mean	74.5	0.624	0.559	0.113	0.452	1.000	0.222	5411.89	7584.42	0.0000
Majority class	Supervised baseline	10 suites	mean	59.6	0.236	0.123	0.006	0.780	1.000	0.000	0.01	0.01	0.0000
Random	Unsupervised baseline	10 suites	mean	58.0	0.180	0.168	0.091	0.863	1.000	0.000	0.02	0.03	0.0000
Hosted Jev	System One	—	—	pending	—	—	—	—	—	—	—	—	—
Kev	System One	—	—	pending	—	—	—	—	—	—	—	—	—
Laya	System One	—	—	pending	—	—	—	—	—	—	—	—	—
LLM structured output	LLM baseline	—	—	pending	—	—	—	—	—	—	—	—	—
Generic System One server	System One	—	—	pending	—	—	—	—	—	—	—	—	—

S1 score by primitive

Unweighted mean of suite S1 scores. The formula is published and every component is on the suite rows. quality = 0.35*macro_f1 + 0.25*accuracy + 0.20*(1 - min(ECE, 1)) + 0.20*(1 - min(Brier / 2, 1)); s1_score = 100 * (0.70*quality + 0.15*schema_validity + 0.15*robustness), where robustness = 1 - the unweighted mean of probe decision-flip rates. If robustness was not measured, its 0.15 weight is dropped and the quality and schema weights are renormalized. Latency and cost are not in the score. Brier is the mean squared L2 error of the outcome distribution (maximum 2). ECE is 10-bin top-label expected calibration error on schema-valid items. Composite id: s1-v1.

Backend	Choice	Score	Noul	All
Strands Decider 2B	73.0	57.1	82.5	74.5
Random	53.1	58.3	70.9	58.0
Majority class	53.0	57.6	77.8	59.6
TF-IDF + logistic regression	83.7	61.9	91.3	84.5

Every published suite run

Suite Primitive


Strands Decider 2B	System One (local default)	ag_news (partial: first 200 test items)	choice	93.3	0.900 [0.860, 0.940]	0.900 [0.858, 0.939]	0.060 [0.036, 0.108]	0.184 [0.126, 0.250]	1.000	0.024	2113.68	2866.38	0.0000
TF-IDF + logistic regression	Supervised baseline	ag_news	choice	86.7	0.863 [0.839, 0.889]	0.861 [0.837, 0.887]	0.310 [0.288, 0.335]	0.358 [0.340, 0.375]	1.000	0.045	0.36	0.42	0.0000
Random	Unsupervised baseline	ag_news	choice	59.6	0.236 [0.207, 0.265]	0.236 [0.206, 0.265]	0.179 [0.152, 0.212]	0.830 [0.812, 0.849]	1.000	0.000	0.01	0.01	0.0000
Majority class	Supervised baseline	ag_news	choice	59.6	0.250 [0.221, 0.280]	0.100 [0.091, 0.109]	0.000 [0.000, 0.034]	0.750 [0.750, 0.750]	1.000	0.000	<0.01	<0.01	0.0000
TF-IDF + logistic regression	Supervised baseline	agent_gate	choice	64.8	0.333 [0.000, 0.667]	0.356 [0.000, 0.556]	0.106 [0.032, 0.447]	0.682 [0.607, 0.755]	1.000	0.100	0.40	0.56	0.0000
Majority class	Supervised baseline	agent_gate	choice	63.2	0.333 [0.000, 0.667]	0.167 [0.000, 0.267]	0.000 [0.000, 0.333]	0.667 [0.667, 0.667]	1.000	0.000	<0.01	0.02	0.0000
Random	Unsupervised baseline	agent_gate	choice	54.9	0.167 [0.000, 0.500]	0.133 [0.000, 0.286]	0.301 [0.130, 0.518]	0.730 [0.628, 0.818]	1.000	0.000	0.01	0.03	0.0000
Strands Decider 2B	System One (local default)	agent_tool_routing (partial: first 200 test items)	choice	52.6	0.370 [0.300, 0.440]	0.098 [0.082, 0.110]	0.166 [0.145, 0.248]	0.804 [0.741, 0.862]	1.000	0.420	17446.90	21289.74	0.0000
TF-IDF + logistic regression	Supervised baseline	banking77	choice	83.3	0.849 [0.837, 0.862]	0.848 [0.834, 0.859]	0.462 [0.450, 0.474]	0.474 [0.465, 0.485]	1.000	0.039	0.43	0.49	0.0000
Random	Unsupervised baseline	banking77	choice	51.4	0.013 [0.009, 0.017]	0.012 [0.009, 0.016]	0.013 [0.009, 0.017]	0.991 [0.991, 0.992]	1.000	0.000	0.04	0.05	0.0000
Majority class	Supervised baseline	banking77	choice	51.2	0.013 [0.009, 0.018]	0.000 [0.000, 0.000]	0.006 [0.001, 0.010]	0.988 [0.988, 0.988]	1.000	0.000	0.02	0.02	0.0000
TF-IDF + logistic regression	Supervised baseline	helpsteer2	score	62.0	0.287 [0.233, 0.340]	0.281 [0.228, 0.330]	0.040 [0.010, 0.091]	0.782 [0.773, 0.791]	1.000	0.122	0.73	1.05	0.0000
Random	Unsupervised baseline	helpsteer2	score	58.4	0.207 [0.160, 0.250]	0.204 [0.158, 0.247]	0.150 [0.110, 0.200]	0.868 [0.841, 0.896]	1.000	0.000	0.01	0.02	0.0000
Majority class	Supervised baseline	helpsteer2	score	57.5	0.200 [0.157, 0.247]	0.067 [0.054, 0.079]	0.000 [0.000, 0.053]	0.800 [0.800, 0.800]	1.000	0.000	<0.01	<0.01	0.0000
Strands Decider 2B	System One (local default)	helpsteer2 (partial: first 200 test items)	score	57.1	0.280 [0.215, 0.350]	0.249 [0.186, 0.314]	0.015 [0.008, 0.084]	0.776 [0.757, 0.798]	1.000	0.418	6306.95	12361.25	0.0000
Majority class	Supervised baseline	passage_relevance	score	59.6	0.250 [0.000, 0.750]	0.100 [0.000, 0.214]	0.000 [0.000, 0.500]	0.750 [0.750, 0.750]	1.000	0.000	<0.01	0.01	0.0000
TF-IDF + logistic regression	Supervised baseline	passage_relevance	score	54.6	0.250 [0.000, 0.750]	0.125 [0.000, 0.250]	0.120 [0.024, 0.512]	0.775 [0.704, 0.865]	1.000	0.250	0.40	0.55	0.0000
Random	Unsupervised baseline	passage_relevance	score	47.3	0.000 [0.000, 0.000]	0.000 [0.000, 0.000]	0.385 [0.330, 0.439]	0.757 [0.639, 0.916]	1.000	0.000	0.01	0.03	0.0000
TF-IDF + logistic regression	Supervised baseline	prompt_injection	noul	84.7	0.767 [0.690, 0.845]	0.759 [0.678, 0.838]	0.080 [0.053, 0.158]	0.329 [0.276, 0.383]	1.000	0.126	0.34	0.41	0.0000
Strands Decider 2B	System One (local default)	prompt_injection (partial: first 116 test items)	noul	76.7	0.664 [0.578, 0.750]	0.630 [0.537, 0.717]	0.191 [0.126, 0.262]	0.368 [0.297, 0.441]	1.000	0.209	1629.42	2335.33	0.0000
Random	Unsupervised baseline	prompt_injection	noul	69.2	0.466 [0.379, 0.560]	0.466 [0.379, 0.552]	0.260 [0.183, 0.354]	0.665 [0.571, 0.756]	1.000	0.000	<0.01	0.01	0.0000
Majority class	Supervised baseline	prompt_injection	noul	68.6	0.483 [0.388, 0.578]	0.326 [0.280, 0.366]	0.146 [0.051, 0.241]	0.542 [0.493, 0.591]	1.000	0.000	<0.01	<0.01	0.0000
Strands Decider 2B	System One (local default)	shell_risk_safe (partial: first 200 test items)	noul	71.1	0.545 [0.475, 0.615]	0.508 [0.438, 0.582]	0.130 [0.065, 0.198]	0.498 [0.451, 0.544]	1.000	0.238	1982.41	2452.58	0.0000
Strands Decider 2B	System One (local default)	sms_spam (partial: first 200 test items)	noul	97.3	1.000 [1.000, 1.000]	1.000 [1.000, 1.000]	0.150 [0.145, 0.156]	0.048 [0.044, 0.053]	1.000	0.016	1403.36	1996.64	0.0000
TF-IDF + logistic regression	Supervised baseline	sms_spam	noul	96.6	0.972 [0.961, 0.983]	0.936 [0.906, 0.961]	0.068 [0.063, 0.079]	0.051 [0.041, 0.062]	1.000	0.004	0.35	0.40	0.0000
Majority class	Supervised baseline	sms_spam	noul	82.9	0.866 [0.843, 0.890]	0.464 [0.457, 0.471]	0.000 [0.000, 0.027]	0.232 [0.197, 0.265]	1.000	0.000	<0.01	<0.01	0.0000
Random	Unsupervised baseline	sms_spam	noul	70.7	0.507 [0.474, 0.543]	0.425 [0.394, 0.458]	0.174 [0.137, 0.208]	0.594 [0.562, 0.626]	1.000	0.000	<0.01	<0.01	0.0000
TF-IDF + logistic regression	Supervised baseline	support_routing	choice	60.4	0.278 [0.056, 0.500]	0.286 [0.062, 0.465]	0.111 [0.028, 0.341]	0.822 [0.807, 0.839]	1.000	0.144	0.35	0.41	0.0000
Random	Unsupervised baseline	support_routing	choice	59.7	0.222 [0.056, 0.389]	0.233 [0.056, 0.360]	0.140 [0.058, 0.302]	0.856 [0.749, 0.957]	1.000	0.000	0.01	0.02	0.0000
Majority class	Supervised baseline	support_routing	choice	56.2	0.167 [0.000, 0.333]	0.048 [0.000, 0.083]	0.000 [0.000, 0.167]	0.833 [0.833, 0.833]	1.000	0.000	<0.01	<0.01	0.0000
Majority class	Supervised baseline	tool_selection	choice	54.7	0.125 [0.000, 0.312]	0.028 [0.000, 0.060]	0.000 [0.000, 0.188]	0.875 [0.875, 0.875]	1.000	0.000	<0.01	<0.01	0.0000
TF-IDF + logistic regression	Supervised baseline	tool_selection	choice	54.7	0.250 [0.062, 0.500]	0.192 [0.031, 0.317]	0.096 [0.025, 0.342]	0.853 [0.833, 0.871]	1.000	0.338	0.35	0.44	0.0000
Random	Unsupervised baseline	tool_selection	choice	54.3	0.125 [0.000, 0.312]	0.100 [0.000, 0.188]	0.127 [0.099, 0.291]	0.933 [0.846, 1.020]	1.000	0.000	0.02	0.02	0.0000
TF-IDF + logistic regression	Supervised baseline	wiki_toxic	noul	84.1	0.770 [0.730, 0.806]	0.770 [0.729, 0.806]	0.134 [0.107, 0.166]	0.350 [0.331, 0.370]	1.000	0.126	0.41	0.54	0.0000
Random	Unsupervised baseline	wiki_toxic	noul	71.6	0.494 [0.452, 0.536]	0.494 [0.452, 0.536]	0.203 [0.160, 0.247]	0.616 [0.577, 0.658]	1.000	0.000	<0.01	0.01	0.0000
Majority class	Supervised baseline	wiki_toxic	noul	71.4	0.500 [0.462, 0.542]	0.333 [0.316, 0.352]	0.000 [0.000, 0.050]	0.500 [0.500, 0.500]	1.000	0.000	<0.01	<0.01	0.0000
Hosted Jev	System One	—	—	pending	—	—	—	—	—	—	—	—	—
Kev	System One	—	—	pending	—	—	—	—	—	—	—	—	—
Laya	System One	—	—	pending	—	—	—	—	—	—	—	—	—
LLM structured output	LLM baseline	—	—	pending	—	—	—	—	—	—	—	—	—
Generic System One server	System One	—	—	pending	—	—	—	—	—	—	—	—	—

Not measured here

Pending is not a score. These backends are part of the protocol. This environment did not run them. Do not paste a number from JevBench, Jevals, sysone-bench, ReflexBench, or a model card into the empty cells.

Strands Decider 2B: Not run yet. Install with uv add strands-decider; weights download from Hugging Face on first use.; uv run s1bench run --backend decider --suite all
Hosted Jev: Not run in the published environment. TYPESAFE_API_KEY was not available, and no external Jev number is copied onto this board.; TYPESAFE_API_KEY=... uv run s1bench run --backend jev --suite all
Kev: Not run in the published environment. No local Kev server or weights were loaded, and no author-reported Kev number is copied onto this board.; S1BENCH_KEV_BASE_URL=http://127.0.0.1:8009 uv run s1bench run --backend kev --suite all
Laya: Not run in the published environment. No local Laya server was loaded, and no author-reported Laya number is copied onto this board.; S1BENCH_LAYA_BASE_URL=http://127.0.0.1:8000 uv run s1bench run --backend laya --suite all
LLM structured output: Not run in the published environment. S1BENCH_LLM_BASE_URL and S1BENCH_LLM_MODEL were not set.; S1BENCH_LLM_BASE_URL=https://api.openai.com/v1 S1BENCH_LLM_API_KEY=... S1BENCH_LLM_MODEL=... uv run s1bench run --backend llm --suite all
Generic System One server: Not run in the published environment. S1BENCH_GENERIC_BASE_URL was not set.; S1BENCH_GENERIC_BASE_URL=http://127.0.0.1:9000 uv run s1bench run --backend generic --suite all