Strands Decider 2B · ag_news
ECE 0.060 · n valid 200 / 200. Bars are accuracy in the bin. Dots are mean confidence. The dashed line is perfect calibration.
Calibration
Expected calibration error uses ten equal-width bins on the probability of the committed decision. Schema-invalid answers are left out of ECE and Brier and counted as wrong for accuracy. Noul confidence is max(p, 1 − p). Choice and Score confidence is the probability of the reported decision, not the provider's separate confidence field.
Majority class · ag_news
ECE 0.000 · n valid 800 / 800. Bars are accuracy in the bin. Dots are mean confidence. The dashed line is perfect calibration.
Random · ag_news
ECE 0.179 · n valid 800 / 800. Bars are accuracy in the bin. Dots are mean confidence. The dashed line is perfect calibration.
TF-IDF + logistic regression · ag_news
ECE 0.310 · n valid 800 / 800. Bars are accuracy in the bin. Dots are mean confidence. The dashed line is perfect calibration.
Majority class · agent_gate
ECE 0.000 · n valid 6 / 6. Bars are accuracy in the bin. Dots are mean confidence. The dashed line is perfect calibration.
Random · agent_gate
ECE 0.301 · n valid 6 / 6. Bars are accuracy in the bin. Dots are mean confidence. The dashed line is perfect calibration.
TF-IDF + logistic regression · agent_gate
ECE 0.106 · n valid 6 / 6. Bars are accuracy in the bin. Dots are mean confidence. The dashed line is perfect calibration.
Strands Decider 2B · agent_tool_routing
ECE 0.166 · n valid 200 / 200. Bars are accuracy in the bin. Dots are mean confidence. The dashed line is perfect calibration.
Majority class · banking77
ECE 0.006 · n valid 3080 / 3080. Bars are accuracy in the bin. Dots are mean confidence. The dashed line is perfect calibration.
Random · banking77
ECE 0.013 · n valid 3080 / 3080. Bars are accuracy in the bin. Dots are mean confidence. The dashed line is perfect calibration.
TF-IDF + logistic regression · banking77
ECE 0.462 · n valid 3080 / 3080. Bars are accuracy in the bin. Dots are mean confidence. The dashed line is perfect calibration.
Majority class · support_routing
ECE 0.000 · n valid 18 / 18. Bars are accuracy in the bin. Dots are mean confidence. The dashed line is perfect calibration.
Random · support_routing
ECE 0.140 · n valid 18 / 18. Bars are accuracy in the bin. Dots are mean confidence. The dashed line is perfect calibration.
TF-IDF + logistic regression · support_routing
ECE 0.111 · n valid 18 / 18. Bars are accuracy in the bin. Dots are mean confidence. The dashed line is perfect calibration.
Majority class · tool_selection
ECE 0.000 · n valid 16 / 16. Bars are accuracy in the bin. Dots are mean confidence. The dashed line is perfect calibration.
Random · tool_selection
ECE 0.127 · n valid 16 / 16. Bars are accuracy in the bin. Dots are mean confidence. The dashed line is perfect calibration.
TF-IDF + logistic regression · tool_selection
ECE 0.096 · n valid 16 / 16. Bars are accuracy in the bin. Dots are mean confidence. The dashed line is perfect calibration.
Strands Decider 2B · prompt_injection
ECE 0.191 · n valid 116 / 116. Bars are accuracy in the bin. Dots are mean confidence. The dashed line is perfect calibration.
Majority class · prompt_injection
ECE 0.146 · n valid 116 / 116. Bars are accuracy in the bin. Dots are mean confidence. The dashed line is perfect calibration.
Random · prompt_injection
ECE 0.260 · n valid 116 / 116. Bars are accuracy in the bin. Dots are mean confidence. The dashed line is perfect calibration.
TF-IDF + logistic regression · prompt_injection
ECE 0.080 · n valid 116 / 116. Bars are accuracy in the bin. Dots are mean confidence. The dashed line is perfect calibration.
Strands Decider 2B · shell_risk_safe
ECE 0.130 · n valid 200 / 200. Bars are accuracy in the bin. Dots are mean confidence. The dashed line is perfect calibration.
Strands Decider 2B · sms_spam
ECE 0.150 · n valid 200 / 200. Bars are accuracy in the bin. Dots are mean confidence. The dashed line is perfect calibration.
Majority class · sms_spam
ECE 0.000 · n valid 836 / 836. Bars are accuracy in the bin. Dots are mean confidence. The dashed line is perfect calibration.
Random · sms_spam
ECE 0.174 · n valid 836 / 836. Bars are accuracy in the bin. Dots are mean confidence. The dashed line is perfect calibration.
TF-IDF + logistic regression · sms_spam
ECE 0.068 · n valid 836 / 836. Bars are accuracy in the bin. Dots are mean confidence. The dashed line is perfect calibration.
Majority class · wiki_toxic
ECE 0.000 · n valid 500 / 500. Bars are accuracy in the bin. Dots are mean confidence. The dashed line is perfect calibration.
Random · wiki_toxic
ECE 0.203 · n valid 500 / 500. Bars are accuracy in the bin. Dots are mean confidence. The dashed line is perfect calibration.
TF-IDF + logistic regression · wiki_toxic
ECE 0.134 · n valid 500 / 500. Bars are accuracy in the bin. Dots are mean confidence. The dashed line is perfect calibration.
Strands Decider 2B · helpsteer2
ECE 0.015 · n valid 200 / 200. Bars are accuracy in the bin. Dots are mean confidence. The dashed line is perfect calibration.
Majority class · helpsteer2
ECE 0.000 · n valid 300 / 300. Bars are accuracy in the bin. Dots are mean confidence. The dashed line is perfect calibration.
Random · helpsteer2
ECE 0.150 · n valid 300 / 300. Bars are accuracy in the bin. Dots are mean confidence. The dashed line is perfect calibration.
TF-IDF + logistic regression · helpsteer2
ECE 0.040 · n valid 300 / 300. Bars are accuracy in the bin. Dots are mean confidence. The dashed line is perfect calibration.
Majority class · passage_relevance
ECE 0.000 · n valid 4 / 4. Bars are accuracy in the bin. Dots are mean confidence. The dashed line is perfect calibration.
Random · passage_relevance
ECE 0.385 · n valid 4 / 4. Bars are accuracy in the bin. Dots are mean confidence. The dashed line is perfect calibration.
TF-IDF + logistic regression · passage_relevance
ECE 0.120 · n valid 4 / 4. Bars are accuracy in the bin. Dots are mean confidence. The dashed line is perfect calibration.
Selective risk
Abstention thresholds 0.50, 0.60, 0.70, 0.80, 0.90, and 0.95 are in each run's summary.json under selective. The curves below are the risk-coverage view: keep the most confident slice, measure its error.
Strands Decider 2B · ag_news
Risk among the most confident fraction of schema-valid decisions. Confidence is the probability of the committed answer.
Majority class · ag_news
Risk among the most confident fraction of schema-valid decisions. Confidence is the probability of the committed answer.
Random · ag_news
Risk among the most confident fraction of schema-valid decisions. Confidence is the probability of the committed answer.
TF-IDF + logistic regression · ag_news
Risk among the most confident fraction of schema-valid decisions. Confidence is the probability of the committed answer.
Majority class · agent_gate
Risk among the most confident fraction of schema-valid decisions. Confidence is the probability of the committed answer.
Random · agent_gate
Risk among the most confident fraction of schema-valid decisions. Confidence is the probability of the committed answer.
TF-IDF + logistic regression · agent_gate
Risk among the most confident fraction of schema-valid decisions. Confidence is the probability of the committed answer.
Strands Decider 2B · agent_tool_routing
Risk among the most confident fraction of schema-valid decisions. Confidence is the probability of the committed answer.
Majority class · banking77
Risk among the most confident fraction of schema-valid decisions. Confidence is the probability of the committed answer.
Random · banking77
Risk among the most confident fraction of schema-valid decisions. Confidence is the probability of the committed answer.
TF-IDF + logistic regression · banking77
Risk among the most confident fraction of schema-valid decisions. Confidence is the probability of the committed answer.
Strands Decider 2B · helpsteer2
Risk among the most confident fraction of schema-valid decisions. Confidence is the probability of the committed answer.
Majority class · helpsteer2
Risk among the most confident fraction of schema-valid decisions. Confidence is the probability of the committed answer.
Random · helpsteer2
Risk among the most confident fraction of schema-valid decisions. Confidence is the probability of the committed answer.
TF-IDF + logistic regression · helpsteer2
Risk among the most confident fraction of schema-valid decisions. Confidence is the probability of the committed answer.
Majority class · passage_relevance
Risk among the most confident fraction of schema-valid decisions. Confidence is the probability of the committed answer.
Random · passage_relevance
Risk among the most confident fraction of schema-valid decisions. Confidence is the probability of the committed answer.
TF-IDF + logistic regression · passage_relevance
Risk among the most confident fraction of schema-valid decisions. Confidence is the probability of the committed answer.
Strands Decider 2B · prompt_injection
Risk among the most confident fraction of schema-valid decisions. Confidence is the probability of the committed answer.
Majority class · prompt_injection
Risk among the most confident fraction of schema-valid decisions. Confidence is the probability of the committed answer.
Random · prompt_injection
Risk among the most confident fraction of schema-valid decisions. Confidence is the probability of the committed answer.
TF-IDF + logistic regression · prompt_injection
Risk among the most confident fraction of schema-valid decisions. Confidence is the probability of the committed answer.
Strands Decider 2B · shell_risk_safe
Risk among the most confident fraction of schema-valid decisions. Confidence is the probability of the committed answer.
Strands Decider 2B · sms_spam
Risk among the most confident fraction of schema-valid decisions. Confidence is the probability of the committed answer.
Majority class · sms_spam
Risk among the most confident fraction of schema-valid decisions. Confidence is the probability of the committed answer.
Random · sms_spam
Risk among the most confident fraction of schema-valid decisions. Confidence is the probability of the committed answer.
TF-IDF + logistic regression · sms_spam
Risk among the most confident fraction of schema-valid decisions. Confidence is the probability of the committed answer.
Majority class · support_routing
Risk among the most confident fraction of schema-valid decisions. Confidence is the probability of the committed answer.
Random · support_routing
Risk among the most confident fraction of schema-valid decisions. Confidence is the probability of the committed answer.
TF-IDF + logistic regression · support_routing
Risk among the most confident fraction of schema-valid decisions. Confidence is the probability of the committed answer.
Majority class · tool_selection
Risk among the most confident fraction of schema-valid decisions. Confidence is the probability of the committed answer.
Random · tool_selection
Risk among the most confident fraction of schema-valid decisions. Confidence is the probability of the committed answer.
TF-IDF + logistic regression · tool_selection
Risk among the most confident fraction of schema-valid decisions. Confidence is the probability of the committed answer.
Majority class · wiki_toxic
Risk among the most confident fraction of schema-valid decisions. Confidence is the probability of the committed answer.
Random · wiki_toxic
Risk among the most confident fraction of schema-valid decisions. Confidence is the probability of the committed answer.
TF-IDF + logistic regression · wiki_toxic
Risk among the most confident fraction of schema-valid decisions. Confidence is the probability of the committed answer.