Robustness
Each probe is a paired comparison against the unperturbed test item. Decision-flip rate is the share of schema-valid pairs whose committed decision changed. Probability drift is total variation for Choice and Score, and the absolute change in the yes-probability for Noul. Score level order is the meaning of the rubric, so level lists are not shuffled.
What the probes are. Order reverse and order shuffle change only the presentation of Choice options or of the Noul true/false criteria. Paraphrase swaps in a frozen alternate wording. Irrelevant context appends a fixed paragraph that does not bear on the label. The adversarial probe appends an instruction that demands the alphabetically first choice, a yes, or the highest score level, whether or not that is the gold label. Attack success is how often the model obeys that demand. It is a stress test of the decision, not a procedure for attacking a deployed system.
| Backend | Suite | order reverse flip | drift | order shuffle flip | drift | paraphrase flip | drift | irrelevant context flip | drift | adversarial flip | drift | attack success |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Strands Decider 2B | ag_news | 0.000 | 0.036 | 0.024 | 0.018 | 0.073 | 0.065 | 0.024 | 0.032 | 0.000 | 0.067 | 0.244 |
| Majority class | ag_news | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 1.000 |
| Random | ag_news | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.340 |
| TF-IDF + logistic regression | ag_news | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.080 | 0.083 | 0.145 | 0.115 | 0.230 |
| Majority class | agent_gate | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 1.000 |
| Random | agent_gate | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.500 |
| TF-IDF + logistic regression | agent_gate | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.333 | 0.027 | 0.167 | 0.008 | 0.500 |
| Strands Decider 2B | agent_tool_routing | 0.488 | 0.332 | 0.439 | 0.286 | 0.293 | 0.265 | 0.195 | 0.133 | 0.683 | 0.474 | 0.659 |
| Majority class | banking77 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 |
| Random | banking77 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.025 |
| TF-IDF + logistic regression | banking77 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.135 | 0.362 | 0.060 | 0.297 | 0.010 |
| Strands Decider 2B | helpsteer2 | — | — | — | — | 0.454 | 0.080 | 0.208 | 0.028 | 0.592 | 0.137 | 0.085 |
| Majority class | helpsteer2 | — | — | — | — | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 |
| Random | helpsteer2 | — | — | — | — | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.280 |
| TF-IDF + logistic regression | helpsteer2 | — | — | — | — | 0.000 | 0.000 | 0.230 | 0.027 | 0.135 | 0.016 | 0.260 |
| Majority class | passage_relevance | — | — | — | — | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 |
| Random | passage_relevance | — | — | — | — | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 |
| TF-IDF + logistic regression | passage_relevance | — | — | — | — | 0.000 | 0.000 | 0.250 | 0.020 | 0.500 | 0.017 | 0.000 |
| Strands Decider 2B | prompt_injection | 0.000 | 0.000 | 0.000 | 0.000 | 0.129 | 0.168 | 0.095 | 0.109 | 0.819 | 0.505 | 1.000 |
| Majority class | prompt_injection | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 |
| Random | prompt_injection | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.483 |
| TF-IDF + logistic regression | prompt_injection | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.043 | 0.088 | 0.586 | 0.245 | 0.888 |
| Strands Decider 2B | shell_risk_safe | 0.000 | 0.000 | 0.000 | 0.000 | 0.736 | 0.219 | 0.038 | 0.023 | 0.415 | 0.125 | 0.509 |
| Strands Decider 2B | sms_spam | 0.000 | 0.000 | 0.000 | 0.000 | 0.079 | 0.055 | 0.000 | 0.034 | 0.000 | 0.066 | 0.132 |
| Majority class | sms_spam | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 |
| Random | sms_spam | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.490 |
| TF-IDF + logistic regression | sms_spam | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.020 | 0.034 | 0.000 | 0.043 | 0.080 |
| Majority class | support_routing | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 1.000 |
| Random | support_routing | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.222 |
| TF-IDF + logistic regression | support_routing | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.333 | 0.037 | 0.389 | 0.033 | 0.000 |
| Majority class | tool_selection | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 1.000 |
| Random | tool_selection | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.250 |
| TF-IDF + logistic regression | tool_selection | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.812 | 0.050 | 0.875 | 0.037 | 0.000 |
| Majority class | wiki_toxic | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 1.000 |
| Random | wiki_toxic | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.515 |
| TF-IDF + logistic regression | wiki_toxic | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.430 | 0.185 | 0.200 | 0.065 | 0.280 |