Submit a backend

A submission is a run directory: config, summary, and the gzipped decision log. The leaderboard is rebuilt from those files. There is no place to type in a score.

Add an adapter that answers the same state and questions every other backend sees. The HTTP adapters speak POST /v1/systemone. A classical or prompted baseline should emit that same answer shape so schema validity stays comparable.
Fit supervised models on the train split only. Do not edit question text after looking at test metrics. A wording change belongs in a new suite version.
Run the frozen test split with probes left on. Prefer the full split. Runs made with --limit are published but labeled partial, with the item count, and shaded on every table.
Put the run directory under results/published/ and rebuild the site. The log has to contain the raw request, the raw response, the model version, timestamps, hardware, and the config.

Hosted Jev

export TYPESAFE_API_KEY=...
# optional: TYPESAFE_BASE_URL=https://api.typesafe.ai
# optional: TYPESAFE_MODEL=jev-latest
uv run s1bench run --backend jev --suite all --out runs/jev

If TYPESAFE_API_KEY is missing the command skips and writes a pending note. It does not invent a score. Optional cost accounting, in USD per million tokens: S1BENCH_JEV_PRICE_PER_MILLION_INPUT and S1BENCH_JEV_PRICE_PER_MILLION_OUTPUT. Without them, cost stays blank.

Local Kev

git clone https://github.com/jaredpalmer/kev.git
cd kev
uv sync --extra serve
uv run --extra serve python -m kev.serve --run jaredpalmer/kev-4b --port 8009

# from this repository, in another shell
export S1BENCH_KEV_BASE_URL=http://127.0.0.1:8009
# optional if the server was started with KEV_API_KEY
export KEV_API_KEY=...
uv run s1bench run --backend kev --suite all --out runs/kev

Local Laya

uv pip install "laya[serve]"
LAYA_HOST=127.0.0.1 LAYA_PORT=8000 LAYA_MODELS=english laya-serve

export S1BENCH_LAYA_BASE_URL=http://127.0.0.1:8000
uv run s1bench run --backend laya --suite all --out runs/laya

Any other System One server

export S1BENCH_GENERIC_BASE_URL=http://127.0.0.1:9000
export S1BENCH_GENERIC_MODEL=your-model-name
# optional: S1BENCH_GENERIC_API_KEY=...
uv run s1bench run --backend generic --suite all --out runs/generic

This covers Jevlike, jev-style, ollaya, OpenJev servers, and anything else that implements the same endpoint. Point the base URL at it. Do not transcribe that project's README accuracy into this table.

OpenAI-compatible LLM

export S1BENCH_LLM_BASE_URL=https://api.openai.com/v1
export S1BENCH_LLM_API_KEY=...
export S1BENCH_LLM_MODEL=gpt-4.1-mini
# optional, USD per million tokens:
# export S1BENCH_LLM_PRICE_PER_MILLION_INPUT=...
# export S1BENCH_LLM_PRICE_PER_MILLION_OUTPUT=...
uv run s1bench run --backend llm --suite all --out runs/llm

Rebuild

uv run s1bench report --runs runs/jev
uv run s1bench leaderboard build --from results/published --out site/dist

Copy the new run directory into results/published before the build if you want it on the site. Full adapter and suite instructions are in Methodology and the repository CONTRIBUTING.md.