Generation
456- 输入Sees
- 题面、约束、I/O 约定、限制、公开样例Statement, constraints, I/O spec, limits, public samples
- 输出Produces
- 解题程序A solution program
- 指标Metric
- pass@1(可选 pass@k):通过
score_case_ids全部测试pass@1 (optionally pass@k): passes every test inscore_case_ids
Polygon workflow · Release 2026-11
面向 Coding Agent 的时间留出评测集:保留 2026 年 6 月至 11 月 Codeforces 与 AtCoder 的 456 道标准输入输出题,按 Polygon 流程重新构建测试,提供 checker、程序池和五类任务定义。 A time-held-out benchmark for coding agents: 456 standard-I/O problems from June–November 2026 Codeforces and AtCoder contests, with tests rebuilt through the Polygon workflow, checkers, program pools, and five task definitions.
收录 2026-06-01 至 2026-11-16 期间已结束比赛中的全部非交互标准输入输出题,覆盖所有难度;题目实际发布日期为 2026-06-06 至 2026-11-15。 All non-interactive standard-I/O problems from contests that finished between 2026-06-01 and 2026-11-16, across every difficulty level. Actual release dates span 2026-06-06 to 2026-11-15.
| 平台Platform | 比赛类型Contest type | 题数Problems |
|---|---|---|
| Codeforces | Div. 1 | 42 |
| Codeforces | Div. 2 | 104 |
| Codeforces | Div. 1 + 2 | 35 |
| Codeforces | Educational | 40 |
| AtCoder | ABC | 167 |
| AtCoder | ARC | 61 |
| AtCoder | AGC | 7 |
| 合计Total | 456 | |
11 月仅统计至 11-15November covers releases through Nov 15
182 道有官方 rating;另有 39 道暂无 rating182 problems with an official rating; 39 have none yet
全部 456 题;标签来自平台或由 Astra 补充,来源见 tag_detailsAll 456 problems; tags come from the platform or are added by Astra — see tag_details
official_test_suite_complete 均为 false,不代表平台官方隐藏测试全集。Supplementary tests are project-generated. Every problem has official_test_suite_complete: false; this is not the platforms’ complete official hidden suite.uid,并通过 family_id 关联。Easy / Hard versions keep separate uids and are linked by family_id.
能力划分参考 LiveCodeBench。Tasks/ 中是按可见性规则导出的模型输入;评分所需的隐藏材料保留在各题包中,通过 task_id 对应。
The ability split follows LiveCodeBench. Tasks/ holds model-facing inputs exported under visibility rules; hidden grading material stays in each problem package and is joined via task_id.
score_case_ids 全部测试pass@1 (optionally pass@k): passes every test in score_case_idsGeneration — 每题 n 次独立采样、c 次通过:with n independent samples per problem and c passing:
pass@k = mean_over_problems(1 − C(n−c, k) / C(n, k)), n ≥ k
Repair — 修复后通过全部评分测试的实例数 / 初始确实失败的实例数。多轮修复需固定轮数或成本预算。instances passing all scoring tests after repair / instances that initially fail. Multi-round repair should fix a round or cost budget.
Prediction — 每题最多 3 份非样例输入,按 token 序列去重;构造题接受任意合法答案。up to 3 non-sample inputs per problem, deduplicated by token sequence; any valid answer is accepted for constructive problems.
Execution — 真值是指定程序的实际 stdout;1 与 1.0 不等价。ground truth is the program's actual stdout; 1 and 1.0 are different.
Testcase Gen — 输入先过 validator,再运行标准解、目标程序和 checker。预算:每实例最多 10 份输入、合计 16 MiB、generator 超时 20 s。inputs must pass the validator, then std, the target program and the checker are run. Budget: ≤ 10 inputs, 16 MiB total, 20 s generator timeout per instance.
数据采用 format 2.0,任务协议为 stdio_v2。每道题一个目录,目录名形如 atcoder_abc461_a,对应 UID atcoder:abc461_a。
Format 2.0, task protocol stdio_v2. One directory per problem, e.g. atcoder_abc461_a for UID atcoder:abc461_a.
OOD-Test-v2/
├── README.md
├── index.html
├── Tasks/
│ ├── generation.json
│ ├── repair.json
│ ├── prediction.json
│ ├── execution.json
│ └── testcase_gen.json
└── problems/
└── <platform>_<problem_id>/
├── description.json
├── cases.json
├── tasks.json
├── expected_verdict.json
├── generator.cpp
├── validator.cpp
├── std.cpp
├── checker.cpp
└── solutions/
└── *.cpp / *.py| 文件File | 内容Contents |
|---|---|
description.json | 题面、约束、tag、rating、题解、限制、运行环境与来源Statement, constraints, tags, rating, editorial, limits, runtime, provenance |
cases.json | 完整 stdin 与标准程序的实际输出Full stdin strings and the reference program's stdout |
tasks.json | 五类任务的可见性配置与评分引用Visibility and scoring config for all five tasks |
expected_verdict.json | 每个程序在每个测试上的预期 verdictExpected verdict of every program on every test |
generator.cpp | 按 cases 中 source.argv 的完整参数重放输入Replays one full input using cases source.argv |
validator.cpp | 读 stdin,退出码 0 表示输入合法Reads stdin; exit code 0 means valid |
std.cpp | C++17 标准解C++17 reference solution |
checker.cpp | checker input output answer;退出码 0 表示接受; exit code 0 means accepted |
solutions/ | 额外的正确或错误程序(C++ / Python)Extra correct or buggy programs (C++ / Python) |
cases.json 示例A cases.json record{
"case_id": "t0000",
"kind": "sample",
"coverage": "official sample",
"source": {
"origin": "source_snapshot",
"url": "https://atcoder.jp/contests/abc461/tasks/abc461_a",
"upstream_case_id": "t0000",
"published_sample_output": "Yes\n",
"input_sha256": "9d63d0ac62176acc5977b66200ca95fbff99307c57a1cff0bc7d445ee1672dde",
"expected_output_sha256": "927580bf287560d4a26a5d6e129f91ffa9cb63031b1a8c772be6ec584e65e82b",
"expected_output_origin": "native_standard_stdout"
},
"input": "4 5\n",
"expected_output": "Yes\n"
}
一份测试输入是一次完整的 stdin,可包含多个题内 test case。生成记录的 source.argv 保存完整重放参数,coverage 记录覆盖意图。原始样例输出保存在 source.published_sample_output;expected_output 是标准程序的实测输出。构造题请使用本题 checker.cpp 判定。
One test input is one complete stdin and may contain several in-problem test cases. Generated records store full replay arguments in source.argv and coverage intent in coverage. Original sample outputs are preserved in source.published_sample_output; expected_output is measured reference stdout. Judge constructive answers with the problem’s checker.cpp.
default_visibility→tasks.<task>.visibility→case_overrides
后者覆盖前者,未开放的字段默认隐藏。题解、tag、标准解、其他程序和目标输出均可分别控制。 Later levels override earlier ones; anything not opened stays hidden. Editorials, tags, the reference solution, other programs and target outputs are controlled independently.
原题限制与数据集执行限制分别存于 limits.original 与 limits.dataset_execution。时限只对较慢的机器放宽,上限 20 秒:
Original and dataset limits are stored in limits.original and limits.dataset_execution. Time limits are only relaxed on slower machines, capped at 20 s:
multiplier = max(1, 4e8 / measured_rate) timeout_seconds = min(20, original_time_seconds × multiplier)
本批六个原生判题实例测得 1.2565e9–1.2780e9 次/秒,均快于 4e8,倍率均为 1;所有题保留原题 CPU 时限,不按更快的硬件缩短。 The six native judge instances measured 1.2565e9–1.2780e9 iterations/s, all above 4e8. Every multiplier is 1: original CPU time limits are retained and are never shortened for faster hardware.
| C++ 标准与编译参数C++ standard & flags | C++17, -O2 -static -pipe -DDOMJUDGE |
| 解答程序内存Solution memory | max(512 MiB, original),由 DOMjudge judgehost 实施, enforced by DOMjudge judgehost |
| 单个输出文件上限Output file limit | 64 MiB |
| 编译超时Compile timeout | 90 s |
| 原生 / 独立复验编译内存Native / QA compile memory | 2 GiB / 1 GiB |
| 判题环境Judge | DOMjudge 9.0.0,Polygon-Replica 原生 Verification;另用 upsolve-judge:1 独立复验, native Polygon-Replica Verification; separate QA with upsolve-judge:1 |
| testlib | 独立编译使用 runtime.standalone_testlib;原生判题使用 runtime.native_testlib 的固定版本Pinned versions in runtime.standalone_testlib for standalone builds and runtime.native_testlib for native judging |
下载数据集,并用 Python 标准库读取一道题及其 Generation 输入。 Download the dataset and read one problem plus its Generation input with the Python standard library.
hf download Sy-SU/OOD-Test-v2 --repo-type dataset --local-dir OOD-Test-v2
import json
from pathlib import Path
root = Path("OOD-Test-v2")
folder = root / "problems" / "atcoder_abc461_a"
def read_json(path):
with path.open(encoding="utf-8") as stream:
return json.load(stream)
description = read_json(folder / "description.json")
cases = read_json(folder / "cases.json")["cases"]
tasks = read_json(folder / "tasks.json")["tasks"]
print(description["uid"], description["title"])
print("test inputs:", len(cases))
print("generation scoring tests:", tasks["generation"]["instances"][0]["score_case_ids"])
model_inputs = read_json(root / "Tasks" / "generation.json")
task = next(item for item in model_inputs
if item["task_id"] == description["uid"] + ":generation")
print(task["description"]["statement"]["text"])
模型请求只使用 Tasks/ 中的可见输入;评测器再按 tasks.json 读取隐藏的评分材料。prediction.json 约 75.6 MiB,execution.json 约 70.9 MiB;批量处理可流式解析(如 ijson)。
Build model requests from Tasks/ only; the evaluator reads hidden grading material via tasks.json. prediction.json is about 75.6 MiB and execution.json about 70.9 MiB; stream parsing (e.g. ijson) is also available for batch runs.
description.json。Generators and validators for all 456 problems were newly written by GPT-6 Astra. Reference solutions, checkers, and extra programs are inherited from the previous release and rejudged on the new tests; official, adapted, and newly authored origins are recorded in description.json.stdio_v2 协议,结果请注明 “OOD-Test-v2 2026-11” 及资源设置,与官方 LiveCodeBench / Pro 分数分开报告。本项目是独立的数据集扩展。OOD-Test-v2 uses its own tests, program pools and the stdio_v2 protocol. Please report results as “OOD-Test-v2 2026-11” with resource settings, separately from official LiveCodeBench / Pro scores. This is an independent dataset extension.@misc{ood_test_v2_2026,
title = {OOD-Test-v2: A Time-Held-Out Coding Agent Benchmark with Polygon-Validated Tests},
author = {Senyang Su},
year = {2026},
howpublished = {\url{https://huggingface.co/datasets/Sy-SU/OOD-Test-v2}}
}