OOD-Test-v2

Polygon workflow · Release 2026-11

OOD-Test-v2

面向 Coding Agent 的时间留出评测集:保留 2026 年 6 月至 11 月 Codeforces 与 AtCoder 的 456 道标准输入输出题,按 Polygon 流程重新构建测试,提供 checker、程序池和五类任务定义。 A time-held-out benchmark for coding agents: 456 standard-I/O problems from June–November 2026 Codeforces and AtCoder contests, with tests rebuilt through the Polygon workflow, checkers, program pools, and five task definitions.

456
题目(CF 221 / AtCoder 235)problems (CF 221 / AtCoder 235)
21,236
完整 stdin(含 779 份公开样例)stdin inputs (779 public samples)
916
额外参考 / 错误程序extra reference / buggy programs
4,084
任务实例,覆盖 5 类能力task instances across 5 abilities

数据构成Composition

收录 2026-06-01 至 2026-11-16 期间已结束比赛中的全部非交互标准输入输出题,覆盖所有难度;题目实际发布日期为 2026-06-06 至 2026-11-15。 All non-interactive standard-I/O problems from contests that finished between 2026-06-01 and 2026-11-16, across every difficulty level. Actual release dates span 2026-06-06 to 2026-11-15.

平台Platform比赛类型Contest type题数Problems
CodeforcesDiv. 142
CodeforcesDiv. 2104
CodeforcesDiv. 1 + 235
CodeforcesEducational40
AtCoderABC167
AtCoderARC61
AtCoderAGC7
合计Total456

按发布月份Problems by release month

11 月仅统计至 11-15November covers releases through Nov 15

AtCoderCodeforces
表格视图Table view

Codeforces 难度分布Codeforces rating distribution

182 道有官方 rating;另有 39 道暂无 rating182 problems with an official rating; 39 have none yet

表格视图Table view

高频标签(前 12)Most frequent tags (top 12)

全部 456 题;标签来自平台或由 Astra 补充,来源见 tag_detailsAll 456 problems; tags come from the platform or are added by Astra — see tag_details

表格视图Table view
  • 共 473 道候选题,其中 16 道交互题与 1 道 output-only 题(ARC226E)未收录。Of 473 candidates, 16 interactive problems and 1 output-only problem (ARC226E) are excluded.
  • 与 上一版 使用相同的 456 个题目 UID。新版保留 779 份原题公开样例,新增 20,457 份可重放的生成输入;每题包含 2–601 份完整 stdin。The same 456 problem UIDs as the previous release. This version preserves 779 official public samples and adds 20,457 reproducible generated inputs, with 2–601 full stdin inputs per problem.
  • 在固定的 912 个独立合成错误机制上,旧套件检出 773 个,新套件检出 870 个:新增检出 97、退步 0、仍漏检 42;正确对照误杀为 0/916。这是测试套件强度实验,尚未报告模型通过率或调用成本。On 912 fixed, independently authored synthetic bug mechanisms, detection rises from 773 to 870: 97 new detections, 0 regressions, and 42 still missed; false rejection is 0/916 correct controls. These measure test-suite strength; model pass rates and API costs have not been evaluated.
  • 补充测试由本项目生成,所有题的 official_test_suite_complete 均为 false,不代表平台官方隐藏测试全集。Supplementary tests are project-generated. Every problem has official_test_suite_complete: false; this is not the platforms’ complete official hidden suite.
  • Easy / Hard 版本各自拥有独立 uid,并通过 family_id 关联。Easy / Hard versions keep separate uids and are linked by family_id.
  • “OOD” 指按题目发布时间留出,不保证所有模型均未见过这些题。报告结果时请注明模型版本与已知训练截止日期。“OOD” denotes a temporal hold-out by release date, not a guarantee that every model is unfamiliar with these problems. Report the model version and known training cutoff.

五类任务Five tasks

能力划分参考 LiveCodeBench。Tasks/ 中是按可见性规则导出的模型输入;评分所需的隐藏材料保留在各题包中,通过 task_id 对应。 The ability split follows LiveCodeBench. Tasks/ holds model-facing inputs exported under visibility rules; hidden grading material stays in each problem package and is joined via task_id.

Generation

456
输入Sees
题面、约束、I/O 约定、限制、公开样例Statement, constraints, I/O spec, limits, public samples
输出Produces
解题程序A solution program
指标Metric
pass@1(可选 pass@k):通过 score_case_ids 全部测试pass@1 (optionally pass@k): passes every test in score_case_ids
Tasks/generation.json

Repair

456
输入Sees
题面、错误程序、失败反馈(verdict 与程序实际输出)Statement, a buggy program, failure feedback (verdict and actual output)
输出Produces
修复后的程序A fixed program
指标Metric
修复成功率;反馈测试与评分测试相互独立Repair success rate; feedback and scoring tests are disjoint
Tasks/repair.json

Prediction

1,358
输入Sees
题面与一份指定输入Statement and one given input
输出Produces
符合题意的输出A correct output for that input
指标Metric
macro / micro accuracy,由题目 checker 判定Macro / micro accuracy, judged by the problem's checker
Tasks/prediction.json · 75.6 MiB

Execution

1,358
输入Sees
一段具体程序、指定输入与运行环境A concrete program, a given input, and the runtime
输出Produces
该程序的实际 stdoutThat program's actual stdout
指标Metric
accuracy:按空白分隔的 token 序列精确匹配Accuracy: exact match on whitespace-separated tokens
Tasks/execution.json · 70.9 MiB

Testcase Gen

456
输入Sees
题面、目标程序、公开样例、生成预算Statement, target program, public samples, a generation budget
输出Produces
测试输入,或输出测试的 generatorTest inputs, or a generator that emits them
指标Metric
输入合法率、错误检出率、误杀率Input validity, bug detection rate, false-kill rate
Tasks/testcase_gen.json

评价指标Metrics

Generation — 每题 n 次独立采样、c 次通过:with n independent samples per problem and c passing:

pass@k = mean_over_problems(1 − C(n−c, k) / C(n, k)),  n ≥ k

Repair — 修复后通过全部评分测试的实例数 / 初始确实失败的实例数。多轮修复需固定轮数或成本预算。instances passing all scoring tests after repair / instances that initially fail. Multi-round repair should fix a round or cost budget.

Prediction — 每题最多 3 份非样例输入,按 token 序列去重;构造题接受任意合法答案。up to 3 non-sample inputs per problem, deduplicated by token sequence; any valid answer is accepted for constructive problems.

Execution — 真值是指定程序的实际 stdout;1 与 1.0 不等价。ground truth is the program's actual stdout; 1 and 1.0 are different.

Testcase Gen — 输入先过 validator,再运行标准解、目标程序和 checker。预算:每实例最多 10 份输入、合计 16 MiB、generator 超时 20 s。inputs must pass the validator, then std, the target program and the checker are run. Budget: ≤ 10 inputs, 16 MiB total, 20 s generator timeout per instance.

数据格式Data format

数据采用 format 2.0,任务协议为 stdio_v2。每道题一个目录,目录名形如 atcoder_abc461_a,对应 UID atcoder:abc461_a。 Format 2.0, task protocol stdio_v2. One directory per problem, e.g. atcoder_abc461_a for UID atcoder:abc461_a.

OOD-Test-v2/
├── README.md
├── index.html
├── Tasks/
│   ├── generation.json
│   ├── repair.json
│   ├── prediction.json
│   ├── execution.json
│   └── testcase_gen.json
└── problems/
    └── <platform>_<problem_id>/
        ├── description.json
        ├── cases.json
        ├── tasks.json
        ├── expected_verdict.json
        ├── generator.cpp
        ├── validator.cpp
        ├── std.cpp
        ├── checker.cpp
        └── solutions/
            └── *.cpp / *.py
文件File内容Contents
description.json题面、约束、tag、rating、题解、限制、运行环境与来源Statement, constraints, tags, rating, editorial, limits, runtime, provenance
cases.json完整 stdin 与标准程序的实际输出Full stdin strings and the reference program's stdout
tasks.json五类任务的可见性配置与评分引用Visibility and scoring config for all five tasks
expected_verdict.json每个程序在每个测试上的预期 verdictExpected verdict of every program on every test
generator.cpp按 cases 中 source.argv 的完整参数重放输入Replays one full input using cases source.argv
validator.cpp读 stdin,退出码 0 表示输入合法Reads stdin; exit code 0 means valid
std.cppC++17 标准解C++17 reference solution
checker.cppchecker input output answer;退出码 0 表示接受; exit code 0 means accepted
solutions/额外的正确或错误程序(C++ / Python)Extra correct or buggy programs (C++ / Python)

cases.json 示例A cases.json record

{
  "case_id": "t0000",
  "kind": "sample",
  "coverage": "official sample",
  "source": {
    "origin": "source_snapshot",
    "url": "https://atcoder.jp/contests/abc461/tasks/abc461_a",
    "upstream_case_id": "t0000",
    "published_sample_output": "Yes\n",
    "input_sha256": "9d63d0ac62176acc5977b66200ca95fbff99307c57a1cff0bc7d445ee1672dde",
    "expected_output_sha256": "927580bf287560d4a26a5d6e129f91ffa9cb63031b1a8c772be6ec584e65e82b",
    "expected_output_origin": "native_standard_stdout"
  },
  "input": "4 5\n",
  "expected_output": "Yes\n"
}

一份测试输入是一次完整的 stdin,可包含多个题内 test case。生成记录的 source.argv 保存完整重放参数,coverage 记录覆盖意图。原始样例输出保存在 source.published_sample_output;expected_output 是标准程序的实测输出。构造题请使用本题 checker.cpp 判定。 One test input is one complete stdin and may contain several in-problem test cases. Generated records store full replay arguments in source.argv and coverage intent in coverage. Original sample outputs are preserved in source.published_sample_output; expected_output is measured reference stdout. Judge constructive answers with the problem’s checker.cpp.

可见性规则Visibility rules

default_visibility→tasks.<task>.visibility→case_overrides

后者覆盖前者,未开放的字段默认隐藏。题解、tag、标准解、其他程序和目标输出均可分别控制。 Later levels override earlier ones; anything not opened stays hidden. Editorials, tags, the reference solution, other programs and target outputs are controlled independently.

执行环境Execution environment

原题限制与数据集执行限制分别存于 limits.original 与 limits.dataset_execution。时限只对较慢的机器放宽,上限 20 秒: Original and dataset limits are stored in limits.original and limits.dataset_execution. Time limits are only relaxed on slower machines, capped at 20 s:

multiplier      = max(1, 4e8 / measured_rate)
timeout_seconds = min(20, original_time_seconds × multiplier)

本批六个原生判题实例测得 1.2565e9–1.2780e9 次/秒,均快于 4e8,倍率均为 1;所有题保留原题 CPU 时限,不按更快的硬件缩短。 The six native judge instances measured 1.2565e9–1.2780e9 iterations/s, all above 4e8. Every multiplier is 1: original CPU time limits are retained and are never shortened for faster hardware.

C++ 标准与编译参数C++ standard & flagsC++17, -O2 -static -pipe -DDOMJUDGE
解答程序内存Solution memorymax(512 MiB, original),由 DOMjudge judgehost 实施, enforced by DOMjudge judgehost
单个输出文件上限Output file limit64 MiB
编译超时Compile timeout90 s
原生 / 独立复验编译内存Native / QA compile memory2 GiB / 1 GiB
判题环境JudgeDOMjudge 9.0.0,Polygon-Replica 原生 Verification;另用 upsolve-judge:1 独立复验, native Polygon-Replica Verification; separate QA with upsolve-judge:1
testlib独立编译使用 runtime.standalone_testlib;原生判题使用 runtime.native_testlib 的固定版本Pinned versions in runtime.standalone_testlib for standalone builds and runtime.native_testlib for native judging

快速开始Quick start

下载数据集,并用 Python 标准库读取一道题及其 Generation 输入。 Download the dataset and read one problem plus its Generation input with the Python standard library.

hf download Sy-SU/OOD-Test-v2 --repo-type dataset --local-dir OOD-Test-v2
import json
from pathlib import Path

root = Path("OOD-Test-v2")
folder = root / "problems" / "atcoder_abc461_a"

def read_json(path):
    with path.open(encoding="utf-8") as stream:
        return json.load(stream)

description = read_json(folder / "description.json")
cases = read_json(folder / "cases.json")["cases"]
tasks = read_json(folder / "tasks.json")["tasks"]

print(description["uid"], description["title"])
print("test inputs:", len(cases))
print("generation scoring tests:", tasks["generation"]["instances"][0]["score_case_ids"])

model_inputs = read_json(root / "Tasks" / "generation.json")
task = next(item for item in model_inputs
            if item["task_id"] == description["uid"] + ":generation")
print(task["description"]["statement"]["text"])

模型请求只使用 Tasks/ 中的可见输入;评测器再按 tasks.json 读取隐藏的评分材料。prediction.json 约 75.6 MiB,execution.json 约 70.9 MiB;批量处理可流式解析(如 ijson)。 Build model requests from Tasks/ only; the evaluator reads hidden grading material via tasks.json. prediction.json is about 75.6 MiB and execution.json about 70.9 MiB; stream parsing (e.g. ijson) is also available for batch runs.

来源与许可Sources & license

  • 题面与题解来自 Codeforces / AtCoder 官方页面及公开数据,部分题面语料来自 CPRet 固定快照。Statements and editorials come from official Codeforces / AtCoder pages and public data, including a pinned CPRet snapshot.
  • 全部 456 题的 generator 和 validator 均由 GPT-6 Astra 重新编写。标准解、checker 与额外程序沿用上一版,经新版测试重新判题;官方代码、移植改写与新编写程序的来源见 description.json。Generators and validators for all 456 problems were newly written by GPT-6 Astra. Reference solutions, checkers, and extra programs are inherited from the previous release and rejudged on the new tests; official, adapted, and newly authored origins are recorded in description.json.
  • 使用 Polygon-Skills 的五路独立审查和 Polygon-Replica 原生 Verification,检查合法性、边界、独立 oracle、checker 和封存错误程序;生成输入全部重放两次。修订后重新封存独立程序评分,公开反馈只进入回归对照。The build uses five independent review roles from Polygon-Skills and native Polygon-Replica Verification: legality, boundaries, independent oracles, checker controls, and sealed incorrect programs. All generated inputs were replayed twice. Revisions are scored on fresh sealed programs; disclosed feedback becomes regression controls.
  • 题面版权归原平台及作者所有;本数据集新增的测试、程序和标注仅供研究使用。Problem statements remain the property of their platforms and authors; the tests, programs and annotations added here are released for research use.
  • 本数据集使用自定义测试、程序池与 stdio_v2 协议,结果请注明 “OOD-Test-v2 2026-11” 及资源设置,与官方 LiveCodeBench / Pro 分数分开报告。本项目是独立的数据集扩展。OOD-Test-v2 uses its own tests, program pools and the stdio_v2 protocol. Please report results as “OOD-Test-v2 2026-11” with resource settings, separately from official LiveCodeBench / Pro scores. This is an independent dataset extension.

引用Citation

@misc{ood_test_v2_2026,
  title        = {OOD-Test-v2: A Time-Held-Out Coding Agent Benchmark with Polygon-Validated Tests},
  author       = {Senyang Su},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/datasets/Sy-SU/OOD-Test-v2}}
}