Contribute
Turn research you have already done into a benchmark task
Earn 12 points, become a co-author
A merged task you authored earns 6, one you reviewed earns 2, and referring a contributor earns 2 once their first task merges.
Reviewing opens up once you have your first good task merged — ask a maintainer to be added as a reviewer.
From the team behind SkillsBench, 170+ citations within 6 months of release.
Timeline
We will submit to ICLR first and then submit to Nature after further polish.
- v0.1
Get tasks merged by 31 August to join the author list of ICLR (and all future paper versions).
- v1.0
Get tasks merged by 31 December to join the author list of the draft submitted to Nature.
Merged, not opened — reviewing and revising a task takes days of back-and-forth, so a PR opened close to a deadline is unlikely to land in time.
Getting started
Join the contributors team
Fill in the join form to get access to the group chats, the shared Google Drive, and the GitHub repository.
Contribute a task — two ways
- 1.Fill in the task form and co-work with reviewers in email threads.
- 2.Send a PR that adds a task to the task repository, following the example PR #23 there. The repository is private — the join form in step 01 grants access.
Who should contribute
- A PhD or current PhD candidate in physics, EECS, or an adjacent field
- Or extensive hands-on experience in a physics lab or an equivalent industry role
What makes an ideal task
All three. A task that misses any one will not merge.
Your own work
Real research you carried out, not a problem invented for the benchmark.
Did I do this myself?
Weeks of effort
At least two weeks of genuine effort, with or without an agent helping.
Did it really take weeks to finish?
Verifiable
The result is right or wrong, and a script can tell which.
Can a script grade it?
Two stages, two graders
Every task is graded in two stages, and you write the grader for each.
Deep research
The agent studies the problem and commits to a research plan. A planning rubric that ships with the task grades that plan.
Execution
The agent carries the plan out. The verifier checks that the final results are accurate.
Key parts of a task
task.md
The task description, in two parts.
- 1.Research — Reviewing literature and making plans. The plan and thinking process are evaluated based on rubrics written by the contributor.
- 2.Implementation — The concrete problem-solving request that can be verified by code scripts.
rubric.json
The item-by-item list of rubrics that describe the expectations from the researchers. It mainly focuses on the research and planning parts that are not verifiable via code scripts.
verifier
The verifier logic for checking the agent's final deliverables. Code scripts for checking verifiable results.
oracle
The ground truth answer provided by the contributor. Always gets reward == 1 on the verifier.
The final submission
Three things. Open it as a draft long before it is finished — reviewing and revising a task takes days of back-and-forth, and a draft is the cheapest way to find out early that an idea will not clear the bar.
- 01
A PR from your fork
Fork this repository and open a pull request against main here. One task per PR, touching only files under tasks/<task-id>/.
- 02
A detailed PR description
What the original work was, the physics it exercises, and where the data came from — plus a table reporting its history against these minimums.
Report Minimum Project time scale — start and end date 2 weeks Actual working hours spent exploring the task 40 hours Estimated hours for a first-year PhD to reproduce it 10 hours - 03
A local test results report
What you ran and what happened, across multiple trials rather than a single run. Example task: PR #23 in the task repository.
- The oracle passes with reward exactly 1.0
- Results for a state-of-the-art agent with skills
- Results for the same agent without skills