Please open-source the `generate_completions.py` used for the v2 completions

[mabuyun]

Please open-source the generate_completions.py used for the v2 completions

Thanks for releasing DevBench! The scoring side is fully reproducible —
running evaluation/compute_pass_at_1.py over the shipped completions/
reproduces the paper's numbers closely.

The generation side is not, because the script that produced the current
completions seems to be missing from the repo. So we can only re-score the
static files you shipped, not generate completions for a new model under the
same protocol or re-derive the paper's Pass@1 from scratch.

What's missing

• generate_completions.py is not in the tree at the pinned v2 revision
  (831495a, "hardened v2: 1,800 tasks, 9 current-gen models").
• The last version in git history targets an older model set (gpt-4o,
  gpt-4.1*, claude-3-7/4-sonnet, deepseek-v3/v3.1) — not the shipped v2
  models (gpt-5.5/5.4-mini/5.4-nano, claude-opus-4-7, claude-sonnet-4-6,
  deepseek-v4-pro, llama-4-maverick, mistral-medium-3.5, qwen3.6-27b).
• The README says the FIM prompt is "defined in evaluation/execute_benchmark.py",
  but that file only contains the execution/scoring harness — no
  prompt-construction or model-call code.

Why it blocks reproduction

With the documented max_tokens=800, current reasoning models spend the budget
on reasoning tokens and return empty content on many tasks (~40%+ empty in
my runs), whereas your shipped completions have a very low empty rate. So a
different generation setup must have been used, and we can't match it blindly.

Request

Could you open-source the exact generation script (or just specify it in the
README), including:

1. The exact per-language system + user / FIM prompt template.
2. Sampling params actually used, especially how reasoning models were
   handled (default settings? what max_tokens?).
3. Response parsing / code-extraction logic.
4. Empty-output/retry handling and the n=5 sampling details.

Even a lightly-cleaned version would make generation reproducible. Thanks!