A research pipeline for predicting Codeforces problem ratings from official API metadata, solved statistics, exposure-aware features, and lightweight statement-structure features. Historical paper artifacts remain available for transparent comparison with the corrected evaluation pipeline.
Research status (updated 2026-07-12): Earlier paper results are retained for transparency and should be interpreted as retrospective. A code audit identified a split-configuration mismatch and test-informed model selection; the corrected controls and scope are documented in the public erratum. No corrected rerun has replaced the legacy full-API headline metrics. A separate locked historical statement-only backtest is now reported below.
If you only want the main idea, key results, and how to read the project quickly, see QUICK_OVERVIEW.md.
Cold-start note: Cold-start here means no solved-count behavior, not necessarily strict pre-contest prediction. Tags, metadata, and statement availability may not exactly match a real pre-contest setting.
Useful companion notes:
| Experiment | Main setting | Legacy reported result | Current interpretation |
|---|---|---|---|
| Contest-grouped prediction | HGB full API | MAE 166.9, within +/-200 = 69.7% | Legacy descriptive evaluation |
| Forward-time prediction | RF full API | MAE 152.5, within +/-200 = 71.2% | Legacy descriptive evaluation |
| Solved-only baseline | solved-count only | MAE 227-274 | Hypothesis-generating comparison |
| Cold-start prediction | metadata only | MAE 318-332 | Legacy descriptive evaluation |
| Statement-structure extension | metadata + text-light | MAE improves by 33.0-42.3 | Requires future confirmation |
| Rolling temporal validation | full API + age norm | average MAE 146.0 | Internal temporal analysis, not the future test |
A separately frozen forward-time retrospective backtest compared problem-index features with the same features plus a fixed statement-structure allowlist. On the committed locked test split (2,468 problems from 355 contests), MAE improved from 477.2151 to 401.2704: a difference of -75.9447 (about 15.9%), with a paired contest-cluster bootstrap 95% interval of [-86.7339, -65.0783].
This is not a prospective result: pages were collected after their contests, the cache is a normalized local representation rather than raw HTTP bytes, and the historical data had prior project exposure. It is a separate statement-only result and does not replace the legacy full-API headline metrics. See the full results, evidence boundary, and committed result artifacts.
How well can Codeforces problem difficulty be predicted across post-publication and cold-start settings using public metadata, solved-count statistics, exposure-aware features, and lightweight problem-statement structure features?
This project separates two settings:
- Post-publication prediction: solved-count statistics are available after a problem has been published and attempted by users.
- Cold-start prediction: solved-count statistics are unavailable, so the model must rely on metadata and statement structure.
Cold-start note: Cold-start here means no solved-count behavior, not necessarily strict pre-contest prediction. Tags, metadata, and statement availability may not exactly match a real pre-contest setting; the project treats this limitation explicitly.
- The processed dataset contains 10,979 rated
PROGRAMMINGproblems from 1,948 contests. - Ratings range from 800 to 3500.
- Solved-count-only is the strongest simple baseline in both contest-grouped and forward-time evaluation.
- Full API models improve over solved-count-only, showing that metadata and tags add complementary signal.
- Removing solved features causes the largest MAE increase in ablation studies.
- Cold-start metadata-only prediction is substantially harder than post-publication prediction.
- The statement text-light experiment improves cold-start prediction beyond
metadata alone:
- contest-grouped: MAE 317.1 -> 284.0;
- forward-time: MAE 331.4 -> 289.1.
- Statement feature coverage is 99.3%: 10,906 parsed pages out of 10,979 rows, with 73 missing-statement rows handled by imputation.
- Age-normalized solved-count features are useful, but they are only partial exposure proxies.
configs/ Experiment configuration
docs/ Overview documentation assets
paper/ Historical papers and generated figures
scripts/ Local helper scripts
src/cf_diff/ Pipeline modules
tests/ Unit tests
outputs/paper_tables/ Small committed paper-ready CSV tables
Large generated data, raw API snapshots, cached HTML pages, trained model artifacts, logs, and most experiment outputs are intentionally not committed. The compact locked-backtest evidence listed in the data manifest is committed.
The canonical config is configs/experiment.yaml.
It is schema-versioned, rejects missing/unknown settings, and its effective
SHA-256 fingerprint is written into downstream metadata. The file
configs/experiment_legacy_v6.yaml
records the split ratios that actually governed historical runs.
Candidate models are now selected only on validation MAE. Test metrics are joined only after selection, and test-only exposure comparisons are explicitly exploratory.
There are two different reproducibility paths:
- Corrected pipeline: run the commands below with
configs/experiment.yaml. - Historical audit: the legacy config records the effective old ratios, but it does not by itself recreate a paper. Exact historical reproduction also requires the corresponding Git tag and the original local data snapshot, which is not fully committed to this repository.
python -m pip install -r requirements.txt
$env:PYTHONPATH = "src"Run from the repository root.
python -m cf_diff.fetch_api --output-root data/raw/codeforces --lang en --sleep-seconds 2.1 --timeout 30 --latest-dir data/raw/codeforces/latest
python -m cf_diff.preprocess --raw-dir data/raw/codeforces/latest --interim-dir data/interim --processed-dir data/processed --log-path outputs/logs/preprocess.log
python -m cf_diff.features --config configs/experiment.yaml --input-path data/processed/rated_programming_problems.parquet --output-dir data/processed/features --log-path outputs/logs/features.log
python -m cf_diff.splits --config configs/experiment.yaml --input-path data/processed/features/model_table.parquet --output-dir data/processed/splits --log-path outputs/logs/splits.log
python -m cf_diff.eda --config configs/experiment.yaml --processed-path data/processed/rated_programming_problems.parquet --feature-path data/processed/features/model_table.parquet --contest-split-path data/processed/splits/contest_grouped_split.parquet --time-split-path data/processed/splits/forward_time_split.parquet --output-dir outputs/eda --log-path outputs/logs/eda.log
python -m cf_diff.baselines --config configs/experiment.yaml --feature-path data/processed/features/model_table.parquet --feature-columns-path data/processed/features/feature_columns.json --contest-split-path data/processed/splits/contest_grouped_split.parquet --time-split-path data/processed/splits/forward_time_split.parquet --output-dir outputs/baselines --log-path outputs/logs/baselines.log
python -m cf_diff.analysis --config configs/experiment.yaml --feature-path data/processed/features/model_table.parquet --feature-columns-path data/processed/features/feature_columns.json --contest-split-path data/processed/splits/contest_grouped_split.parquet --time-split-path data/processed/splits/forward_time_split.parquet --baseline-metrics-dir outputs/baselines/metrics --baseline-predictions-dir outputs/baselines/predictions --output-dir outputs/analysis --log-path outputs/logs/analysis.log
python -m cf_diff.ablations --config configs/experiment.yaml --feature-path data/processed/features/model_table.parquet --feature-columns-path data/processed/features/feature_columns.json --contest-split-path data/processed/splits/contest_grouped_split.parquet --time-split-path data/processed/splits/forward_time_split.parquet --output-dir outputs/ablations --log-path outputs/logs/ablations.log
python -m cf_diff.robustness --config configs/experiment.yaml --processed-path data/processed/rated_programming_problems.parquet --feature-path data/processed/features/model_table.parquet --feature-columns-path data/processed/features/feature_columns.json --contest-split-path data/processed/splits/contest_grouped_split.parquet --time-split-path data/processed/splits/forward_time_split.parquet --output-dir outputs/robustness --log-path outputs/logs/robustness.log
python -m cf_diff.statement_cold_start --feature-path data/processed/features/model_table.parquet --statement-feature-path data/processed/statement_features/statement_features.parquet --output-dir outputs/statement_cold_start --log-path outputs/logs/statement_cold_start.logIf statement features need to be regenerated locally:
python -m cf_diff.statement_features `
--feature-path data/processed/features/model_table.parquet `
--cache-dir data/raw/codeforces/problem_pages `
--output-dir data/processed/statement_features `
--sleep-seconds 2.5 `
--timeout 30 `
--log-path outputs/logs/statement_features.logThe extractor resumes from cached pages, so closing and restarting the process does not discard already downloaded problem pages. The raw HTML cache and generated statement feature outputs remain local artifacts.
The GitHub Pages site is an evidence-first guide to the current study. Its interactive evidence explorer traces the frozen question, chronological split, validation-only selection, locked test result, uncertainty calculation, and committed result artifacts. It does not load a trained model or present a heuristic prediction as an experimental result.
The historical semantic statement-text experiment extends cold-start prediction with classical TF-IDF features extracted from normalized Codeforces problem-statement text.
This is not BERT, transformers, embeddings, LLMs, or deep semantic understanding. It is a lightweight bag-of-words semantic baseline designed to test whether statement text adds signal beyond metadata and v5 text-light structure features.
Standalone paper artifacts:
Main historical findings:
- TF-IDF alone is weak.
- Metadata + TF-IDF improves over metadata only.
- Metadata + text-light + TF-IDF improves over metadata + text-light.
- The improvement is larger on forward-time validation than contest-grouped validation.
- This extension does not replace the historical full API benchmark; its
full_api_referenceis a ridge-based internal comparison.
Compact historical results:
- Contest-grouped: metadata_only MAE 340.5 -> metadata_plus_tfidf MAE 311.1.
- Forward-time: metadata_only MAE 365.4 -> metadata_plus_tfidf MAE 325.4.
- Contest-grouped: metadata_plus_text_light MAE 310.6 -> metadata_plus_text_light_plus_tfidf MAE 298.5.
- Forward-time: metadata_plus_text_light MAE 335.8 -> metadata_plus_text_light_plus_tfidf MAE 316.2.
python -m pytest -qThe CI badge above and the command output are the source of truth for the current test count.
The full API setting is a post-publication reference because solved-count features are observed after problems have accumulated submissions. Cold-start results exclude solved behavior. Statement text-light features are approximate HTML-derived structure features, not semantic embeddings or deep NLP.
For data and artifact availability details, see
docs/data_manifest.md.