|
| 1 | +# CodaLab and Codabench newsletter |
| 2 | + |
| 3 | +## What happened in 2025? |
| 4 | + |
| 5 | +2025 was a year of transition and consolidation for our community. After 13 |
| 6 | +years of service, [CodaLab Competitions](https://codalab.lisn.fr/) was |
| 7 | +officially phased out, closing an important chapter in the history of open |
| 8 | +scientific challenges. At the same time, [Codabench](https://codabench.org/) |
| 9 | +matured into the central platform for benchmarking, concentrating both usage and |
| 10 | +development efforts. |
| 11 | + |
| 12 | +Beyond the symbolic handover, the year was marked by strong community |
| 13 | +engagement, growing activity on Codabench, and steady progress on the software |
| 14 | +itself. This newsletter offers a snapshot of that journey: key numbers, standout |
| 15 | +competitions, and the latest advances shaping the platform. |
| 16 | + |
| 17 | + |
| 18 | + |
| 19 | +# Bye bye, CodaLab! |
| 20 | + |
| 21 | +After 13 years and millions of submissions made, |
| 22 | +[CodaLab Competitions](https://codalab.lisn.fr/) and its main servers were |
| 23 | +finally phased out at end of 2025, passing the torch to |
| 24 | +[Codabench](https://codabench.org/). |
| 25 | + |
| 26 | +Today, Codabench is where the community's energy and development efforts are |
| 27 | +fully focused. As a modernized evolution of the CodaLab platform, it preserves |
| 28 | +familiar workflows while introducing improved performance, live logs, greater |
| 29 | +transparency, data-centric benchmarks, and more. |
| 30 | + |
| 31 | +If you haven't made the transition yet as an organizer, good news: CodaLab |
| 32 | +bundles are fully compatible with Codabench, making the move straightforward. |
| 33 | +The process is documented step by step here: |
| 34 | +[How to transition from CodaLab to Codabench](https://docs.codabench.org/latest/Organizers/Benchmark_Creation/How-to-transition-from-CodaLab-to-Codabench/) |
| 35 | + |
| 36 | +# Some statistics |
| 37 | + |
| 38 | +Codabench continued to grow strongly throughout the year, reaching **519 public |
| 39 | +competitions** created and welcoming **31,608 new users**! Daily activity also |
| 40 | +increased steadily, from around 500 submissions per day in January to **over |
| 41 | +1,000 daily submissions by December**, reflecting sustained community |
| 42 | +engagement. |
| 43 | + |
| 44 | +CodaLab, while entering its sunset phase, still saw **100 public competitions** |
| 45 | +created and **14,854 new users** over the year. Submission activity peaked in |
| 46 | +March (around 850 submissions per day), before gradually declining to fewer than |
| 47 | +200 daily submissions in December, as usage progressively shifted towards |
| 48 | +Codabench. |
| 49 | + |
| 50 | +# Spotlight on competitions |
| 51 | + |
| 52 | +2025 featured many notable competitions across scientific and industrial fields. |
| 53 | +From NeurIPS and ICML to challenges in health and medical research, |
| 54 | +environmental science, industrial applications, language processing, and |
| 55 | +education, the diversity of topics continued to grow. |
| 56 | + |
| 57 | +#### NeurIPS and ICML |
| 58 | + |
| 59 | +- [EEG Foundation Challenge](https://www.codabench.org/competitions/9975/), |
| 60 | + aiming to advance the field of electroencephalogram (EEG) decoding by |
| 61 | + addressing two critical challenges, (1) models that can transfer knowledge from any cognitive |
| 62 | + EEG tasks to active task and (2) creating representations that generalize across different subjects. |
| 63 | + It was the most popular competition this year, |
| 64 | + featuring **1220 participants**, was the NeurIPS 2025 competition. |
| 65 | +- [NeurIPS 2025 Weak Lensing Uncertainty Challenge](https://www.codabench.org/competitions/8934/), |
| 66 | +exploring uncertainty-aware and out-of-distribution detection AI techniques for |
| 67 | +Weak Gravitational Lensing Cosmology. |
| 68 | +- [NeurIPS 2025: Fairness in AI Face Detection Challenge](https://www.codabench.org/competitions/7166/), |
| 69 | + where the goal is to advance the development of fair and robust AI-generated |
| 70 | + face detection systems by addressing the critical challenge of fairness |
| 71 | + generalization under real-world deployment conditions. |
| 72 | +- [ICML 2025 AI for Math Workshop & Challenge 1 - APE-Bench I](https://www.codabench.org/competitions/8357/), |
| 73 | + designed to evaluate systems that can automate proof engineering in |
| 74 | + large-scale formal mathematics libraries. |
| 75 | + |
| 76 | +#### Health and medical research |
| 77 | + |
| 78 | +- [MAMA-MIA Challenge](https://www.codabench.org/competitions/7425/), studying |
| 79 | +breast cancer through magnetic resonance imaging (MRI) data, that turned into a |
| 80 | +long-term benchmark. |
| 81 | +[Universal UltraSound Image Challenge: Multi-Organ Classification and Segmentation](https://www.codabench.org/competitions/9106/), |
| 82 | + aiming at evaluating algorithms for multi-organ classification and |
| 83 | + segmentation using diverse, real-world ultrasound data collected across |
| 84 | + multiple centers and devices. --> |
| 85 | +- [NSF HDR Scientific Modeling out of distribution: Neural Forecasting](https://www.codabench.org/competitions/9806/), |
| 86 | + in which algorithms forecast the activations of a cluster of neurons given |
| 87 | + previous signals from the same cluster. |
| 88 | +- [The Algonauts Project 2025 Challenge](https://www.codabench.org/competitions/4313/), |
| 89 | +aiming at providing a platform where biological and artificial intelligence |
| 90 | +scientists can cooperate and compete in developing cutting-edge encoding models |
| 91 | +of neural responses to multimodal naturalistic movies that well generalize |
| 92 | +outside of their training distribution. |
| 93 | +<!-- [IUGC2025-validation](https://www.codabench.org/competitions/7105/), landmark |
| 94 | + detection for intrapartum ultrasound measurement, aiming at improving |
| 95 | + childbirth experiences.--> |
| 96 | + |
| 97 | +#### Environmental research |
| 98 | + |
| 99 | +- [MIT ARCLab Prize for Space AI Innovation 2025](https://www.codabench.org/competitions/5547/), |
| 100 | + where the objective is to develop cutting-edge AI algorithms for nowcasting |
| 101 | + and forecasting space weather-driven changes in atmospheric density across low |
| 102 | + earth orbit using historical space weather observations. |
| 103 | +- TreeAI4Species Competition: |
| 104 | + [Semantic Segmentation](https://www.codabench.org/competitions/9168/) and |
| 105 | + [Object detection](https://www.codabench.org/competitions/8485/), studying |
| 106 | + algorithms for identifying tree species from high-resolution aerial imagery. |
| 107 | +- [Water Scarcity](https://www.codabench.org/competitions/4335/), leveraging |
| 108 | + data science to address water scarcity issues through simulations. |
| 109 | + |
| 110 | +#### Industrial applications |
| 111 | + |
| 112 | +- [ICPR 2026 Competition on Low-Resolution License Plate Recognition](https://www.codabench.org/competitions/12259/), |
| 113 | + a computer vision challenge on low resolution images which gathered more than |
| 114 | + 500 participants. |
| 115 | +- [AssetOpsBench](https://www.codabench.org/competitions/10206/), in which |
| 116 | + participants propose agents that solve realistic industrial tasks across the |
| 117 | + full pipeline: "Sensing → Reasoning → Actuation". |
| 118 | +- [Universal Behavioral Modeling Data Challenge](https://www.codabench.org/competitions/7230/), |
| 119 | +designed to promote a unified approach to behavior modeling and data analytics. |
| 120 | +- [WWW 2025: SMARTMEM Memory Failure Prediction Competition](https://www.codabench.org/competitions/3586/), |
| 121 | + where the task is to predict memory failures for data centers. |
| 122 | +<!-- [MM-CTR: Multimodal CTR Prediction Challenge at the WWW 2025 EReL@MIR Workshop](https://www.codabench.org/competitions/5372), |
| 123 | + focusing on multimodal recommandation tasks. |
| 124 | +- [Inventory Control Problem](https://www.codabench.org/competitions/9675/), |
| 125 | + with the objective of providing agents to solve the Inventory Control Problem. |
| 126 | +- [RoboSense - Track1](https://www.codabench.org/competitions/9285/), |
| 127 | + challenging participants to develop intelligent driving systems that can |
| 128 | + understand and act upon natural language instructions in dynamic driving |
| 129 | + environments. --> |
| 130 | + |
| 131 | +#### Natural Language Processing |
| 132 | + |
| 133 | +SemEval (Semantic Evaluation) is an international series of shared tasks in |
| 134 | +natural language processing that provides standardized benchmarks to evaluate |
| 135 | +and compare systems on semantic understanding challenges. More than 12 tasks |
| 136 | +(with sub-tracks) were organized on Codabench in 2025, accounting for more than |
| 137 | +20,000 submissions. |
| 138 | + |
| 139 | +- SemEval competitions suite: Task [1](https://www.codabench.org/competitions/9719/), [2](https://www.codabench.org/competitions/9963/), [3](https://www.codabench.org/competitions/10918/), |
| 140 | + [4](https://www.codabench.org/competitions/10273/), [5](https://www.codabench.org/competitions/10877/), [6](https://www.codabench.org/competitions/10879/), |
| 141 | + [7](https://www.codabench.org/competitions/3737/), [8](https://www.codabench.org/competitions/3360/), [9](https://www.codabench.org/competitions/10522/), |
| 142 | + [10](https://www.codabench.org/competitions/10749/), [11](https://www.codabench.org/competitions/3863/), [12](https://www.codabench.org/competitions/12446/) |
| 143 | + |
| 144 | +<!-- SemEval-2026 Task 3 - Dimensional Aspect-Based Sentiment Analysis - |
| 145 | + [Track A](https://www.codabench.org/competitions/10918/) - |
| 146 | + [Track B](https://www.codabench.org/competitions/11139/) |
| 147 | +- [SemEval 2026 Task 4: Narrative Similarity](https://www.codabench.org/competitions/10273/) |
| 148 | +- [SemEval 2026 Task 9 - Subtask 1: Multilingual Text Classification Challenge - Polarization Detection ](https://www.codabench.org/competitions/10522/) |
| 149 | +- [SemEval 2025 - Task 8: DataBench, Question-Answering over Tabular Data ](https://www.codabench.org/competitions/3360/) |
| 150 | +- [Bridging the Gap in Text-Based Emotion Detection - SemEval 2025 Task 11](https://www.codabench.org/competitions/3863/) |
| 151 | + --> |
| 152 | + |
| 153 | +Other notable NLP benchmarks: |
| 154 | + |
| 155 | +- [Behind the Secrets of Large Language Models](https://www.codabench.org/competitions/11605/) |
| 156 | +- [VLSP2025 DRiLL shared task](https://www.codabench.org/competitions/9722/) |
| 157 | + |
| 158 | +#### Education |
| 159 | + |
| 160 | +- [IndoML 2025 Datathon Tack-1: Mistake Identification](https://www.codabench.org/competitions/7189/), |
| 161 | + a task focusing on mistake identification for education application. |
| 162 | +- [Compétition Algorithmique Avancée CS 3A INFO -- TSP-rd](https://www.codabench.org/competitions/9896/), |
| 163 | +a competition used as a training for students in computer science, receiving |
| 164 | +more than 3000 submissions. |
| 165 | +<!-- [Machine Translation Challenge! CS-779](https://www.codabench.org/competitions/10523), |
| 166 | + a natural language processing student competition that received more than 4000 |
| 167 | + submissions. |
| 168 | +- [IACV 25 - Exercise 5](https://www.codabench.org/competitions/11709), a |
| 169 | + competition used for image analysis and computer vision classes. --> |
| 170 | +- [Tokam2D - Structure detection in fusion plasma simulations - datacamp 2025](https://www.codabench.org/competitions/11224/), |
| 171 | + physics based data science training at Université Paris-Saclay. |
| 172 | + |
| 173 | +A huge thank you to everyone in the community for these **outstanding scientific contributions** |
| 174 | +across a wide variety of fields. You can discover **many more challenges in the [public competition listing](https://www.codabench.org/competitions/public/?page=1)**. |
| 175 | + |
| 176 | + |
| 177 | + |
| 178 | +# Novelty in the software |
| 179 | + |
| 180 | +Our contributors community was very active, with **139 pull requests merged** |
| 181 | +this year. Many new features, bug fixes, and back-end changes were made. We |
| 182 | +present some of them below. |
| 183 | + |
| 184 | +#### New features for participants and organizers |
| 185 | + |
| 186 | +- Public datasets listing: https://www.codabench.org/datasets/public/?page=1 |
| 187 | +- [Croissant](https://docs.mlcommons.org/croissant/) standard compatibility |
| 188 | +- New documentation website: https://docs.codabench.org |
| 189 | +- Users can delete their submissions and manage their individual storage |
| 190 | +- Leaderboards are now public for everyone without login required |
| 191 | + |
| 192 | +#### Back-end changes for developpers and hosters |
| 193 | + |
| 194 | +- Using Playwright instead of Selenium for automatic tests |
| 195 | +- Logs are now colored and easier to read |
| 196 | +- Django and other packages version upgrades |
| 197 | + |
| 198 | +#### What's to come |
| 199 | + |
| 200 | +The trend is to make the project more easy to deploy for independant hosters. |
| 201 | + |
| 202 | +- Unified and lighter compute worker image, making it more stable |
| 203 | +- Make compute worker its own repository, which means it can be more easily used |
| 204 | + for other projects if needed |
| 205 | +- Django Admin upgrades to make it easier to manage the website as a site admin |
| 206 | + |
| 207 | +# Community |
| 208 | + |
| 209 | +Reminder on our communication tools: |
| 210 | + |
| 211 | +- Join our [google forum](https://groups.google.com/g/codalab-competitions) to |
| 212 | + communicate your competitions and events |
| 213 | +- Contact us for any question: info@codabench.org |
| 214 | +- Write an issue on [github](https://github.com/codalab/codabench) about |
| 215 | + interesting suggestions |
| 216 | + |
| 217 | +Please cite this paper when working with Codabench: |
| 218 | + |
| 219 | +``` |
| 220 | +@article{codabench, |
| 221 | + title = {Codabench: Flexible, easy-to-use, and reproducible meta-benchmark platform}, |
| 222 | + author = {Zhen Xu and Sergio Escalera and Adrien Pavão and Magali Richard and |
| 223 | + Wei-Wei Tu and Quanming Yao and Huan Zhao and Isabelle Guyon}, |
| 224 | + journal = {Patterns}, |
| 225 | + volume = {3}, |
| 226 | + number = {7}, |
| 227 | + pages = {100543}, |
| 228 | + year = {2022}, |
| 229 | + issn = {2666-3899}, |
| 230 | + doi = {https://doi.org/10.1016/j.patter.2022.100543}, |
| 231 | + url = {https://www.sciencedirect.com/science/article/pii/S2666389922001465} |
| 232 | +} |
| 233 | +``` |
| 234 | + |
| 235 | +# Closing words |
| 236 | + |
| 237 | +Thank you for reading the our newsletter. We're not done yet! More projects, |
| 238 | +more challenges, and more science ahead. Our open platform is becoming a |
| 239 | +powerful actor for building reliable and innovative AI benchmarks. See you on |
| 240 | +Codabench. |
| 241 | + |
| 242 | + |
0 commit comments