Skip to content

Commit e6108a9

Browse files
DidayoloAnne-Catherine L.
andauthored
Newsletter 2025 (#2171)
* Newsletter 2025 * Fix spacing * Update newsletter 2025 * Formatting * Add break * update newsletter 2025 * Update competitions list in newsletter * Proof reading --------- Co-authored-by: Anne-Catherine L. <acl@lisn.fr>
1 parent 65d0028 commit e6108a9

6 files changed

Lines changed: 245 additions & 3 deletions

File tree

documentation/docs/Newsletters_Archive/CodaLab-in-2024.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,9 @@
1-
## CodaLab in 2024
1+
## CodaLab in 2024
22
### A Year of Breakthroughs and New Horizons with Codabench
33

44
Welcome to the first edition of CodaLab’s newsletter! This year has been full of novelty, success, and scientific progress. The platform is breaking records of participation and number of organized competitions, and [Codabench](https://codabench.org/), the new version of [CodaLab](https://codalab.lisn.fr/), had a very promising launch. Let’s dive into more details.
55

6-
![image_2025_01_30T15_23_35_261Z](../_attachments/991c18dc-3baa-4d69-b157-6a5561e6798d_17528513106457524.jpg)
6+
![image_2025_01_30T15_23_35_261Z](_attachments/991c18dc-3baa-4d69-b157-6a5561e6798d_17528513106457524.jpg)
77

88

99
## Unprecedented engagement
@@ -13,7 +13,7 @@ In October, Codabench has registered its **10,000th user**! From about 100 daily
1313

1414
Contributors community is very active with **143 pull requests** this year. Since the platform is still relatively new, the primary focus has been on bug fixes, security and performance enhancements, and administrative features, accounting for approximately two-thirds of the pull requests. Nevertheless, we are keen on improving the experience for both participants and organizers. We have set a versioning and a release-notes follow-up to give more visibility to the platform evolution and maturity.
1515

16-
![pie-chart](../_attachments/2b41574f-9597-47c0-b492-cf427eba7eff_17528513107025487.jpg)
16+
![pie-chart](_attachments/2b41574f-9597-47c0-b492-cf427eba7eff_17528513107025487.jpg)
1717

1818

1919
## Introducing Codabench
Lines changed: 242 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,242 @@
1+
# CodaLab and Codabench newsletter
2+
3+
## What happened in 2025?
4+
5+
2025 was a year of transition and consolidation for our community. After 13
6+
years of service, [CodaLab Competitions](https://codalab.lisn.fr/) was
7+
officially phased out, closing an important chapter in the history of open
8+
scientific challenges. At the same time, [Codabench](https://codabench.org/)
9+
matured into the central platform for benchmarking, concentrating both usage and
10+
development efforts.
11+
12+
Beyond the symbolic handover, the year was marked by strong community
13+
engagement, growing activity on Codabench, and steady progress on the software
14+
itself. This newsletter offers a snapshot of that journey: key numbers, standout
15+
competitions, and the latest advances shaping the platform.
16+
17+
![image_header](_attachments/header_2025.png)
18+
19+
# Bye bye, CodaLab!
20+
21+
After 13 years and millions of submissions made,
22+
[CodaLab Competitions](https://codalab.lisn.fr/) and its main servers were
23+
finally phased out at end of 2025, passing the torch to
24+
[Codabench](https://codabench.org/).
25+
26+
Today, Codabench is where the community's energy and development efforts are
27+
fully focused. As a modernized evolution of the CodaLab platform, it preserves
28+
familiar workflows while introducing improved performance, live logs, greater
29+
transparency, data-centric benchmarks, and more.
30+
31+
If you haven't made the transition yet as an organizer, good news: CodaLab
32+
bundles are fully compatible with Codabench, making the move straightforward.
33+
The process is documented step by step here:
34+
[How to transition from CodaLab to Codabench](https://docs.codabench.org/latest/Organizers/Benchmark_Creation/How-to-transition-from-CodaLab-to-Codabench/)
35+
36+
# Some statistics
37+
38+
Codabench continued to grow strongly throughout the year, reaching **519 public
39+
competitions** created and welcoming **31,608 new users**! Daily activity also
40+
increased steadily, from around 500 submissions per day in January to **over
41+
1,000 daily submissions by December**, reflecting sustained community
42+
engagement.
43+
44+
CodaLab, while entering its sunset phase, still saw **100 public competitions**
45+
created and **14,854 new users** over the year. Submission activity peaked in
46+
March (around 850 submissions per day), before gradually declining to fewer than
47+
200 daily submissions in December, as usage progressively shifted towards
48+
Codabench.
49+
50+
# Spotlight on competitions
51+
52+
2025 featured many notable competitions across scientific and industrial fields.
53+
From NeurIPS and ICML to challenges in health and medical research,
54+
environmental science, industrial applications, language processing, and
55+
education, the diversity of topics continued to grow.
56+
57+
#### NeurIPS and ICML
58+
59+
- [EEG Foundation Challenge](https://www.codabench.org/competitions/9975/),
60+
aiming to advance the field of electroencephalogram (EEG) decoding by
61+
addressing two critical challenges, (1) models that can transfer knowledge from any cognitive
62+
EEG tasks to active task and (2) creating representations that generalize across different subjects.
63+
It was the most popular competition this year,
64+
featuring **1220 participants**, was the NeurIPS 2025 competition.
65+
- [NeurIPS 2025 Weak Lensing Uncertainty Challenge](https://www.codabench.org/competitions/8934/),
66+
exploring uncertainty-aware and out-of-distribution detection AI techniques for
67+
Weak Gravitational Lensing Cosmology.
68+
- [NeurIPS 2025: Fairness in AI Face Detection Challenge](https://www.codabench.org/competitions/7166/),
69+
where the goal is to advance the development of fair and robust AI-generated
70+
face detection systems by addressing the critical challenge of fairness
71+
generalization under real-world deployment conditions.
72+
- [ICML 2025 AI for Math Workshop & Challenge 1 - APE-Bench I](https://www.codabench.org/competitions/8357/),
73+
designed to evaluate systems that can automate proof engineering in
74+
large-scale formal mathematics libraries.
75+
76+
#### Health and medical research
77+
78+
- [MAMA-MIA Challenge](https://www.codabench.org/competitions/7425/), studying
79+
breast cancer through magnetic resonance imaging (MRI) data, that turned into a
80+
long-term benchmark.
81+
[Universal UltraSound Image Challenge: Multi-Organ Classification and Segmentation](https://www.codabench.org/competitions/9106/),
82+
aiming at evaluating algorithms for multi-organ classification and
83+
segmentation using diverse, real-world ultrasound data collected across
84+
multiple centers and devices. -->
85+
- [NSF HDR Scientific Modeling out of distribution: Neural Forecasting](https://www.codabench.org/competitions/9806/),
86+
in which algorithms forecast the activations of a cluster of neurons given
87+
previous signals from the same cluster.
88+
- [The Algonauts Project 2025 Challenge](https://www.codabench.org/competitions/4313/),
89+
aiming at providing a platform where biological and artificial intelligence
90+
scientists can cooperate and compete in developing cutting-edge encoding models
91+
of neural responses to multimodal naturalistic movies that well generalize
92+
outside of their training distribution.
93+
<!-- [IUGC2025-validation](https://www.codabench.org/competitions/7105/), landmark
94+
detection for intrapartum ultrasound measurement, aiming at improving
95+
childbirth experiences.-->
96+
97+
#### Environmental research
98+
99+
- [MIT ARCLab Prize for Space AI Innovation 2025](https://www.codabench.org/competitions/5547/),
100+
where the objective is to develop cutting-edge AI algorithms for nowcasting
101+
and forecasting space weather-driven changes in atmospheric density across low
102+
earth orbit using historical space weather observations.
103+
- TreeAI4Species Competition:
104+
[Semantic Segmentation](https://www.codabench.org/competitions/9168/) and
105+
[Object detection](https://www.codabench.org/competitions/8485/), studying
106+
algorithms for identifying tree species from high-resolution aerial imagery.
107+
- [Water Scarcity](https://www.codabench.org/competitions/4335/), leveraging
108+
data science to address water scarcity issues through simulations.
109+
110+
#### Industrial applications
111+
112+
- [ICPR 2026 Competition on Low-Resolution License Plate Recognition](https://www.codabench.org/competitions/12259/),
113+
a computer vision challenge on low resolution images which gathered more than
114+
500 participants.
115+
- [AssetOpsBench](https://www.codabench.org/competitions/10206/), in which
116+
participants propose agents that solve realistic industrial tasks across the
117+
full pipeline: "Sensing → Reasoning → Actuation".
118+
- [Universal Behavioral Modeling Data Challenge](https://www.codabench.org/competitions/7230/),
119+
designed to promote a unified approach to behavior modeling and data analytics.
120+
- [WWW 2025: SMARTMEM Memory Failure Prediction Competition](https://www.codabench.org/competitions/3586/),
121+
where the task is to predict memory failures for data centers.
122+
<!-- [MM-CTR: Multimodal CTR Prediction Challenge at the WWW 2025 EReL@MIR Workshop](https://www.codabench.org/competitions/5372),
123+
focusing on multimodal recommandation tasks.
124+
- [Inventory Control Problem](https://www.codabench.org/competitions/9675/),
125+
with the objective of providing agents to solve the Inventory Control Problem.
126+
- [RoboSense - Track1](https://www.codabench.org/competitions/9285/),
127+
challenging participants to develop intelligent driving systems that can
128+
understand and act upon natural language instructions in dynamic driving
129+
environments. -->
130+
131+
#### Natural Language Processing
132+
133+
SemEval (Semantic Evaluation) is an international series of shared tasks in
134+
natural language processing that provides standardized benchmarks to evaluate
135+
and compare systems on semantic understanding challenges. More than 12 tasks
136+
(with sub-tracks) were organized on Codabench in 2025, accounting for more than
137+
20,000 submissions.
138+
139+
- SemEval competitions suite: Task [1](https://www.codabench.org/competitions/9719/), [2](https://www.codabench.org/competitions/9963/), [3](https://www.codabench.org/competitions/10918/),
140+
[4](https://www.codabench.org/competitions/10273/), [5](https://www.codabench.org/competitions/10877/), [6](https://www.codabench.org/competitions/10879/),
141+
[7](https://www.codabench.org/competitions/3737/), [8](https://www.codabench.org/competitions/3360/), [9](https://www.codabench.org/competitions/10522/),
142+
[10](https://www.codabench.org/competitions/10749/), [11](https://www.codabench.org/competitions/3863/), [12](https://www.codabench.org/competitions/12446/)
143+
144+
<!-- SemEval-2026 Task 3 - Dimensional Aspect-Based Sentiment Analysis -
145+
[Track A](https://www.codabench.org/competitions/10918/) -
146+
[Track B](https://www.codabench.org/competitions/11139/)
147+
- [SemEval 2026 Task 4: Narrative Similarity](https://www.codabench.org/competitions/10273/)
148+
- [SemEval 2026 Task 9 - Subtask 1: Multilingual Text Classification Challenge - Polarization Detection ](https://www.codabench.org/competitions/10522/)
149+
- [SemEval 2025 - Task 8: DataBench, Question-Answering over Tabular Data ](https://www.codabench.org/competitions/3360/)
150+
- [Bridging the Gap in Text-Based Emotion Detection - SemEval 2025 Task 11](https://www.codabench.org/competitions/3863/)
151+
-->
152+
153+
Other notable NLP benchmarks:
154+
155+
- [Behind the Secrets of Large Language Models](https://www.codabench.org/competitions/11605/)
156+
- [VLSP2025 DRiLL shared task](https://www.codabench.org/competitions/9722/)
157+
158+
#### Education
159+
160+
- [IndoML 2025 Datathon Tack-1: Mistake Identification](https://www.codabench.org/competitions/7189/),
161+
a task focusing on mistake identification for education application.
162+
- [Compétition Algorithmique Avancée CS 3A INFO -- TSP-rd](https://www.codabench.org/competitions/9896/),
163+
a competition used as a training for students in computer science, receiving
164+
more than 3000 submissions.
165+
<!-- [Machine Translation Challenge! CS-779](https://www.codabench.org/competitions/10523),
166+
a natural language processing student competition that received more than 4000
167+
submissions.
168+
- [IACV 25 - Exercise 5](https://www.codabench.org/competitions/11709), a
169+
competition used for image analysis and computer vision classes. -->
170+
- [Tokam2D - Structure detection in fusion plasma simulations - datacamp 2025](https://www.codabench.org/competitions/11224/),
171+
physics based data science training at Université Paris-Saclay.
172+
173+
A huge thank you to everyone in the community for these **outstanding scientific contributions**
174+
across a wide variety of fields. You can discover **many more challenges in the [public competition listing](https://www.codabench.org/competitions/public/?page=1)**.
175+
176+
![image_break](_attachments/break_2025.png)
177+
178+
# Novelty in the software
179+
180+
Our contributors community was very active, with **139 pull requests merged**
181+
this year. Many new features, bug fixes, and back-end changes were made. We
182+
present some of them below.
183+
184+
#### New features for participants and organizers
185+
186+
- Public datasets listing: https://www.codabench.org/datasets/public/?page=1
187+
- [Croissant](https://docs.mlcommons.org/croissant/) standard compatibility
188+
- New documentation website: https://docs.codabench.org
189+
- Users can delete their submissions and manage their individual storage
190+
- Leaderboards are now public for everyone without login required
191+
192+
#### Back-end changes for developpers and hosters
193+
194+
- Using Playwright instead of Selenium for automatic tests
195+
- Logs are now colored and easier to read
196+
- Django and other packages version upgrades
197+
198+
#### What's to come
199+
200+
The trend is to make the project more easy to deploy for independant hosters.
201+
202+
- Unified and lighter compute worker image, making it more stable
203+
- Make compute worker its own repository, which means it can be more easily used
204+
for other projects if needed
205+
- Django Admin upgrades to make it easier to manage the website as a site admin
206+
207+
# Community
208+
209+
Reminder on our communication tools:
210+
211+
- Join our [google forum](https://groups.google.com/g/codalab-competitions) to
212+
communicate your competitions and events
213+
- Contact us for any question: info@codabench.org
214+
- Write an issue on [github](https://github.com/codalab/codabench) about
215+
interesting suggestions
216+
217+
Please cite this paper when working with Codabench:
218+
219+
```
220+
@article{codabench,
221+
title = {Codabench: Flexible, easy-to-use, and reproducible meta-benchmark platform},
222+
author = {Zhen Xu and Sergio Escalera and Adrien Pavão and Magali Richard and
223+
Wei-Wei Tu and Quanming Yao and Huan Zhao and Isabelle Guyon},
224+
journal = {Patterns},
225+
volume = {3},
226+
number = {7},
227+
pages = {100543},
228+
year = {2022},
229+
issn = {2666-3899},
230+
doi = {https://doi.org/10.1016/j.patter.2022.100543},
231+
url = {https://www.sciencedirect.com/science/article/pii/S2666389922001465}
232+
}
233+
```
234+
235+
# Closing words
236+
237+
Thank you for reading the our newsletter. We're not done yet! More projects,
238+
more challenges, and more science ahead. Our open platform is becoming a
239+
powerful actor for building reliable and innovative AI benchmarks. See you on
240+
Codabench.
241+
242+
![partners](_attachments/96bbc3a0-cb9b-479a-85c8-ba795c59d6f4_1753436671847627.jpg)

documentation/docs/_attachments/2b41574f-9597-47c0-b492-cf427eba7eff_17528513107025487.jpg renamed to documentation/docs/Newsletters_Archive/_attachments/2b41574f-9597-47c0-b492-cf427eba7eff_17528513107025487.jpg

File renamed without changes.

documentation/docs/_attachments/991c18dc-3baa-4d69-b157-6a5561e6798d_17528513106457524.jpg renamed to documentation/docs/Newsletters_Archive/_attachments/991c18dc-3baa-4d69-b157-6a5561e6798d_17528513106457524.jpg

File renamed without changes.
68.7 KB
Loading
72.9 KB
Loading

0 commit comments

Comments
 (0)