MDBF-1185 msan/ubsan/asan with clang 22 by grooverdan · Pull Request #912 · MariaDB/buildbot

grooverdan · 2026-02-23T04:00:26Z

Here we phase in clang 22 as a msan and ubasan builder.

If all goes well, we'll remove clang-20 out.

Add New Build template

Checklist

[N/A] Make changes os_info.yaml
Schedule the builder for the appropriate branch in constants.py
[Reused ] Add builder configuration
[N/A] Update workflows as appropriate i.e.: .github/workflows/build-X-based.yml
[N/A?] Update front page - master-web/templates/home.jade
[N/A - not a release platform] Add to MariaDB Documentation page for Platform Deprecation Policy

RazvanLiviuVarzaru · 2026-02-23T07:24:27Z

My suggestion is to use sg-bbw1 as a worker machine.
@fauust can you please configure it to connect to master-migration?

This is the estimated load of this patch: https://github.com/MariaDB/buildbot/actions/runs/22293877010?pr=912

grooverdan · 2026-02-25T03:12:16Z

Arch: amd64

Worker	Total Jobs	Requested Jobs
hz-bbw6	96	20
hz-bbw8	110	110
hz-bbw9	110	110
bg-bbw5-x64	8	0
sg-bbw1	128	56

note llvm/clang 22 release hasn't happened. I think we'll container rebuild when that happens for a better dev test.

RazvanLiviuVarzaru · 2026-02-25T07:25:42Z

Arch: amd64

Worker Total Jobs Requested Jobs
hz-bbw6 96 20
hz-bbw8 110 110
hz-bbw9 110 110
bg-bbw5-x64 8 0
sg-bbw1 128 56
note llvm/clang 22 release hasn't happened. I think we'll container rebuild when that happens for a better dev test.

Thanks, @grooverdan .
Can you tell me please when this should be scheduled for Production?
Because I can apply the changes in buildbot.dev.mariadb.org without actually merging and then you can use my server-fork to test as much as you like.

I am asking this because merging to DEV it is also a commitment to "soon will be released in buildbot production" and if this is not happening anytime soon it might get reverted if I want to do a release.

grooverdan · 2026-02-26T05:11:07Z

https://discourse.llvm.org/t/llvm-22-1-0-released/89950

Released - hence #916

Can you tell me please when this should be scheduled for Production?
Because I can apply the changes in buildbot.dev.mariadb.org without actually merging and then you can use my server-fork to test as much as you like.

I've done the local #916 build and its passed its non-debug builds, server 10.11 on MSAN and ASAN/UBSAN. I'm not expecting massive failure.

Sure - lets take a non-merged test on DEV

If you've got time to trigger a 10.11, main, and if resources 11.4 (as a middle), I'll go through the results and report back here.

fauust · 2026-05-29T13:07:13Z

Guys, sorry, missed that one, let me know if still relevant!

RazvanLiviuVarzaru · 2026-06-02T07:14:34Z

@fauust This is to configure the sg-bbw1 machine for Buildbot:
https://git.mariadb.org/MariaDB/sysadmin/pulls/71
If ok, please merge & deploy.

@grooverdan Since clang-22 builders are resource demanding, the plan would be:

have the builds run on the Quape machine for now
since the machine is idle at the moment, we can do as many dev tests as we like
have the builders deployed in production on the Quape machine. Since there's only one worker we can expect a fair amount of queues on busy days
once they are stable in production, ping me and I will handle removing clang-20 builders and assigning the clang-22 builders to a bigger worker pool (that can handle branch protection requirements for load and stability)

fauust · 2026-06-02T08:43:28Z

@fauust This is to configure the sg-bbw1 machine for Buildbot: https://git.mariadb.org/MariaDB/sysadmin/pulls/71 If ok, please merge & deploy.

done

RazvanLiviuVarzaru · 2026-06-02T16:59:16Z

Hi, @grooverdan !

Here's the patch status on all branches, from the HEAD commits at the time of triggering.
I'm sorry for such a long delay in testing.
Moving forward, the builders should be green before merging this patch. I've re-assigned the MDBF to you.

Branch	Commit	Builder	State
10.6	`70a88a6b6e11d3cb840d5b27d05d967aff099ef0`	amd64-msan-clang-22-debug	❌ failed compile (failure)
10.6	`70a88a6b6e11d3cb840d5b27d05d967aff099ef0`	amd64-ubasan-clang-22-debug	❌ failed compile (failure)
10.6	`70a88a6b6e11d3cb840d5b27d05d967aff099ef0`	amd64-msan-clang-22	❌ failed MTR S3 (failure)
10.6	`70a88a6b6e11d3cb840d5b27d05d967aff099ef0`	amd64-ubasan-clang-22	❌ failed MTR normal (failure) MTR S3 (failure)
10.11	`936f04c4f9bd5d66014942bd27e1996b2e740626`	amd64-msan-clang-22-debug	❌ failed compile (failure)
10.11	`936f04c4f9bd5d66014942bd27e1996b2e740626`	amd64-ubasan-clang-22-debug	❌ failed MTR normal (failure) MTR S3 (failure)
10.11	`936f04c4f9bd5d66014942bd27e1996b2e740626`	amd64-msan-clang-22	❌ failed MTR S3 (failure)
10.11	`936f04c4f9bd5d66014942bd27e1996b2e740626`	amd64-ubasan-clang-22	❌ failed MTR normal (failure) MTR S3 (failure)
11.4	`d2ebea4febdbc102da09a3b51bea4ef2819a3843`	amd64-msan-clang-22-debug	❌ failed compile (failure)
11.4	`d2ebea4febdbc102da09a3b51bea4ef2819a3843`	amd64-ubasan-clang-22-debug	❌ failed MTR normal (failure) MTR S3 (failure)
11.4	`d2ebea4febdbc102da09a3b51bea4ef2819a3843`	amd64-msan-clang-22	❌ failed MTR S3 (failure)
11.4	`d2ebea4febdbc102da09a3b51bea4ef2819a3843`	amd64-ubasan-clang-22	❌ failed MTR normal (failure) MTR S3 (failure)
11.8	`080a7318acc2fc578b5915d1908f838b6f942de0`	amd64-msan-clang-22-debug	❌ failed compile (failure)
11.8	`080a7318acc2fc578b5915d1908f838b6f942de0`	amd64-ubasan-clang-22-debug	❌ failed MTR normal (failure) MTR S3 (failure)
11.8	`080a7318acc2fc578b5915d1908f838b6f942de0`	amd64-msan-clang-22	❌ failed MTR S3 (failure)
11.8	`080a7318acc2fc578b5915d1908f838b6f942de0`	amd64-ubasan-clang-22	❌ failed MTR normal (failure) MTR S3 (failure)
12.3	`8fd382adf6380ee299937b831ffdb4f22eaab37e`	amd64-msan-clang-22-debug	❌ failed compile (failure)
12.3	`8fd382adf6380ee299937b831ffdb4f22eaab37e`	amd64-ubasan-clang-22-debug	❌ failed MTR normal (failure) MTR S3 (failure)
12.3	`8fd382adf6380ee299937b831ffdb4f22eaab37e`	amd64-msan-clang-22	❌ failed MTR S3 (failure)
12.3	`8fd382adf6380ee299937b831ffdb4f22eaab37e`	amd64-ubasan-clang-22	❌ failed MTR normal (failure) MTR S3 (failure) MTR cursor (failure)
13.0	`09b9772eec85b72bca633f5885b4c4a029326214`	amd64-msan-clang-22-debug	❌ failed compile (failure)
13.0	`09b9772eec85b72bca633f5885b4c4a029326214`	amd64-ubasan-clang-22-debug	❌ failed MTR normal (failure) MTR S3 (failure)
13.0	`09b9772eec85b72bca633f5885b4c4a029326214`	amd64-msan-clang-22	❌ failed MTR S3 (failure)
13.0	`09b9772eec85b72bca633f5885b4c4a029326214`	amd64-ubasan-clang-22	❌ failed MTR normal (failure) MTR S3 (failure) MTR cursor (failure) MTR view (failure)
main	`a06be0288f63f520aec95982cc6f872d26c4f23c`	amd64-msan-clang-22-debug	❌ failed compile (failure)
main	`a06be0288f63f520aec95982cc6f872d26c4f23c`	amd64-ubasan-clang-22-debug	❌ failed MTR normal (failure) MTR S3 (failure)
main	`a06be0288f63f520aec95982cc6f872d26c4f23c`	amd64-msan-clang-22	❌ failed MTR S3 (failure)
main	`a06be0288f63f520aec95982cc6f872d26c4f23c`	amd64-ubasan-clang-22	❌ failed MTR normal (failure) MTR S3 (failure) MTR cursor (failure) MTR view (failure)

grooverdan · 2026-06-03T01:31:11Z

Thanks. Yikes.

10.6 was only intended to gain clang-22-msan per constants.

Bugs lodged and linked to MDBF-1185.

view/cursor bugs waiting on merge up.

ubasasn-debug - normal failures - waiting on mroonga merge up (10.11 only still) and connector ODBC update or code error supressions.

With the amd64-msan-clang-22-debug compile failure, can you trigger some 10.11+ builds with CMAKE_C{,XX}_FLAGS=-O1 to see what remains after the build step. I've verified this can get through the build. - never mind - crashes on bootstrap MDEV-38419

RazvanLiviuVarzaru · 2026-07-06T12:14:51Z

Looks good but now the capacity of the target worker is only:

  Model name:                Intel(R) Xeon(R) Platinum 8260 CPU @ 2.40GHz
    CPU family:              6
    Model:                   85
    Thread(s) per core:      1
    Core(s) per socket:      20
    Socket(s):               1

So we can only host 2 builders given we scale them down to parallel=10,
unless @fauust has some other ideas. The first sg-bbw1 was really good.

grooverdan · 2026-07-06T22:46:34Z

So amd64-msan-clang-22-debug is on hold - MDEV-38419, this currently 3 builder addition, so only 1 needs to be dropped.

I've a small preference for option1 freeing more resources and requiring less consequential changes.

Edit: ODBC mem leaks need https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1141600 as a stable backport (preferably) otherwise we pull a testing/unstable into the Deb 13 msan image.

Option 1: Drop `amd64-ubasan-clang-20` and `amd64-ubasan-clang-20-debug` early

Dropping amd64-ubasan-clang-20-debug and amd64-ubasan-clang-20.

With experimental changes (view + cursor) moved to msan, there is no run difference to ubasan-20. The current 10.11 always red on odbc memory leaks (MDEV-38838, resolved in clang-22 image). main.long_unique_bugs has a fix in 10.11 waiting merge that should make the non-debug amd64-ubasan-clang-22 green soon. For ubasan-clang-22-debug - MDEV-39827 - roles.acl_load_mutex-5170, isn't a requestion.

And they are not a protected branch builders.

Follow up action:

amd64-ubasan-clang-22 and amd64-msan-clang-22 to protected branches
amd64-msan-clang-20 removal

Option 2: Defer `amd64-ubasan-clang-22-debug`

This gives us amd64-ubasan-clang-22 and amd64-msan-clang-22 deployed early as the two likely to be most stable to transition to protected branch builders.

Having amd64-ubasan-clang-22 deployed will keep the ubsan-clang-22 compatibility testing mostly there. The small delta can be addressed as amd64-ubasan-clang-20 and amd64-msan-clang-20 become retired.

Follow up action:

amd64-ubasan-clang-22 and amd64-msan-clang-22 to protected branches
amd64-msan-clang-20 and amd64-ubasan-clang-20 removal
amd64-ubasan-clang-22-debug deploy via dev env.

Follow up action 2:

amd64-ubasan-clang-20-debug removal

RazvanLiviuVarzaru · 2026-07-07T04:30:45Z

So amd64-msan-clang-22-debug is on hold - MDEV-38419, this currently 3 builder addition, so only 1 needs to be dropped.

I've a small preference for option1 freeing more resources and requiring less consequential changes.

Edit: ODBC mem leaks need https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1141600 as a stable backport (preferably) otherwise we pull a testing/unstable into the Deb 13 msan image.

Option 1: Drop amd64-ubasan-clang-20 and amd64-ubasan-clang-20-debug early

Dropping amd64-ubasan-clang-20-debug and amd64-ubasan-clang-20.

With experimental changes (view + cursor) moved to msan, there is no run difference to ubasan-20. The current 10.11 always red on odbc memory leaks (MDEV-38838, resolved in clang-22 image). main.long_unique_bugs has a fix in 10.11 waiting merge that should make the non-debug amd64-ubasan-clang-22 green soon. For ubasan-clang-22-debug - MDEV-39827 - roles.acl_load_mutex-5170, isn't a requestion.

And they are not a protected branch builders.

Follow up action:

amd64-ubasan-clang-22 and amd64-msan-clang-22 to protected branches

amd64-msan-clang-20 removal

Option 2: Defer amd64-ubasan-clang-22-debug

This gives us amd64-ubasan-clang-22 and amd64-msan-clang-22 deployed early as the two likely to be most stable to transition to protected branch builders.

Having amd64-ubasan-clang-22 deployed will keep the ubsan-clang-22 compatibility testing mostly there. The small delta can be addressed as amd64-ubasan-clang-20 and amd64-msan-clang-20 become retired.

Follow up action:

amd64-ubasan-clang-22 and amd64-msan-clang-22 to protected branches

amd64-msan-clang-20 and amd64-ubasan-clang-20 removal

amd64-ubasan-clang-22-debug deploy via dev env.

Follow up action 2:

amd64-ubasan-clang-20-debug removal

@grooverdan Option 1 seems like a good plan. The way I look at it:
(x) : is jobs current.

For a future protected branches we should use Hetzner, preferably 3 workers.

hz-bbw6 (27/96):

(+20) MSAN CLANG 22
(+12) UBASAN CLANG 22
(+12) UBASAN CLANG 22 DEBUG

^ OK, enough capacity.

hz-bbw8 and hz-bbw9 (105/110):

(+20) MSAN CLANG 22
(+12) UBASAN CLANG 22
(-12) removed now UBASAN CLANG 20
(-12) removed now UBASAN CLANG 20 DEBUG

^ not ok, there's an extra + 8. What can we sacrifice? MSAN-20 DEBUG?

sg-bbw1 (0/20):

(+12) UBASAN CLANG 22 DEBUG

^ enough capacity, I don't want to use this worker for protected builders hence adding it as a backup worker for UBASAN CLANG 22 DEBUG.

grooverdan · 2026-07-20T02:07:42Z

msan-20-debug sounds the most sacraficial

Phase in Clang 22 MSAN/UBSAN builders and update their worker assignments. Remove the obsolete Clang 20 builders and the MSAN debug configuration. Add an experimental mode to evaluate new suites and options with the Clang 22 builders. Extend Spider coverage to the non-debug MSAN build after it proved quick and stable locally. Evaluate cursor and view protocols across MSAN/UBSAN debug and non-debug builds. UBSAN mainly reproduced known minor failures. MSAN view exposed a failure that was also reproducible without MSAN, so run cursor and view tests on the faster non-debug MSAN builder. Use -Og for debug sanitizer builds to reduce instrumentation and improve CI runtime while retaining useful debugging information. Recognize DuckDB as a plugin, but disable it under MSAN/UBSAN until its linking issue is fixed.

RazvanLiviuVarzaru

From a bb config perspective will approve and merge
outstanding test failures fixes will hopefully catch up.
will include this patch in 21 jul 2026 bb release.

https://buildbot.dev.mariadb.org/#/builders/110/builds/383

If you suspect a bb environment related issue
that needs my immediate attention, ping on #Buildbot > Buildbot release 06.07.2026

grooverdan force-pushed the msan_ubsan_22 branch from a8aae3f to fc7bff6 Compare February 23, 2026 05:17

grooverdan force-pushed the msan_ubsan_22 branch 2 times, most recently from 063a9c2 to 2531155 Compare February 25, 2026 02:59

grooverdan mentioned this pull request Feb 26, 2026

MDBF-1185: Clang 22 released - update image #916

Merged

grooverdan force-pushed the msan_ubsan_22 branch from 2531155 to ff553bf Compare February 26, 2026 05:13

grooverdan marked this pull request as draft April 2, 2026 07:08

grooverdan marked this pull request as ready for review April 3, 2026 06:53

grooverdan mentioned this pull request Apr 3, 2026

MSAN: unwind (fixing clang 22) odbc + connect + sqlite. #942

Merged

grooverdan force-pushed the msan_ubsan_22 branch 3 times, most recently from 912ac74 to 5d07cfb Compare May 27, 2026 08:02

grooverdan requested a review from RazvanLiviuVarzaru May 27, 2026 08:02

RazvanLiviuVarzaru reviewed Jun 2, 2026

View reviewed changes

Comment thread configuration/builders/sequences/sanitizers.py Outdated

grooverdan force-pushed the msan_ubsan_22 branch 2 times, most recently from b4a6066 to 199a61e Compare July 6, 2026 06:28

grooverdan requested a review from RazvanLiviuVarzaru July 6, 2026 06:33

RazvanLiviuVarzaru force-pushed the msan_ubsan_22 branch from 01b05f7 to d162bbe Compare July 20, 2026 07:25

RazvanLiviuVarzaru approved these changes Jul 20, 2026

View reviewed changes

RazvanLiviuVarzaru merged commit ab1823a into MariaDB:dev Jul 20, 2026
4 checks passed

grooverdan deleted the msan_ubsan_22 branch July 20, 2026 08:38

Uh oh!

Conversation

grooverdan commented Feb 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Add New Build template

Checklist

Uh oh!

RazvanLiviuVarzaru commented Feb 23, 2026

Uh oh!

grooverdan commented Feb 25, 2026

Arch: amd64

Uh oh!

RazvanLiviuVarzaru commented Feb 25, 2026

Arch: amd64

Uh oh!

grooverdan commented Feb 26, 2026

Uh oh!

fauust commented May 29, 2026

Uh oh!

RazvanLiviuVarzaru commented Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fauust commented Jun 2, 2026

Uh oh!

Uh oh!

RazvanLiviuVarzaru commented Jun 2, 2026

Uh oh!

grooverdan commented Jun 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

RazvanLiviuVarzaru commented Jul 6, 2026

Uh oh!

grooverdan commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Option 1: Drop amd64-ubasan-clang-20 and amd64-ubasan-clang-20-debug early

Option 2: Defer amd64-ubasan-clang-22-debug

Uh oh!

RazvanLiviuVarzaru commented Jul 7, 2026

Option 1: Drop amd64-ubasan-clang-20 and amd64-ubasan-clang-20-debug early

Option 2: Defer amd64-ubasan-clang-22-debug

Uh oh!

grooverdan commented Jul 20, 2026

Uh oh!

RazvanLiviuVarzaru left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

3 participants

grooverdan commented Feb 23, 2026 •

edited

Loading

RazvanLiviuVarzaru commented Jun 2, 2026 •

edited

Loading

grooverdan commented Jun 3, 2026 •

edited

Loading

grooverdan commented Jul 6, 2026 •

edited

Loading

Option 1: Drop `amd64-ubasan-clang-20` and `amd64-ubasan-clang-20-debug` early

Option 2: Defer `amd64-ubasan-clang-22-debug`

Option 1: Drop `amd64-ubasan-clang-20` and `amd64-ubasan-clang-20-debug` early

Option 2: Defer `amd64-ubasan-clang-22-debug`