hotfix - safeguard invalid scalar indices to prevent segmentation faults during LBC/IC generation by guoqing-noaa · Pull Request #263 · ufs-community/MPAS-Model

guoqing-noaa · 2026-06-19T22:04:51Z

@keenan and I got the same lbc task crash on Hera recently. All lbc files were generated correctly but the task failed due to segmentation fault occurred:

+2s exrrfs_lbc.sh[101]: srun ./init_atmosphere_model.x
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image              PC                Routine            Line        Source
libpthread-2.28.s  000014DB92A91990  Unknown               Unknown  Unknown
libc-2.28.so       000014DB923C2811  cfree                 Unknown  Unknown
libpnetcdf.so.4.0  000014DB9D9D5198  for_deallocate_ha     Unknown  Unknown
init_atmosphere_m  00000000006491D8  Unknown               Unknown  Unknown
init_atmosphere_m  0000000000658966  Unknown               Unknown  Unknown
init_atmosphere_m  000000000065877A  Unknown               Unknown  Unknown
init_atmosphere_m  0000000000643855  Unknown               Unknown  Unknown
init_atmosphere_m  00000000005B38D5  Unknown               Unknown  Unknown
init_atmosphere_m  0000000000408FC0  Unknown               Unknown  Unknown
init_atmosphere_m  0000000000408BC6  Unknown               Unknown  Unknown
init_atmosphere_m  0000000000408B3D  Unknown               Unknown  Unknown
libc-2.28.so       000014DB92361865  __libc_start_main     Unknown  Unknown
init_atmosphere_m  0000000000408A5E  Unknown               Unknown  Unknown
forrtl: severe (174): SIGSEGV, segmentation fault occurred

It looks similar to the behavior we met before when running the model where the executable just could not terminate correctly.

I tried a debug build and identified that the crash happened at line 8577 of mpas_init_atm_cases.F:

+2s exrrfs_lbc.sh[101]: srun ./init_atmosphere_model.x
forrtl: severe (408): fort: (3): Subscript #1 of the array SCALARS has value -1 which is less than the lower bound of 1

Image              PC                Routine            Line        Source
init_atmosphere_m  000000000081DDE8  init_atm_cases_mp        8577  mpas_init_atm_cases.F
init_atmosphere_m  0000000000556207  init_atm_cases_mp         404  mpas_init_atm_cases.F
init_atmosphere_m  000000000054C788  init_atm_core_mp_          92  mpas_init_atm_core.F
init_atmosphere_m  000000000040E70C  mpas_subdriver_mp         417  mpas_subdriver.F
init_atmosphere_m  00000000004085CD  MAIN__                     20  mpas.F
init_atmosphere_m  00000000004084FD  Unknown               Unknown  Unknown
libc-2.28.so       000014ED04092865  __libc_start_main     Unknown  Unknown
init_atmosphere_m  000000000040841E  Unknown               Unknown  Unknown
forrtl: severe (408): fort: (3): Subscript #1 of the array SCALARS has value -1 which is less than the lower bound of 1

and lines 8577-8581 read as follows:

        scalars(index_nc,:,iCell) = 0._RKIND
        scalars(index_ni,:,iCell) = 0._RKIND
        scalars(index_nr,:,iCell) = 0._RKIND
        scalars(index_ns,:,iCell) = 0._RKIND
        scalars(index_ng,:,iCell) = 0._RKIND

So it suggested a -1 of index_nc caused the crash.

In src/core_init_atmosphere/Registry.xml, we can find that when lbc_hydrometeors_gfs is true while lbc_hydrometeors_rrfs is false, we only have lbc_qv, lbc_qc, lbc_qr, lbc_qi, lbc_qs, lbc_qg, do not have lbc_nc/ni,..... Hence we will of course get -1 for their indexes (index_nc, ect).

                <var_array name="lbc_scalars" type="real" dimensions="nVertLevels nCells Time" packages="lbcs">
                        <var name="lbc_qv" array_group="moist" units="kg kg^{-1}"
                             description="Water vapor mixing ratio on lateral boundary cells"/>

                        <var name="lbc_qc" array_group="moist" units="kg kg^{-1}"
                             description="Cloud water mixing ratio on lateral boundary cells"
                             packages="lbc_hydrometeors_rrfs;lbc_hydrometeors_gfs"/>

                        <var name="lbc_qr" array_group="moist" units="kg kg^{-1}"
                             description="Rain water mixing ratio on lateral boundary cells"
                             packages="lbc_hydrometeors_rrfs;lbc_hydrometeors_gfs"/>

                        <var name="lbc_qi" array_group="moist" units="kg kg^{-1}"
                             description="Lateral boundary tendency of ice mixing ratio"
                             packages="lbc_hydrometeors_rrfs;lbc_hydrometeors_gfs"/>

                        <var name="lbc_qs" array_group="moist" units="kg kg^{-1}"
                             description="Lateral boundary tendency of snow mixing ratio"
                             packages="lbc_hydrometeors_rrfs;lbc_hydrometeors_gfs"/>

                        <var name="lbc_qg" array_group="moist" units="kg kg^{-1}"
                             description="Lateral boundary tendency of graupel mixing ratio"
                             packages="lbc_hydrometeors_rrfs;lbc_hydrometeors_gfs"/>

                        <var name="lbc_qh" array_group="moist"  units="kg kg^{-1} s^{-1}"
                             description="Lateral boundary tendency of hail mixing ratio"
                             packages="lbc_hydrometeors_rrfs"/>

                        <var name="lbc_nr" array_group="number" units="kg^{-1}"
                             description="Lateral boundary tendency of rain number concentration"
                             packages="lbc_hydrometeors_rrfs"/>

                        <var name="lbc_ni" array_group="number" units="kg^{-1}"
                             description="Lateral boundary tendency of ice number concentration"
                             packages="lbc_hydrometeors_rrfs"/>

                        <var name="lbc_nc" array_group="number" units="kg^{-1}"
                             description="Lateral boundary tendency of droplet number concentration"
                             packages="lbc_hydrometeors_rrfs"/>
......
......

This PR provides a hotfix by checking if (index_nc > 0) before setting scalars(index_nc,:,iCell) = 0._RKIND.
Similar safeguards are applied to IC generation where index_nwfa and index_nifa are only available for specific packages.

I don't have a full answer to why this only crashes on Hera but not on other machines.
Different machines have different memory/heap layout, the out-of-bounds address may still be within a valid memory page, so no SIGSEGV occurs and the program finishes "successfully".

Mandatory Questions

Does this PR include any additions or changes to external inputs (e.g., microphysics lookup tables, static data for gravity-wave drag -- things like that)?
- no
Does this PR require updating one or more baselines for the CI tests? If so, what?
- no

Reviews

Is this PR currently draft, still being worked on, or ready for review?
- ready for review
For when this PR begins review, please list the developers/collaborators you'd like to prioritize for review; e.g.,:
- @clark-evans
- @dustinswales

…g LBC/IC genereation

AndersJensen-NOAA · 2026-07-09T03:03:46Z

@guoqing-noaa, I was able to recreate this issue. I think that the config variables config_lbc_hydrometeors_rrfs and config_lbc_hydrometeors_gfs should be used in the lbc subroutine where this code is failing.

if (config_lbc_hydrometors_gfs) then
   scalars(index_qc,:,iCell) = 0._RKIND
   ...
   ...

AndersJensen-NOAA · 2026-07-09T03:05:20Z

         if (associated(index_dust_coarse ) ) sorted_arrQ13(:,:) = -999.0
-         scalars(index_nwfa,:,iCell) = 0._RKIND
-         scalars(index_nifa,:,iCell) = 0._RKIND
+         if (index_nwfa > 0) scalars(index_nwfa,:,iCell) = 0._RKIND


I wouldn't touch anything that is not in the init_atm_case_lbc subroutine for now.

AndersJensen-NOAA · 2026-07-09T03:05:45Z

+            if (index_nwfa > 0) scalars(index_nwfa,k,iCell) = vertical_interp(target_z, nfglevels_actual-1, &
                                        sorted_arrQ9(:,1:nfglevels_actual-1), order=1, extrap=0)
-            scalars(index_nifa,k,iCell) = vertical_interp(target_z, nfglevels_actual-1, &
+            if (index_nifa > 0) scalars(index_nifa,k,iCell) = vertical_interp(target_z, nfglevels_actual-1, &


I wouldn't touch anything that is not in the init_atm_case_lbc subroutine for now.

AndersJensen-NOAA · 2026-07-09T03:06:13Z

         if (associated(index_smoke_fine  )) sorted_arrQ11(:,:) = -999.0
         if (associated(index_dust_fine   )) sorted_arrQ12(:,:) = -999.0
         if (associated(index_dust_coarse )) sorted_arrQ13(:,:) = -999.0
-         scalars(index_nwfa,:,iCell) = 0._RKIND


see comment on suggested changes using the config variable, which is safer

@AndersJensen-NOAA I understand your preference to use config_lbc_hydrometeors_gfs/config_lbc_hydrometeors_rrfs to do the safe guard. However, why using the config_lbc options is safer? Could you elaborate more?

To my understanding, if (index_qc > 0) scalars(index_qc,:,iCell) = 0._RKIND is a more generic safe guard. Even one changes what variables exist for each different config_lbc options, the if (index_qc > 0) logic is still valid and don't need modifications.

clark-evans · 2026-07-24T20:24:11Z

@guoqing-noaa if you're able to address Anders' comments before next Wednesday (7/29), we should be able to get this merged by then.

guoqing-noaa · 2026-07-26T05:13:31Z

@AndersJensen-NOAA Thanks for the review and disucssion.

I would think if (index_qc > 0) scalars(index_qc,k,iCell) = .... is a much more generic safeguard than that based on the config_lbc options.

Similar strategy has been used in other places in mpas_init_atm_cases.F, such as:

MPAS-Model/src/core_init_atmosphere/mpas_init_atm_cases.F

Lines 8858 to 8867 in 02295fe

    
           IF ( index_qs > 0 .and. index_ns > 0 ) THEN 
        
             IF ( scalars(index_qs,k,iCell) > 0.0 .and. scalars(index_ns,k,iCell) == 0.0 ) THEN 
        
               scalars(index_ns,k,iCell) = calcnfromq(qsw=scalars(index_qs,k,iCell),dn=rho_zz(k,iCell) ) 
        
             ENDIF 
        
           ENDIF 
        
           IF ( index_qg > 0 .and. index_ng > 0 ) THEN 
        
             IF ( scalars(index_qg,k,iCell) > 0.0 .and. scalars(index_ng,k,iCell) == 0.0 ) THEN 
        
               scalars(index_ng,k,iCell) = calcnfromq(qhw=scalars(index_qg,k,iCell),dn=rho_zz(k,iCell) ) 
        
             ENDIF 
        
           ENDIF

However, if you would like to use the config_lbc option, I am fine to change accordingly to retrieve the config_lbc options inside the subroutine and use them to do the safeguard.
call mpas_pool_get_config(configs, 'config_lbc_hydrometeors_gfs', config_lbc_hydrometeors_gfs).....

For

         scalars(index_nwfa,:,iCell) = 0._RKIND
         scalars(index_nifa,:,iCell) = 0._RKIND

I will restore to leave them untouched. But we know it poses a potential issue there since the availability of nifa and nwfa depends on a certain package and it may give us a similar "silent" segmentation fault in some situations.

guoqing-noaa · 2026-07-26T05:47:01Z

Update: @AndersJensen-NOAA I think I got your concern. In some systems, dereferencing a disassociated pointer is undefined behavior, so if (index_qc > 0) may crash or NOT work as expected. Thanks for bringing this up.

On the other hand, we may want to use a more robust safe guard that handles any package combination that might enable/disable the field.

I would suggest we adopt the method used by chemistry code (see below examples) to be more robust and future-proof.

integer, pointer :: index_qv, index_qc, index_qr, index_qi, index_qs, index_qg, index_ni, index_nc, index_nr, index_ns, index_ng
integer, pointer :: index_nwfa, index_nifa
integer, pointer :: index_smoke_fine => null(), index_dust_fine => null(), index_dust_coarse => null()
......
if (associated(index_smoke_fine)) then
   if (index_smoke_fine > 0) ...
endif

…c subroutine

Safeguard invalid scalar indices to prevent segmentation faults durin…

e66493a

…g LBC/IC genereation

guoqing-noaa requested review from clark-evans and dustinswales as code owners June 19, 2026 22:04

github-project-automation Bot added this to MPAS-A PRs to process Jun 19, 2026

github-project-automation Bot moved this to Not assigned in MPAS-A PRs to process Jun 19, 2026

clark-evans approved these changes Jun 21, 2026

View reviewed changes

clark-evans moved this from Not assigned to Needs review in MPAS-A PRs to process Jun 21, 2026

guoqing-noaa mentioned this pull request Jun 22, 2026

fix the MPASSIT build failure and lbc task crashes on Hera; allow setting up RRFSv2x and RRFSv2x_ens directly out-of-the-box; add the SAT_USELIST option to turn on/off radiance DA explicitly NOAA-EMC/rrfs-workflow#1559

Merged

dustinswales approved these changes Jun 22, 2026

View reviewed changes

guoqing-noaa force-pushed the segfault_hotfix4ufs branch from e4a5d7c to e66493a Compare June 26, 2026 22:54

clark-evans requested a review from AndersJensen-NOAA July 1, 2026 21:51

clark-evans assigned guoqing-noaa Jul 1, 2026

AndersJensen-NOAA reviewed Jul 9, 2026

View reviewed changes

Merge branch 'noaa/develop' into segfault_hotfix4ufs

21e6db8

guoqing-noaa added 3 commits July 26, 2026 07:53

Change to a more robust safe guard

9400e36

revert changes on index_nwfa, index_nifa outside the init_atm_case_lb…

250ffc1

…c subroutine

fine tuning

dab4c63

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

hotfix - safeguard invalid scalar indices to prevent segmentation faults during LBC/IC generation - #263

hotfix - safeguard invalid scalar indices to prevent segmentation faults during LBC/IC generation#263
guoqing-noaa wants to merge 5 commits into
ufs-community:noaa/developfrom
guoqing-noaa:segfault_hotfix4ufs

guoqing-noaa commented Jun 19, 2026 •

edited

Loading

Uh oh!

AndersJensen-NOAA commented Jul 9, 2026

Uh oh!

AndersJensen-NOAA Jul 9, 2026

Uh oh!

AndersJensen-NOAA Jul 9, 2026

Uh oh!

AndersJensen-NOAA Jul 9, 2026

Uh oh!

guoqing-noaa Jul 26, 2026

Uh oh!

clark-evans commented Jul 24, 2026

Uh oh!

guoqing-noaa commented Jul 26, 2026

Uh oh!

guoqing-noaa commented Jul 26, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

guoqing-noaa commented Jun 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Mandatory Questions

Reviews

Uh oh!

AndersJensen-NOAA commented Jul 9, 2026

Uh oh!

AndersJensen-NOAA Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

AndersJensen-NOAA Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

AndersJensen-NOAA Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

guoqing-noaa Jul 26, 2026

Choose a reason for hiding this comment

Uh oh!

clark-evans commented Jul 24, 2026

Uh oh!

guoqing-noaa commented Jul 26, 2026

Uh oh!

guoqing-noaa commented Jul 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

guoqing-noaa commented Jun 19, 2026 •

edited

Loading

guoqing-noaa commented Jul 26, 2026 •

edited

Loading