nvhpc test fails since update to match alpha08l externals

[peverwhee]

What happened?

Since #1511,

The nvhpc test fails the restart comparison. This test works with intel, so we did not fix it for that PR, but it needs to be investigated.

The differences are in the cpl.hi file:

/glade/derecho/scratch/cacraig/aux_cam_nvhpc_20260321143458/ERS_Ln9.ne30pg3_ne30pg3_mt232.FHISTC_LTso.derecho_nvhpc.cam-outfrq9s_gpu_default.GC.aux_cam_nvhpc_20260321143458/run/ERS_Ln9.ne30pg3_ne30pg3_mt232.FHISTC_LTso.derecho_nvhpc.cam-outfrq9s_gpu_default.GC.aux_cam_nvhpc_20260321143458.cpl.hi.1979-01-01-16200.nc.base.cprnc.out

What are the steps to reproduce the bug?

1. Check out cam_development
2. Run bin/git-fleximod update
3. Run the nvhpc test: env CAM_FC=nvhpc BL_TESTDIR=<latest baselines> ./test_driver.sh -f
4. Observe the failure in the test case (TestStatus)

What CAM tag were you using?

cam6_4_158

What machine were you running CAM on?

CISL machine (e.g. cheyenne)

What compiler were you using?

NVHPC

Path to a case directory, if applicable

/glade/derecho/scratch/cacraig/aux_cam_nvhpc_20260321143458/ERS_Ln9.ne30pg3_ne30pg3_mt232.FHISTC_LTso.derecho_nvhpc.cam-outfrq9s_gpu_default.GC.aux_cam_nvhpc_20260321143458

Will you be addressing this bug yourself?

Any CAM SE can do this

Extra info

No response

[sjsprecious]

So I am working with an NVIDIA engineer to resolve this issue. Some findings so far:

• Using DEBUG=TRUE option passes the ERS test with nvhpc/25.9.
• Using explicit -O1 optimization level passes the ERS test with nvhpc/25.9.
• Currently CAM uses -O optimization level, which is similar to -O1 level but adds LLVM optimization.

[sjsprecious]

The NVIDIA engineer provided a temporary workaround to fix this issue. The ccs_config PR is at ESMCI/ccs_config_cesm#284 and the new tag is ccs_config_cesm1.0.81.

[cacraigucar]

Using the ccs_config_cesm1.0.81, the nvhpc test is still failing baseline comparisons. See the job at: /glade/derecho/scratch/cacraig/aux_cam_nvhpc_20260410140246. Note that I suspect this ccs_config is also giving me a couple of other problems, so I will most likely be backing it out of my current misc tag and try including it at a later date

[sjsprecious]

Thanks @cacraigucar for testing it. Interesting, I checked out the cam6_4_158 tag on Derecho with the changes in ccs_config_cesm1.0.81. The ERS passed for the GPU test.

aux_cam_nvhpc_20260413130619: 1 test
  ERS_Ln9.ne30pg3_ne30pg3_mt232.FHISTC_LTso.derecho_nvhpc.cam-outfrq9s_gpu_default (Overall: PASS) details:
    PASS ERS_Ln9.ne30pg3_ne30pg3_mt232.FHISTC_LTso.derecho_nvhpc.cam-outfrq9s_gpu_default CREATE_NEWCASE
    PASS ERS_Ln9.ne30pg3_ne30pg3_mt232.FHISTC_LTso.derecho_nvhpc.cam-outfrq9s_gpu_default XML
    PASS ERS_Ln9.ne30pg3_ne30pg3_mt232.FHISTC_LTso.derecho_nvhpc.cam-outfrq9s_gpu_default SETUP
    PASS ERS_Ln9.ne30pg3_ne30pg3_mt232.FHISTC_LTso.derecho_nvhpc.cam-outfrq9s_gpu_default SHAREDLIB_BUILD time=760
    PASS ERS_Ln9.ne30pg3_ne30pg3_mt232.FHISTC_LTso.derecho_nvhpc.cam-outfrq9s_gpu_default MODEL_BUILD time=777
    PASS ERS_Ln9.ne30pg3_ne30pg3_mt232.FHISTC_LTso.derecho_nvhpc.cam-outfrq9s_gpu_default SUBMIT
    PASS ERS_Ln9.ne30pg3_ne30pg3_mt232.FHISTC_LTso.derecho_nvhpc.cam-outfrq9s_gpu_default RUN time=5549
    PASS ERS_Ln9.ne30pg3_ne30pg3_mt232.FHISTC_LTso.derecho_nvhpc.cam-outfrq9s_gpu_default COMPARE_base_rest
    PASS ERS_Ln9.ne30pg3_ne30pg3_mt232.FHISTC_LTso.derecho_nvhpc.cam-outfrq9s_gpu_default MEMLEAK insufficient data for memleak test
    PASS ERS_Ln9.ne30pg3_ne30pg3_mt232.FHISTC_LTso.derecho_nvhpc.cam-outfrq9s_gpu_default SHORT_TERM_ARCHIVER

The output can be found at /glade/derecho/scratch/sunjian/aux_cam_nvhpc_20260413130619.

I assume that you are using a newer CAM tag. What changes are introduced since cam6_4_158?

[cacraigucar]

The source code I am testing is in: https://github.com/ESCOMP/CAM/pull/1530/files

It is currently based on cam6_4_162 with some additional misc mods. I will be merging cam6_4_163 into it very soon.

The changes since cam6_4_158:

• cam6_4_159: Updates for namelists for spin-up
• cam6_4_160: TUV-X photolysis
• cam6_4_161: fixes for root dirs in cesm blds (for TUV-X)
• cam6_4_162: Derecho gnu compiler workarounds and fix issue with aerosol optics diagnostics

[sjsprecious]

Thanks @cacraigucar . I checked out the cam6_4_162 tag on Derecho with the changes in ccs_config_cesm1.0.81. The ERS still passed for the GPU test.

aux_cam_nvhpc_20260413181130: 1 test
  ERS_Ln9.ne30pg3_ne30pg3_mt232.FHISTC_LTso.derecho_nvhpc.cam-outfrq9s_gpu_default (Overall: PASS) details:
    PASS ERS_Ln9.ne30pg3_ne30pg3_mt232.FHISTC_LTso.derecho_nvhpc.cam-outfrq9s_gpu_default CREATE_NEWCASE
    PASS ERS_Ln9.ne30pg3_ne30pg3_mt232.FHISTC_LTso.derecho_nvhpc.cam-outfrq9s_gpu_default XML
    PASS ERS_Ln9.ne30pg3_ne30pg3_mt232.FHISTC_LTso.derecho_nvhpc.cam-outfrq9s_gpu_default SETUP
    PASS ERS_Ln9.ne30pg3_ne30pg3_mt232.FHISTC_LTso.derecho_nvhpc.cam-outfrq9s_gpu_default SHAREDLIB_BUILD time=777
    PASS ERS_Ln9.ne30pg3_ne30pg3_mt232.FHISTC_LTso.derecho_nvhpc.cam-outfrq9s_gpu_default MODEL_BUILD time=936
    PASS ERS_Ln9.ne30pg3_ne30pg3_mt232.FHISTC_LTso.derecho_nvhpc.cam-outfrq9s_gpu_default SUBMIT
    PASS ERS_Ln9.ne30pg3_ne30pg3_mt232.FHISTC_LTso.derecho_nvhpc.cam-outfrq9s_gpu_default RUN time=5583
    PASS ERS_Ln9.ne30pg3_ne30pg3_mt232.FHISTC_LTso.derecho_nvhpc.cam-outfrq9s_gpu_default COMPARE_base_rest
    PASS ERS_Ln9.ne30pg3_ne30pg3_mt232.FHISTC_LTso.derecho_nvhpc.cam-outfrq9s_gpu_default MEMLEAK insufficient data for memleak test
    PASS ERS_Ln9.ne30pg3_ne30pg3_mt232.FHISTC_LTso.derecho_nvhpc.cam-outfrq9s_gpu_default SHORT_TERM_ARCHIVER

The output can be found at /glade/derecho/scratch/sunjian/aux_cam_nvhpc_20260413181130.

I will take a look at your PR and see if there is anything that could break the nvhpc test.