BUG: Checkpointing index failure with ExtrudedMesh.

[Molin-Han]

Describe the bug

Checkpointing fails with appending mode after 157 iterations within a KSP monitor function.

Steps to Reproduce

Here is the MFE to reproduce the bug.

from firedrake import *
from petsc4py import PETSc
print = PETSc.Sys.Print

m = PeriodicIntervalMesh(10, 1.0)
mesh = ExtrudedMesh(m, 20, extrusion_type='uniform')
x, y = SpatialCoordinate(mesh)
CG = FiniteElement("CG", interval, 1)
DG = FiniteElement("DG", interval, 0)
CG_DG = TensorProductElement(CG, DG)
RT_horiz = HDivElement(CG_DG)
DG_CG = TensorProductElement(DG, CG)
RT_vert = HDivElement(DG_CG)
RT_e = RT_horiz + RT_vert
V_2D = FunctionSpace(mesh, RT_e)
Pressure = FunctionSpace(mesh, 'DG', 0)
W = V_2D * Pressure
U = Function(W)
u, p = split(U)
w, phi = TestFunctions(W)

f = Function(V_2D).project(as_vector([x ** 2 + y ** 2, x*y]))
equ = inner(u, w) * dx  + p * phi * dx - inner(f, w) * dx

params = {
    'snes_type': 'ksponly',
    'snes_max_it':200,
    'ksp_type':'gmres',
    'ksp_monitor':None,
    'ksp_atol': 0,
    'ksp_rtol':1e-50,
    'ksp_max_it': 200,
    'ksp_gmres_restart':1000,
    'pc_type': 'none',
}

nprob = NonlinearVariationalProblem(equ, U)
nsolver = NonlinearVariationalSolver(nprob, solver_parameters= params)
sol_it = Function(W, name='sol_it')

def monitor(ksp, iteration_number, norm0):
    if iteration_number == 0:
        with CheckpointFile('sol_its.h5', 'w') as chk:
            print('Checkpointing the first solution.')
            chk.save_function(sol_it, idx=iteration_number, name=f'sol_{iteration_number}')
            print('Checkpointed the first solution.')
    else:
        with CheckpointFile('sol_its.h5', 'a') as chk:
            print(f'Checkpointing the solution at iteration {iteration_number}.')
            chk.save_function(sol_it, idx=iteration_number, name=f'sol_{iteration_number}')
            print(f'Checkpointed the solution at iteration {iteration_number}.')
nsolver.snes.ksp.setMonitor(monitor)
nsolver.solve()

Expected behavior

The monitor function for saving the solution after each iteration will fail after 157 iterations.

Error message

Traceback (most recent call last):
File "petsc4py/PETSc/petscksp.pxi", line 273, in petsc4py.PETSc.KSP_Monitor
File "/home/mh2520/molin/phd/yr2/data_test/../Boussinesq_implicit/checkpoint_mfe_it63.py", line 49, in monitor
with CheckpointFile('sol_its.h5', 'a') as chk:
File "/home/mh2520/developer-firedrake-2026-01/firedrake/firedrake/checkpointing.py", line 556, in exit
self.close()
File "/home/mh2520/developer-firedrake-2026-01/firedrake/firedrake/checkpointing.py", line 1728, in close
self.viewer.destroy()
File "petsc4py/PETSc/Viewer.pyx", line 176, in petsc4py.PETSc.Viewer.destroy
petsc4py.PETSc.Error: error code 76
[0] PetscViewerDestroy() at /home/mh2520/petsc-firedrake/petsc/src/sys/classes/viewer/interface/view.c:101
[0] PetscViewerDestroy_HDF5() at /home/mh2520/petsc-firedrake/petsc/src/sys/classes/viewer/impls/hdf5/hdf5v.c:128
[0] PetscViewerFileClose_HDF5() at /home/mh2520/petsc-firedrake/petsc/src/sys/classes/viewer/impls/hdf5/hdf5v.c:109
[0] Error in external library
[0] Error in HDF5 call H5Fclose() Status -1

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/mh2520/molin/phd/yr2/data_test/../Boussinesq_implicit/checkpoint_mfe_it63.py", line 55, in
nsolver.solve()
File "petsc4py/PETSc/Log.pyx", line 250, in petsc4py.PETSc.Log.EventDecorator.decorator.wrapped_func
File "petsc4py/PETSc/Log.pyx", line 251, in petsc4py.PETSc.Log.EventDecorator.decorator.wrapped_func
File "/home/mh2520/developer-firedrake-2026-01/firedrake/firedrake/adjoint_utils/variational_solver.py", line 108, in wrapper
out = solve(self, **kwargs)
File "/home/mh2520/developer-firedrake-2026-01/firedrake/firedrake/variational_solver.py", line 393, in solve
self.snes.solve(None, work)
File "petsc4py/PETSc/SNES.pyx", line 1740, in petsc4py.PETSc.SNES.solve
petsc4py.PETSc.Error: error code 101
[0] SNESSolve() at /home/mh2520/petsc-firedrake/petsc/src/snes/interface/snes.c:4914
[0] SNESSolve_KSPONLY() at /home/mh2520/petsc-firedrake/petsc/src/snes/impls/ksponly/ksponly.c:49
[0] KSPSolve() at /home/mh2520/petsc-firedrake/petsc/src/ksp/ksp/interface/itfunc.c:1094
[0] KSPSolve_Private() at /home/mh2520/petsc-firedrake/petsc/src/ksp/ksp/interface/itfunc.c:915
[0] KSPSolve_GMRES() at /home/mh2520/petsc-firedrake/petsc/src/ksp/ksp/impls/gmres/gmres.c:215
[0] KSPGMRESCycle() at /home/mh2520/petsc-firedrake/petsc/src/ksp/ksp/impls/gmres/gmres.c:129
[0] KSPMonitor() at /home/mh2520/petsc-firedrake/petsc/src/ksp/ksp/interface/itfunc.c:2277

Environment:

• Linux Ubuntu 22.04.4 LTS
• Python version: 3.10.12
• Firedrake is installed with developer mode.

[connorjward]

Can you find a more minimal failing example? What if you use a simpler element? What if the mesh isn't extruded?

[JHopeCollins]

What if you use a simpler element? What if the mesh isn't extruded?

If I remember our discussion correctly, this failure doesn't show up if the mesh isn't extruded, or if you don't have a mixed space.
It also doesn't happen if the checkpointing is just in a loop rather than in the custom monitor.

@Molin-Han Is this correct?

[JHopeCollins]

@Molin-Han what are your HDF5 and h5py versions? Can you show the output of apt show libhdf5-mpi-dev and pip show h5py?

[Molin-Han]

What if you use a simpler element? What if the mesh isn't extruded?

If I remember our discussion correctly, this failure doesn't show up if the mesh isn't extruded, or if you don't have a mixed space. It also doesn't happen if the checkpointing is just in a loop rather than in the custom monitor.

@Molin-Han Is this correct?

Yes, that is correct.

[Molin-Han]

@Molin-Han what are your HDF5 and h5py versions? Can you show the output of apt show libhdf5-mpi-dev and pip show h5py?

The HDF5 version is 1.10.7+repack-4ubuntu2, and the h5py version is 3.15.1.

Here is more information with outputs:
HDF5:
Package: libhdf5-mpi-dev Version: 1.10.7+repack-4ubuntu2 Priority: extra Section: universe/libdevel Source: hdf5 Origin: Ubuntu Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com> Original-Maintainer: Gilles Filippini <pini@debian.org> Bugs: https://bugs.launchpad.net/ubuntu/+filebug Installed-Size: 49.2 kB Depends: libhdf5-openmpi-dev, mpi-default-dev Homepage: http://hdfgroup.org/HDF5/ Download-Size: 4,610 B APT-Manual-Installed: yes APT-Sources: http://gb.archive.ubuntu.com/ubuntu jammy/universe amd64 Packages Description: HDF5 - development files - default MPI version Hierarchical Data Format 5 (HDF5) is a file format and library for storing scientific data. HDF5 was designed and implemented to address the deficiencies of HDF4.x. It has a more powerful and flexible data model, supports files larger than 2 GB, and supports parallel I/O. . This package depends on the default MPI version of HDF5 for each platform.

h5py:
Name: h5py Version: 3.15.1 Summary: Read and write HDF5 files from Python Home-page: Author: Author-email: Andrew Collette <andrew.collette@gmail.com> License: Location: /home/mh2520/developer-firedrake-2026-01/venv-firedrake-developer-2026-01/lib/python3.10/site-packages Requires: mpi4py, numpy Required-by: firedrake