Avoid infinite loop in EasyConfig cache by Flamefire · Pull Request #5164 · easybuilders/easybuild-framework

Flamefire · 2026-04-09T08:04:48Z

(created using eb --new-pr)

I'm not fully sure why but #4818 leads to an infinite loop or at least heavy slowdown when parsing an easyconfig because of our way of implementing EasyConfig::copy:
We simply reparse the whole easyconfig from scratch and finalize_dependencies will parse the dependencies causing copies in the cache handling which parses their dependencies again leading to parsing the same EasyConfig literally thousands of times until I had to abort it.

The change of adding it to the cache before doing the copy of the returned easyconfig makes it better, seems to break the infinite loop at least for a test I had. Not sure we should keep that though.

Example:

`eb PyTorch-bundle-2.1.2-foss-2023a-CUDA-12.1.1.eb`

== Temporary log file in case of crash /tmp/easybuild-tmplog/easybuild-5ch8zcy7.log
== found valid index for /easybuild-easyconfigs/easybuild/easyconfigs, so using it...
== found valid index for /easybuild-easyconfigs/easybuild/easyconfigs, so using it...
COPY foss foss EasyConfig @ /easybuild-easyconfigs/easybuild/easyconfigs/f/foss/foss-2023a.eb
COPY foss foss EasyConfig @ /easybuild-easyconfigs/easybuild/easyconfigs/f/foss/foss-2023a.eb
COPY GCC GCC EasyConfig @ /easybuild-easyconfigs/easybuild/easyconfigs/g/GCC/GCC-12.3.0.eb
COPY GCC GCC EasyConfig @ /easybuild-easyconfigs/easybuild/easyconfigs/g/GCC/GCC-12.3.0.eb
COPY GCC GCC EasyConfig @ /easybuild-easyconfigs/easybuild/easyconfigs/g/GCC/GCC-12.3.0.eb
COPY GCCcore GCCcore EasyConfig @ /easybuild-easyconfigs/easybuild/easyconfigs/g/GCCcore/GCCcore-12.3.0.eb
COPY GCCcore GCCcore EasyConfig @ /easybuild-easyconfigs/easybuild/easyconfigs/g/GCCcore/GCCcore-12.3.0.eb
COPY GCCcore GCCcore EasyConfig @ /easybuild-easyconfigs/easybuild/easyconfigs/g/GCCcore/GCCcore-12.3.0.eb
COPY M4 M4 EasyConfig @ /easybuild-easyconfigs/easybuild/easyconfigs/m/M4/M4-1.4.19.eb
COPY binutils binutils EasyConfig @ /easybuild-easyconfigs/easybuild/easyconfigs/b/binutils/binutils-2.40.eb
COPY binutils binutils EasyConfig @ /easybuild-easyconfigs/easybuild/easyconfigs/b/binutils/binutils-2.40-GCCcore-12.3.0.eb
COPY GCCcore GCCcore EasyConfig @ /easybuild-easyconfigs/easybuild/easyconfigs/g/GCCcore/GCCcore-12.3.0.eb
COPY GCCcore GCCcore EasyConfig @ /easybuild-easyconfigs/easybuild/easyconfigs/g/GCCcore/GCCcore-12.3.0.eb
COPY M4 M4 EasyConfig @ /easybuild-easyconfigs/easybuild/easyconfigs/m/M4/M4-1.4.19.eb
COPY binutils binutils EasyConfig @ /easybuild-easyconfigs/easybuild/easyconfigs/b/binutils/binutils-2.40.eb
COPY OpenMPI OpenMPI EasyConfig @ /easybuild-easyconfigs/easybuild/easyconfigs/o/OpenMPI/OpenMPI-4.1.5-GCC-12.3.0.eb
COPY GCC GCC EasyConfig @ /easybuild-easyconfigs/easybuild/easyconfigs/g/GCC/GCC-12.3.0.eb
COPY GCC GCC EasyConfig @ /easybuild-easyconfigs/easybuild/easyconfigs/g/GCC/GCC-12.3.0.eb
COPY FlexiBLAS FlexiBLAS EasyConfig @ /easybuild-easyconfigs/easybuild/easyconfigs/f/FlexiBLAS/FlexiBLAS-3.3.1-GCC-12.3.0.eb
COPY GCC GCC EasyConfig @ /easybuild-easyconfigs/easybuild/easyconfigs/g/GCC/GCC-12.3.0.eb
COPY GCC GCC EasyConfig @ /easybuild-easyconfigs/easybuild/easyconfigs/g/GCC/GCC-12.3.0.eb
COPY FFTW FFTW EasyConfig @ /easybuild-easyconfigs/easybuild/easyconfigs/f/FFTW/FFTW-3.3.10-GCC-12.3.0.eb
COPY GCC GCC EasyConfig @ /easybuild-easyconfigs/easybuild/easyconfigs/g/GCC/GCC-12.3.0.eb
COPY gompi gompi EasyConfig @ /easybuild-easyconfigs/easybuild/easyconfigs/g/gompi/gompi-2023a.eb
COPY gompi gompi EasyConfig @ /easybuild-easyconfigs/easybuild/easyconfigs/g/gompi/gompi-2023a.eb
COPY GCC GCC EasyConfig @ /easybuild-easyconfigs/easybuild/easyconfigs/g/GCC/GCC-12.3.0.eb
COPY OpenMPI OpenMPI EasyConfig @ /easybuild-easyconfigs/easybuild/easyconfigs/o/OpenMPI/OpenMPI-4.1.5-GCC-12.3.0.eb
COPY GCC GCC EasyConfig @ /easybuild-easyconfigs/easybuild/easyconfigs/g/GCC/GCC-12.3.0.eb
COPY GCC GCC EasyConfig @ /easybuild-easyconfigs/easybuild/easyconfigs/g/GCC/GCC-12.3.0.eb
COPY GCCcore GCCcore EasyConfig @ /easybuild-easyconfigs/easybuild/easyconfigs/g/GCCcore/GCCcore-12.3.0.eb
COPY binutils binutils EasyConfig @ /easybuild-easyconfigs/easybuild/easyconfigs/b/binutils/binutils-2.40-GCCcore-12.3.0.eb
COPY GCCcore GCCcore EasyConfig @ /easybuild-easyconfigs/easybuild/easyconfigs/g/GCCcore/GCCcore-12.3.0.eb
COPY GCCcore GCCcore EasyConfig @ /easybuild-easyconfigs/easybuild/easyconfigs/g/GCCcore/GCCcore-12.3.0.eb
COPY M4 M4 EasyConfig @ /easybuild-easyconfigs/easybuild/easyconfigs/m/M4/M4-1.4.19.eb
COPY binutils binutils EasyConfig @ /easybuild-easyconfigs/easybuild/easyconfigs/b/binutils/binutils-2.40.eb
COPY FFTW.MPI FFTW.MPI EasyConfig @ /easybuild-easyconfigs/easybuild/easyconfigs/f/FFTW.MPI/FFTW.MPI-3.3.10-gompi-2023a.eb
COPY gompi gompi EasyConfig @ /easybuild-easyconfigs/easybuild/easyconfigs/g/gompi/gompi-2023a.eb
COPY gompi gompi EasyConfig @ /easybuild-easyconfigs/easybuild/easyconfigs/g/gompi/gompi-2023a.eb
COPY ScaLAPACK ScaLAPACK EasyConfig @ /easybuild-easyconfigs/easybuild/easyconfigs/s/ScaLAPACK/ScaLAPACK-2.2.0-gompi-2023a-fb.eb
COPY gompi gompi EasyConfig @ /easybuild-easyconfigs/easybuild/easyconfigs/g/gompi/gompi-2023a.eb
COPY gompi gompi EasyConfig @ /easybuild-easyconfigs/easybuild/easyconfigs/g/gompi/gompi-2023a.eb
COPY GCC GCC EasyConfig @ /easybuild-easyconfigs/easybuild/easyconfigs/g/GCC/GCC-12.3.0.eb
COPY OpenMPI OpenMPI EasyConfig @ /easybuild-easyconfigs/easybuild/easyconfigs/o/OpenMPI/OpenMPI-4.1.5-GCC-12.3.0.eb
COPY GCC GCC EasyConfig @ /easybuild-easyconfigs/easybuild/easyconfigs/g/GCC/GCC-12.3.0.eb
COPY gfbf gfbf EasyConfig @ /easybuild-easyconfigs/easybuild/easyconfigs/g/gfbf/gfbf-2023a.eb
COPY GCC GCC EasyConfig @ /easybuild-easyconfigs/easybuild/easyconfigs/g/GCC/GCC-12.3.0.eb
COPY FlexiBLAS FlexiBLAS EasyConfig @ /easybuild-easyconfigs/easybuild/easyconfigs/f/FlexiBLAS/FlexiBLAS-3.3.1-GCC-12.3.0.eb
COPY GCC GCC EasyConfig @ /easybuild-easyconfigs/easybuild/easyconfigs/g/GCC/GCC-12.3.0.eb
COPY FFTW FFTW EasyConfig @ /easybuild-easyconfigs/easybuild/easyconfigs/f/FFTW/FFTW-3.3.10-GCC-12.3.0.eb
COPY GCC GCC EasyConfig @ /easybuild-easyconfigs/easybuild/easyconfigs/g/GCC/GCC-12.3.0.eb
COPY GCC GCC EasyConfig @ /easybuild-easyconfigs/easybuild/easyconfigs/g/GCC/GCC-12.3.0.eb
COPY GCCcore GCCcore EasyConfig @ /easybuild-easyconfigs/easybuild/easyconfigs/g/GCCcore/GCCcore-12.3.0.eb
COPY binutils binutils EasyConfig @ /easybuild-easyconfigs/easybuild/easyconfigs/b/binutils/binutils-2.40-GCCcore-12.3.0.eb
COPY GCCcore GCCcore EasyConfig @ /easybuild-easyconfigs/easybuild/easyconfigs/g/GCCcore/GCCcore-12.3.0.eb
COPY GCCcore GCCcore EasyConfig @ /easybuild-easyconfigs/easybuild/easyconfigs/g/GCCcore/GCCcore-12.3.0.eb
COPY M4 M4 EasyConfig @ /easybuild-easyconfigs/easybuild/easyconfigs/m/M4/M4-1.4.19.eb
COPY binutils binutils EasyConfig @ /easybuild-easyconfigs/easybuild/easyconfigs/b/binutils/binutils-2.40.eb
COPY PyTorch-bundle PyTorch-bundle EasyConfig @ /easybuild-easyconfigs/easybuild/easyconfigs/p/PyTorch-bundle/PyTorch-bundle-2.1.2-foss-2023a-CUDA-12.1.1.eb
COPY foss foss EasyConfig @ /easybuild-easyconfigs/easybuild/easyconfigs/f/foss/foss-2023a.eb
== PyTorch-bundle/2.1.2-foss-2023a-CUDA-12.1.1 is already installed (module found), skipping

Copies GCCcore 47 times triggering a parse each

boegel · 2026-04-09T09:37:19Z

@Flamefire This is too much to include (very) last minute in EasyBuild v5.3.0 (which is now blocked by this issue).

We should first make the minimal changes necessary to circumvent the problem that was introduced in #4818

It seems like that would be reverting to not using _copy_ec_dicts in process_easyconfig, and going back to using [ec.copy() ...] for now.

Flamefire · 2026-04-09T10:18:20Z

We should first make the minimal changes necessary to circumvent the problem that was introduced in #4818

I meant to do that but included unrelated changes due to --new-pr. Updated.

…syconfigs are independent

boegel · 2026-04-09T10:22:21Z

This indeed also fixing the immediate problem, but I prefer going with my (even more) minimal PR, which reverts back to how things were done before #4818 was released:

avoid excessive creating of EasyConfig instances when copies are created #5166

We can take the time to properly figure out what we overlooked, and also make sure a CI test is added that catches the problem (if only by searching for the Creating.*EasyConfig instance log messages I've included in my PR)

Flamefire · 2026-04-09T10:40:02Z

Closing in favor of #5166

boegel · 2026-04-09T10:40:43Z

@Flamefire Can you open an issue with some details, as a reminder to come with a proper fix, which undoes the rollback done in #5166?

Flamefire · 2026-04-09T11:30:25Z

Done:

EasyConfigs retrieved from cache are not as-if parsed, i.e. modification causes unexpected/wrong results in later uses #5167

boegel added this to the next release (5.3.0) milestone Apr 9, 2026

boegel added the bug fix label Apr 9, 2026

boegel mentioned this pull request Apr 9, 2026

avoid excessive creating of EasyConfig instances when copies are created #5166

Merged

Avoid infinite loop in EasyConfig cache

28f4277

Flamefire force-pushed the 20260409100445_new_pr_QDUmsZcFOS branch from aa174c9 to 28f4277 Compare April 9, 2026 10:13

disable part of test_easyconfigs_caches that checks whether cached ea…

d7e5ae4

…syconfigs are independent

Flamefire closed this Apr 9, 2026

Flamefire deleted the 20260409100445_new_pr_QDUmsZcFOS branch April 9, 2026 10:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Avoid infinite loop in EasyConfig cache#5164

Avoid infinite loop in EasyConfig cache#5164
Flamefire wants to merge 2 commits intoeasybuilders:developfrom
Flamefire:20260409100445_new_pr_QDUmsZcFOS

Flamefire commented Apr 9, 2026 •

edited

Loading

Uh oh!

boegel commented Apr 9, 2026

Uh oh!

Flamefire commented Apr 9, 2026

Uh oh!

boegel commented Apr 9, 2026

Uh oh!

Flamefire commented Apr 9, 2026

Uh oh!

boegel commented Apr 9, 2026

Uh oh!

Flamefire commented Apr 9, 2026 •

edited by boegel

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Flamefire commented Apr 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

boegel commented Apr 9, 2026

Uh oh!

Flamefire commented Apr 9, 2026

Uh oh!

boegel commented Apr 9, 2026

Uh oh!

Flamefire commented Apr 9, 2026

Uh oh!

boegel commented Apr 9, 2026

Uh oh!

Flamefire commented Apr 9, 2026 • edited by boegel Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Flamefire commented Apr 9, 2026 •

edited

Loading

Flamefire commented Apr 9, 2026 •

edited by boegel

Loading