Cache Shell Script of `spack load` and `spack env activate` by RikkiButler20 · Pull Request #47755 · spack/spack

RikkiButler20 · 2024-11-23T02:39:33Z

This will fix the problem of spack load and spack env activate taking too long and currently iterating over all the package's dependency's python package files. This will also prevents the drifting of the installed spec from the current, updated version on Spack

adamjstewart · 2024-11-23T11:14:05Z

Very excited for this and happy to test on fish!

aweits · 2024-11-24T13:47:59Z

What if we cached the environment changes at install-time instead? That would also solve issues related to allowing older installs to still function/load... (willing to help with this, would allow me to jettison some local code that is getting harder and harder to maintain)

tgamblin · 2024-11-24T14:06:12Z

What if we cached the environment changes at install-time instead?

This is the plan. There are some details in the slack discussion linked from #47603, and the bit about “package drift” there.

i think @RikkiButler20 is doing this in steps (see checklist) — probably need one or two more to move the logic to package install time and env install time.

@aweits I think if you want to help, iterating on reviews would be great so we can get this in for 1.0!

RikkiButler20 · 2024-11-27T01:07:41Z

@spackbot Fix style

spackbot-app · 2024-11-27T01:07:44Z

Let me see if I can fix that for you!

spackbot-app · 2024-11-27T01:09:19Z

I was able to run spack style --fix for you!

spack style --fix

==> Running style checks on spack
  selected: import, isort, black, flake8, mypy
==> Modified files
  lib/spack/spack/cmd/load.py
  lib/spack/spack/test/cmd/load.py
==> Running import checks
import check requires Python 3.9 or later
  import checks were clean
==> Running isort checks
  isort checks were clean
==> Running black checks
reformatted lib/spack/spack/cmd/load.py
reformatted lib/spack/spack/test/cmd/load.py
All done! ✨ 🍰 ✨
2 files reformatted.
  black checks were clean
==> Running flake8 checks
  flake8 checks were clean
==> Running mypy checks
lib/spack/spack/version/version_types.py:145: error: Argument 2 to "StandardVersion" has incompatible type "*Tuple[Tuple[Any, ...], Tuple[Any, ...]]"; expected "Tuple[Tuple[Any, ...], Tuple[Any, ...]]"  [arg-type]
lib/spack/spack/version/version_types.py:452: error: Argument 2 to "StandardVersion" has incompatible type "*Tuple[Tuple[Any, ...], Tuple[Any, ...]]"; expected "Tuple[Tuple[Any, ...], Tuple[Any, ...]]"  [arg-type]
lib/spack/spack/version/version_types.py:481: error: Argument 2 to "StandardVersion" has incompatible type "*Tuple[Tuple[Any, ...], Tuple[Any, ...]]"; expected "Tuple[Tuple[Any, ...], Tuple[Any, ...]]"  [arg-type]
lib/spack/spack/variant.py:131: error: Unsupported right operand type for in ("Union[Collection[Any], Callable[..., Any]]")  [operator]
Found 4 errors in 2 files (checked 624 source files)
  mypy found errors

Keep in mind that I cannot fix your flake8 or mypy errors, so if you have any you'll need to fix them and update the pull request. If I was able to push to your branch, if you make further changes you will need to pull from your updated branch before pushing again.

I've updated the branch with style fixes.

RikkiButler20 · 2024-12-05T01:15:56Z

@spackbot Fix style

spackbot-app · 2024-12-05T01:15:59Z

Let me see if I can fix that for you!

spackbot-app · 2024-12-05T01:17:28Z

I was able to run spack style --fix for you!

spack style --fix

==> Running style checks on spack
  selected: import, isort, black, flake8, mypy
==> Modified files
  lib/spack/spack/cmd/load.py
  lib/spack/spack/test/cmd/load.py
==> Running import checks
import check requires Python 3.9 or later
  import checks were clean
==> Running isort checks
  isort checks were clean
==> Running black checks
reformatted lib/spack/spack/test/cmd/load.py
All done! ✨ 🍰 ✨
1 file reformatted, 1 file left unchanged.
  black checks were clean
==> Running flake8 checks
  flake8 checks were clean
==> Running mypy checks
lib/spack/spack/version/version_types.py:145: error: Argument 2 to "StandardVersion" has incompatible type "*Tuple[Tuple[Any, ...], Tuple[Any, ...]]"; expected "Tuple[Tuple[Any, ...], Tuple[Any, ...]]"  [arg-type]
lib/spack/spack/version/version_types.py:452: error: Argument 2 to "StandardVersion" has incompatible type "*Tuple[Tuple[Any, ...], Tuple[Any, ...]]"; expected "Tuple[Tuple[Any, ...], Tuple[Any, ...]]"  [arg-type]
lib/spack/spack/version/version_types.py:481: error: Argument 2 to "StandardVersion" has incompatible type "*Tuple[Tuple[Any, ...], Tuple[Any, ...]]"; expected "Tuple[Tuple[Any, ...], Tuple[Any, ...]]"  [arg-type]
lib/spack/spack/variant.py:131: error: Unsupported right operand type for in ("Union[Collection[Any], Callable[..., Any]]")  [operator]
Found 4 errors in 2 files (checked 624 source files)
  mypy found errors

Keep in mind that I cannot fix your flake8 or mypy errors, so if you have any you'll need to fix them and update the pull request. If I was able to push to your branch, if you make further changes you will need to pull from your updated branch before pushing again.

I've updated the branch with style fixes.

adamjstewart · 2025-01-15T11:01:48Z

Going to unsubscribe from this PR while it's still a WIP (too many notifications) but please ping me to review fish support once it is complete.

RikkiButler20 · 2025-01-16T20:27:11Z

@spackbot Fix style

spackbot-app · 2025-01-16T20:27:14Z

Let me see if I can fix that for you!

spackbot-app · 2025-01-16T20:28:19Z

I was able to run spack style --fix for you!

spack style --fix

==> Running style checks on spack
  selected: import, isort, black, flake8, mypy
==> Modified files
  lib/spack/spack/cmd/load.py
  lib/spack/spack/hooks/__init__.py
  lib/spack/spack/hooks/cache_shell_script.py
  lib/spack/spack/test/environment_modifications.py
  lib/spack/spack/test/hooks/cache_shell_script.py
  lib/spack/spack/test/util/environment.py
  lib/spack/spack/util/environment.py
==> Running import checks
import check requires Python 3.9 or later
  import checks were clean
==> Running isort checks
Fixing /tmp/tmpu87evsa3/spack/lib/spack/spack/test/hooks/cache_shell_script.py
  isort checks were clean
==> Running black checks
reformatted lib/spack/spack/test/hooks/cache_shell_script.py
reformatted lib/spack/spack/util/environment.py
All done! ✨ 🍰 ✨
2 files reformatted, 5 files left unchanged.
  black checks were clean
==> Running flake8 checks
lib/spack/spack/cmd/load.py:12: [F401] 'spack.user_environment as uenv' imported but unused
lib/spack/spack/cmd/load.py:13: [F401] 'spack.util.environment' imported but unused
  flake8 found errors
==> Running mypy checks
lib/spack/spack/util/environment.py:662: error: Item "NameModifier" of "Union[NameModifier, NameValueModifier]" has no attribute "_cache_str"  [union-attr]
lib/spack/spack/util/environment.py:662: error: Item "NameValueModifier" of "Union[NameModifier, NameValueModifier]" has no attribute "_cache_str"  [union-attr]
lib/spack/spack/version/version_types.py:135: error: Incompatible types in assignment (expression has type "Tuple[Any, ...]", variable has type "Tuple[str]")  [assignment]
lib/spack/spack/variant.py:130: error: Unsupported right operand type for in ("Union[Collection[Any], Callable[..., Any]]")  [operator]
lib/spack/spack/build_environment.py:171: error: Overloaded function signatures 1 and 2 overlap with incompatible return types  [misc]
lib/spack/spack/build_environment.py:171: error: Overloaded function signatures 1 and 3 overlap with incompatible return types  [misc]
Found 6 errors in 4 files (checked 636 source files)
  mypy found errors

Keep in mind that I cannot fix your flake8 or mypy errors, so if you have any you'll need to fix them and update the pull request. If I was able to push to your branch, if you make further changes you will need to pull from your updated branch before pushing again.

I've updated the branch with style fixes.

Remove dead arguments hace windows shells use _spack_env_set & unset Signed-off-by: Kayla Butler <butler59@llnl.gov>

Signed-off-by: Kayla Butler <butler59@llnl.gov>

Fix style Regenerate deactivation script when lockfile is updated Signed-off-by: Kayla Butler <butler59@llnl.gov>

Signed-off-by: Kayla Butler <butler59@llnl.gov>

RikkiButler20 · 2026-06-05T03:55:03Z

@spackbot fix style

spackbot-app · 2026-06-05T03:55:06Z

Let me see if I can fix that for you!

spackbot-app · 2026-06-05T03:56:06Z

I was able to run spack style --fix for you!

spack style --fix

==> Running style checks on spack
  selected: import, ruff-format, ruff-check, mypy
==> Running import checks
  import checks were clean
==> Running ruff-format checks
  ruff-format checks were clean
==> Running ruff-check checks
  ruff-check checks were clean
==> Running mypy checks
Success: no issues found in 639 source files
  mypy checks were clean
==> spack style checks were clean

Keep in mind that I cannot fix your flake8 or mypy errors, so if you have any you'll need to fix them and update the pull request. If I was able to push to your branch, if you make further changes you will need to pull from your updated branch before pushing again.

I've updated the branch with style fixes.

Signed-off-by: Kayla Butler <butler59@llnl.gov>

The separator is different for windows scripts Signed-off-by: Kayla Butler <butler59@llnl.gov>

Signed-off-by: Kayla Butler <butler59@llnl.gov>

RikkiButler20 · 2026-06-09T03:20:05Z

@spackbot fix style

spackbot-app · 2026-06-09T03:20:08Z

Let me see if I can fix that for you!

spackbot-app · 2026-06-09T03:21:04Z

I was able to run spack style --fix for you!

spack style --fix

==> Running style checks on spack
  selected: import, ruff-format, ruff-check, mypy
==> Running import checks
  import checks were clean
==> Running ruff-format checks
  ruff-format checks were clean
==> Running ruff-check checks
E501 Line too long (100 > 99)
   --> lib/spack/spack/test/hooks/generate_spec_scripts.py:104:100
    |
102 |         )
103 |         assert re.search(
104 |             f"_spack_env_remove_value {uenv.spack_loaded_hashes_var} {pkg.dag_hash()} {os.pathsep}",
    |                                                                                                    ^
105 |             unload_script,
106 |         )
    |
  ruff-check found errors
==> Running mypy checks
Success: no issues found in 639 source files
  mypy checks were clean

Keep in mind that I cannot fix your flake8 or mypy errors, so if you have any you'll need to fix them and update the pull request. If I was able to push to your branch, if you make further changes you will need to pull from your updated branch before pushing again.

I've updated the branch with style fixes.

RikkiButler20 · 2026-06-09T03:32:10Z

@spackbot fix style

spackbot-app · 2026-06-09T03:32:13Z

Let me see if I can fix that for you!

spackbot-app · 2026-06-09T03:33:08Z

I was able to run spack style --fix for you!

spack style --fix

==> Running style checks on spack
  selected: import, ruff-format, ruff-check, mypy
==> Running import checks
  import checks were clean
==> Running ruff-format checks
  ruff-format checks were clean
==> Running ruff-check checks
E501 Line too long (100 > 99)
   --> lib/spack/spack/test/hooks/generate_spec_scripts.py:104:100
    |
102 |         )
103 |         assert re.search(
104 |             f"_spack_env_remove_value {uenv.spack_loaded_hashes_var} {pkg.dag_hash()} {os.pathsep}",
    |                                                                                                    ^
105 |             unload_script,
106 |         )
    |
  ruff-check found errors
==> Running mypy checks
Success: no issues found in 639 source files
  mypy checks were clean

Keep in mind that I cannot fix your flake8 or mypy errors, so if you have any you'll need to fix them and update the pull request. If I was able to push to your branch, if you make further changes you will need to pull from your updated branch before pushing again.

I wasn't able to make any further changes, but please see the message above for remaining issues you can fix locally!

Signed-off-by: Kayla Butler <butler59@llnl.gov>

RikkiButler20 added feature A feature is missing in Spack commands shell-support labels Nov 23, 2024

RikkiButler20 self-assigned this Nov 23, 2024

spackbot-app Bot added core PR affects Spack core functionality tests General test capability(ies) labels Nov 23, 2024

adamjstewart added this to the v0.24 milestone Nov 23, 2024

becker33 modified the milestones: v1.0.0, v1.0.0 stretch goals Nov 25, 2024

tgamblin modified the milestones: v1.0.0 stretch goals, v1.0.0 Nov 26, 2024

aweits reviewed Dec 2, 2024

View reviewed changes

Comment thread lib/spack/spack/cmd/load.py Outdated

spackbot-app Bot added the utilities label Dec 18, 2024

spackbot-app Bot added the documentation Improvements or additions to documentation label Jan 3, 2025

tldahlgren reviewed Jan 14, 2025

View reviewed changes

Comment thread lib/spack/spack/hooks/cache_shell_script.py Outdated

tldahlgren reviewed Jan 14, 2025

View reviewed changes

Comment thread lib/spack/spack/test/hooks/cache_shell_script.py Outdated

RikkiButler20 marked this pull request as draft January 16, 2025 20:26

RikkiButler20 force-pushed the feature/cache-spack-load-and-activate branch from 4a2d1ce to a8b77fc Compare June 4, 2026 22:07

RikkiButler20 added 5 commits June 4, 2026 17:06

Update shell.py

3bfc5fb

Remove dead arguments hace windows shells use _spack_env_set & unset Signed-off-by: Kayla Butler <butler59@llnl.gov>

Fix style

3e6ff55

Signed-off-by: Kayla Butler <butler59@llnl.gov>

Test env scripts regenerated with lockfile

473341d

Signed-off-by: Kayla Butler <butler59@llnl.gov>

Add test_manpath_trailing_colon

3924b60

Fix style Regenerate deactivation script when lockfile is updated Signed-off-by: Kayla Butler <butler59@llnl.gov>

Fix style

3394a63

Signed-off-by: Kayla Butler <butler59@llnl.gov>

[@spackbot] updating style on behalf of RikkiButler20

9c05f82

RikkiButler20 added 7 commits June 4, 2026 21:25

Get view name from environ var

7491db2

Signed-off-by: Kayla Butler <butler59@llnl.gov>

Fix style

6ec046c

Signed-off-by: Kayla Butler <butler59@llnl.gov>

Fix parametrize

064cba9

Signed-off-by: Kayla Butler <butler59@llnl.gov>

Use shell for _get_load_cmds

f8d9325

Signed-off-by: Kayla Butler <butler59@llnl.gov>

Fix extension for test_paths_to_shell_cached

652206a

Signed-off-by: Kayla Butler <butler59@llnl.gov>

Remove sep from load script test

96627ce

The separator is different for windows scripts Signed-off-by: Kayla Butler <butler59@llnl.gov>

Us correct separator

5730c29

Signed-off-by: Kayla Butler <butler59@llnl.gov>

[@spackbot] updating style on behalf of RikkiButler20

ab496c2

RikkiButler20 added 3 commits June 8, 2026 21:01

Fix long line

256172b

Signed-off-by: Kayla Butler <butler59@llnl.gov>

commands update-completion

9e64d57

Signed-off-by: Kayla Butler <butler59@llnl.gov>

Simplify asserts

61b45f4

Signed-off-by: Kayla Butler <butler59@llnl.gov>

RikkiButler20 force-pushed the feature/cache-spack-load-and-activate branch from c1c1829 to 61b45f4 Compare June 10, 2026 19:13

RikkiButler20 added 2 commits June 10, 2026 15:44

stdout load & unload cmds

d68d37e

Signed-off-by: Kayla Butler <butler59@llnl.gov>

Use the right syntax to source scripts

eede5ae

Signed-off-by: Kayla Butler <butler59@llnl.gov>

Conversation

RikkiButler20 commented Nov 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adamjstewart commented Nov 23, 2024

Uh oh!

aweits commented Nov 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tgamblin commented Nov 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

RikkiButler20 commented Nov 27, 2024

Uh oh!

spackbot-app Bot commented Nov 27, 2024

Uh oh!

spackbot-app Bot commented Nov 27, 2024

Uh oh!

Uh oh!

RikkiButler20 commented Dec 5, 2024

Uh oh!

spackbot-app Bot commented Dec 5, 2024

Uh oh!

spackbot-app Bot commented Dec 5, 2024

Uh oh!

Uh oh!

Uh oh!

adamjstewart commented Jan 15, 2025

Uh oh!

RikkiButler20 commented Jan 16, 2025

Uh oh!

spackbot-app Bot commented Jan 16, 2025

Uh oh!

spackbot-app Bot commented Jan 16, 2025

Uh oh!

RikkiButler20 commented Jun 5, 2026

Uh oh!

spackbot-app Bot commented Jun 5, 2026

Uh oh!

spackbot-app Bot commented Jun 5, 2026

Uh oh!

RikkiButler20 commented Jun 9, 2026

Uh oh!

spackbot-app Bot commented Jun 9, 2026

Uh oh!

spackbot-app Bot commented Jun 9, 2026

Uh oh!

RikkiButler20 commented Jun 9, 2026

Uh oh!

spackbot-app Bot commented Jun 9, 2026

Uh oh!

spackbot-app Bot commented Jun 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

RikkiButler20 commented Nov 23, 2024 •

edited

Loading

aweits commented Nov 24, 2024 •

edited

Loading

tgamblin commented Nov 24, 2024 •

edited

Loading