Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 10 additions & 2 deletions e2e/nodejs_host/BUILD.bazel
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
load("@bazel_features//:features.bzl", "bazel_features")
load("@rules_shell//shell:sh_test.bzl", "sh_test")

# Dummy test target that imports various targets from the
Expand All @@ -23,7 +24,9 @@ load("@rules_shell//shell:sh_test.bzl", "sh_test")
"node16",
"node16_nvmrc",
"node17_custom",
]
] + (
["node26_facts"] if bazel_features.external_deps.has_facts else []

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't we still run the tests, it just won't be reproducible?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No. With the current PR, we'll hit this line eventually:

fail("No nodejs is available for {} at version {}".format(host_os, node_version) +
"\n Consider upgrading by setting node_version in a call to node_repositories in WORKSPACE." +
"\n Note that Node 16.x is the minimum published for Apple Silicon (M1 Macs), and 20.x is the minimum for Windows ARM64.")

(because we pass empty node_repositories).

What we can do, is to call fetch_node_repositories in the extension even if facts are not supported.
Then, if we fetched something, we can return reproducible = False. This will snapshot the repository rule invocation parameters in the lockfile (at least with sufficiently new bazel versions when I checked this last). I will take a stab at this (I'll open a separate PR).

What I would advise against is to make the repository rule itself not reproducible (especially w/o explicit opt in). IMHO this would be a step back.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alternative: #3919

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What we can do, is to call fetch_node_repositories in the extension even if facts are not supported.
Then, if we fetched something, we can return reproducible = False. This will snapshot the repository rule invocation parameters in the lockfile (at least with sufficiently new bazel versions when I checked this last). I will take a stab at this (I'll open a separate PR).

This sounds ideal to me to me I think, IIUC.

What I would advise against is to make the repository rule itself not reproducible (especially w/o explicit opt in). IMHO this would be a step back.

That would only be a step back when (1) there checksums are not in rules_nodejs so your new feature is used, but (2) there is no lockfile (it's not checked in) or the bazel version is too old to persist facts. So it would only be a step back when your new feature is used with old bazel versions and no lockfile? That doesn't sound horrible to me, and encourages the use of lockfiles...

We could potentially output a warning or even fail() if we had to fetch node to determine checksums but there is no facts API? That would only be for old bazel versions basically though...

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This sounds ideal to me to me I think, IIUC.

That's what #3919 does. So let's focus on that one.

Regarding the point about the repository rule not being reproducible:

The execution of repository rules does not get snapshotted to the lockfile (this is in contrast to module extensions). So even when running on an up-to-date bazel version with a checked-in lockfile, a non-reproducible repository rule is kind of bad.

Just to be clear: Another way of saying what I'm trying to say is we should not call fetch_node_repositories in a repository rule. We should always call it in the module extension and then call nodejs_register_toolchains with the resolved digests (both this PR and #3919 do that).

I have tested with older bazel versions down to 7.1.0 and they all snapshot the repository rule invocations and their parameters to the lockfile (like this). So we'll only end up with something non-reproducible if the lockfile is not checked in (which always was the case).

)
]

[
Expand Down Expand Up @@ -66,5 +69,10 @@ load("@rules_shell//shell:sh_test.bzl", "sh_test")
("node16_nvmrc", "npx"),
("node17_custom", "npm"),
("node17_custom", "npx"),
]
] + (
[
("node26_facts", "npm"),
("node26_facts", "npx"),
] if bazel_features.external_deps.has_facts else []
)
]
12 changes: 12 additions & 0 deletions e2e/nodejs_host/MODULE.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ local_path_override(

bazel_dep(name = "bazel_lib", version = "3.0.0-beta.1")
bazel_dep(name = "bazel_skylib", version = "1.7.1")
bazel_dep(name = "bazel_features", version = "1.49.0")
Comment thread
gzm0 marked this conversation as resolved.
bazel_dep(name = "rules_shell", version = "0.6.1")

node = use_extension("@rules_nodejs//nodejs:extensions.bzl", "node")
Expand Down Expand Up @@ -35,6 +36,10 @@ node.toolchain(
node_urls = ["https://nodejs.org/dist/v17.0.1/{filename}"],
node_version = "17.0.1.custom",
)
node.toolchain(
name = "node26_facts",
node_version = "26.3.0",
)

# FIXME(6.0): a repo rule with name=foo should create a repo named @foo, not @foo_toolchains
use_repo(
Expand All @@ -60,6 +65,13 @@ use_repo(
"node17_custom_linux_arm64",
"node17_custom_toolchains",
"node17_custom_windows_amd64",
"node26_facts",
"node26_facts_darwin_amd64",
"node26_facts_darwin_arm64",
"node26_facts_linux_amd64",
"node26_facts_linux_arm64",
"node26_facts_toolchains",
"node26_facts_windows_amd64",
"nodejs",
"nodejs_darwin_amd64",
"nodejs_darwin_arm64",
Expand Down
9 changes: 8 additions & 1 deletion nodejs/BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,13 @@ package(default_visibility = ["//visibility:public"])
bzl_library(
name = "extensions",
srcs = ["extensions.bzl"],
deps = [":repositories"],
deps = [
":repositories",
"//nodejs/private:fetch_node_repositories",
"//nodejs/private:node_versions",
"//nodejs/private:version_from_attr",
"@bazel_features//:features",
],
)

bzl_library(
Expand All @@ -29,6 +35,7 @@ bzl_library(
"//nodejs/private:nodejs_repo_host_os_alias",
"//nodejs/private:nodejs_toolchains_repo",
"//nodejs/private:os_name",
"//nodejs/private:version_from_attr",
],
)

Expand Down
38 changes: 35 additions & 3 deletions nodejs/extensions.bzl
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,10 @@ use_repo(node, "nodejs_toolchains")
```
"""

load("@bazel_features//:features.bzl", "bazel_features")
load("//nodejs/private:fetch_node_repositories.bzl", "fetch_node_repositories")
load("//nodejs/private:node_versions.bzl", "NODE_VERSIONS")
load("//nodejs/private:version_from_attr.bzl", "version_from_attr")
load(
":repositories.bzl",
"DEFAULT_NODE_REPOSITORY",
Expand Down Expand Up @@ -52,17 +56,45 @@ def _toolchain_extension(module_ctx):
else:
registrations[toolchain.name] = toolchain

supports_facts = hasattr(module_ctx, "facts")
new_repository_facts = {}

for k, v in registrations.items():
node_version = version_from_attr(module_ctx, v)
node_repositories = v.node_repositories or NODE_VERSIONS.get(node_version, {})

if supports_facts and not node_repositories:
node_repositories = (
new_repository_facts.get(node_version) or
module_ctx.facts.get(node_version) or
# TODO: Add support for node_urls?
fetch_node_repositories(module_ctx, node_version)
)
new_repository_facts[node_version] = node_repositories

nodejs_register_toolchains(
name = k,
node_version = v.node_version,
node_version_from_nvmrc = v.node_version_from_nvmrc,
node_version = node_version,
node_urls = v.node_urls,
node_repositories = v.node_repositories,
node_repositories = node_repositories,
include_headers = v.include_headers,
register = False,
)

if not hasattr(module_ctx, "extension_metadata"):
return # buildifier: disable=return-value (allow no value)

if not bazel_features.external_deps.extension_metadata_has_reproducible:
return module_ctx.extension_metadata()

if not supports_facts:
return module_ctx.extension_metadata(reproducible = True)

return module_ctx.extension_metadata(
reproducible = True,
facts = new_repository_facts,
)

_ATTRS = {
"name": attr.string(
doc = "Base name for generated repositories",
Expand Down
10 changes: 10 additions & 0 deletions nodejs/private/BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -41,3 +41,13 @@ bzl_library(
srcs = ["user_build_settings.bzl"],
deps = ["//nodejs/private/providers:user_build_settings"],
)

bzl_library(
name = "version_from_attr",
srcs = ["version_from_attr.bzl"],
)

bzl_library(
name = "fetch_node_repositories",
srcs = ["fetch_node_repositories.bzl"],
)
58 changes: 58 additions & 0 deletions nodejs/private/fetch_node_repositories.bzl
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
"""Implementation of node SHASUM fetching for facts."""

_REPOSITORY_TYPES = {
"darwin-arm64.tar.gz": "darwin_arm64",
"darwin-x64.tar.gz": "darwin_amd64",
"linux-x64.tar.xz": "linux_amd64",
"linux-arm64.tar.xz": "linux_arm64",
"linux-s390x.tar.xz": "linux_s390x",
"win-x64.zip": "windows_amd64",
"win-arm64.zip": "windows_arm64",
"linux-ppc64le.tar.xz": "linux_ppc64le",
}

def fetch_node_repositories(module_ctx, version):
"""Fetches node repositories for the given node version.

Port of scripts/update-nodejs-versions.js

Args:
module_ctx: Module context
version: The node version to fetch repositories for.

Returns:
A dictionary in the shape of node_repositories.
"""

shasums_filename = "{version}-SHASUMS256.txt".format(version = version)
url = "https://nodejs.org/dist/v{version}/SHASUMS256.txt".format(version = version)

result = module_ctx.download(url = url, output = shasums_filename)
if not result.success:
fail("Failed to fetch node shasums:", url, result, sep = "\n")

shasums = module_ctx.read(shasums_filename)

result = {}

for line in shasums.splitlines():
line = line.strip()
if not line:
continue

parts = line.split(" ")
if len(parts) != 2:
fail("{url} contains unexpected line:\n{line}".format(
url = url,
line = line,
))

sha, filename = parts
type = _REPOSITORY_TYPES.get(filename.removeprefix("node-v%s-" % version))
if not type:
continue

strip_prefix = filename.removesuffix(".tar.gz").removesuffix(".tar.xz").removesuffix(".zip")
result[version + "-" + type] = (filename, strip_prefix, sha)

return result
28 changes: 28 additions & 0 deletions nodejs/private/version_from_attr.bzl
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
"""Helper to get node version from user config."""

def _verify_version_is_valid(version):
major, minor, patch = (version.split(".") + [None, None, None])[:3]
if not major.isdigit() or not minor.isdigit() or not patch.isdigit():
fail("Invalid node version: %s" % version)

def version_from_attr(ctx, attr):
"""Extract the node version from attr.

Verifies if the extracted version is valid.

Args:
ctx: repository or module context
attr: A struct with fields node_version and node_version_from_nvmrc

Returns:
The node version.
"""

node_version = attr.node_version

if attr.node_version_from_nvmrc:
node_version = str(ctx.read(attr.node_version_from_nvmrc)).strip()

_verify_version_is_valid(node_version)

return node_version
13 changes: 2 additions & 11 deletions nodejs/repositories.bzl
Comment thread
gzm0 marked this conversation as resolved.
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
load("//nodejs/private:node_versions.bzl", "NODE_VERSIONS")
load("//nodejs/private:nodejs_repo_host_os_alias.bzl", "nodejs_repo_host_os_alias")
load("//nodejs/private:nodejs_toolchains_repo.bzl", "PLATFORMS", "nodejs_toolchains_repo")
load("//nodejs/private:version_from_attr.bzl", "version_from_attr")

# Default base name for node toolchain repositories
# created by the module extension
Expand Down Expand Up @@ -66,12 +67,7 @@ def _download_node(repository_ctx):
# @nodejs_PLATFORM where PLATFORM is one of BUILT_IN_NODE_PLATFORMS
host_os = repository_ctx.attr.platform or repository_ctx.name.split("nodejs_", 1)[1]

node_version = repository_ctx.attr.node_version

if repository_ctx.attr.node_version_from_nvmrc:
node_version = str(repository_ctx.read(repository_ctx.attr.node_version_from_nvmrc)).strip()

_verify_version_is_valid(node_version)
node_version = version_from_attr(repository_ctx, repository_ctx.attr)

node_repositories = repository_ctx.attr.node_repositories

Expand Down Expand Up @@ -302,11 +298,6 @@ def _strip_bin(path):

return path[len("bin/"):]

def _verify_version_is_valid(version):
major, minor, patch = (version.split(".") + [None, None, None])[:3]
if not major.isdigit() or not minor.isdigit() or not patch.isdigit():
fail("Invalid node version: %s" % version)

def _nodejs_repositories_impl(repository_ctx):
reproducible = _download_node(repository_ctx)
_prepare_node(repository_ctx)
Expand Down
Loading