fix(grep): build upstream GNU grep

docs-internal/registry-parity-worklist.md

@@ -1,6 +1,6 @@
 # Registry Linux-Parity Worklist
 
-Status: worklist · Owner: registry · Last updated: 2026-07-07
+Status: worklist · Owner: registry · Last updated: 2026-07-08
 
 ## Goal (hand this to the driver agent)
 
@@ -85,7 +85,6 @@ actual backing:
 | Command(s) | Backing |
 |---|---|
 | coreutils (`sh`+80) | **uutils** (`uucore`) — established Rust project |
-| ripgrep (`rg`) | real ripgrep |
 | duckdb, vim | real upstream C source, patched for WASI |
 | sqlite3 **engine** | real SQLite amalgamation (⚠️ but the *CLI* is ours — see below) |
 | jq | **jaq** (`jaq-core/std/json`) — established Rust jq |
@@ -107,7 +106,8 @@ actual backing:
 | **fd** | DONE | our `secureexec-fd` on raw `regex` (not sharkdp/fd) | real **fd** (sharkdp) |
 | **findutils** (`find`,`xargs`) | TODO | our hand-rolled on `regex`/shims | real GNU findutils, or `uutils/findutils` |
 | **tree** | DONE | our hand-rolled, zero deps | real `tree`, or an established one |
-| **grep** | TODO | our `secureexec-grep` on raw `regex` (**not** an established grep pkg) | **real GNU grep**, or a popular established grep (ripgrep's `grep` crates) |
+| **grep** | DONE | our `secureexec-grep` on raw `regex` (**not** an established grep pkg) | real **GNU grep** |
+| **ripgrep** (`rg`) | TODO | our `secureexec-grep` recursive search shim, not real ripgrep | real upstream **ripgrep** |
 | **zip** | DONE | our 203-line `zip.c` over zlib/minizip (not Info-ZIP) | real Info-ZIP, or an established lib's CLI |
 | **unzip** | DONE | our 669-line `unzip.c` over zlib/minizip | real Info-ZIP unzip |
 | **sqlite3 CLI** | DONE | our 558-line `sqlite3_cli.c` (engine is real SQLite; the shell is ours) | real SQLite `shell.c` (its official CLI) |
@@ -245,18 +245,39 @@ works (`wasi-spawn` broker), so `xargs` is not a blocker.
   including `fd --version` reporting `fd 10.4.2`, in
   `2026-07-08T06-25-00-0700-fd-vitest-upstream-after-dir-format.log`.
   Rev: `mrskpomv` — `fix(fd): build upstream fd-find`.
-- **grep — moderate.** Decision is real GNU grep (gnulib cascade → sysroot stubs)
-  or ripgrep's `grep-*` crates (threads → serial patch). NB the current
-  `secureexec-grep` also backs `rg`, so the shipped "ripgrep" is custom too.
+- **grep — DONE.** Replaced the `@agentos-software/grep` package's custom
+  `secureexec-grep` command wrapper with upstream GNU grep 3.12 from the official
+  GNU release tarball. The real GNU `grep` binary builds through the C toolchain;
+  `egrep` and `fgrep` are separate tiny WASM launchers that preserve GNU's
+  upstream obsolescent scripts by spawning `grep -E` / `grep -F` through the
+  AgentOS process broker. Sysroot fix stayed one layer down: wasi-libc no longer
+  advertises/exports its nonstandard two-argument `opendirat`, which conflicted
+  with gnulib's helper. Proof: official GNU release listing captured in
+  `2026-07-08T06-29-11-0700-grep-gnu-ftp-latest.log`; configure options captured
+  in `2026-07-08T06-29-39-0700-grep-upstream-configure-help-probe.log`;
+  sysroot `opendirat` patch check and rebuild pass in
+  `2026-07-08T06-34-27-0700-grep-wasi-libc-opendirat-patch-check-definition.log`
+  and `2026-07-08T06-34-33-0700-grep-sysroot-rebuild-opendirat-definition.log`;
+  GNU grep build passes in
+  `2026-07-08T06-35-01-0700-grep-gnu-wasm-build-after-opendirat-symbol.log`;
+  `egrep`/`fgrep` launcher builds pass in
+  `2026-07-08T06-40-07-0700-grep-egrep-wrapper-build.log` and
+  `2026-07-08T06-40-42-0700-grep-fgrep-wrapper-build.log`; package build and
+  check-types pass in `2026-07-08T06-42-24-0700-grep-package-build-final.log`
+  and `2026-07-08T06-42-18-0700-grep-check-types-final.log`; e2e grep tests
+  pass 8/8 in `2026-07-08T06-43-59-0700-grep-vitest-after-wrapper-cache-repair.log`.
+  Rev: `uyukolvr` — `fix(grep): build upstream GNU grep`.
+  NB: `secureexec-grep` remains only because `@agentos-software/ripgrep` still
+  uses it for `rg`; replace `rg` with real ripgrep in a separate rev.
 - **zip / unzip — moderate.** Real **Info-ZIP** source (fetch+pin like zlib/sqlite);
   zlib is already vendored. Filesystem + `isatty`/`utime`/`chmod`/perms stubs; no
   sockets/threads/spawn. Friction is Info-ZIP's crufty build, not syscalls.
 - **findutils — moderate→hard.** `find` is fs+regex (easy); `xargs` spawn already
   works. **uutils/findutils** (Rust) avoids gnulib; **GNU findutils** (C) hits the
   same gnulib cascade as wget. Prefer uutils unless GNU parity is required.
 
-Ranked easiest→hardest: **sqlite3-CLI · http-get (drop) · tree · fd · grep · zip ·
-unzip · findutils.**
+Ranked easiest→hardest: **sqlite3-CLI · http-get (drop) · tree · fd · grep ·
+ripgrep · zip · unzip · findutils.**
 
 ## Status tracking (how the driver reports progress in this doc)
 

software/grep/agentos-package.json

@@ -1,11 +1,9 @@
 {
 	"commands": [
-		"grep"
+		"grep",
+		"egrep",
+		"fgrep"
 	],
-	"aliases": {
-		"egrep": "grep",
-		"fgrep": "grep"
-	},
 	"registry": {
 		"title": "grep",
 		"description": "GNU grep pattern matching (grep, egrep, fgrep).",

software/grep/native/crates/cmd-egrep/Cargo.toml

@@ -0,0 +1,14 @@
+[package]
+workspace = "../../../../../toolchain"
+name = "cmd-egrep"
+version.workspace = true
+edition.workspace = true
+license.workspace = true
+description = "egrep launcher for GNU grep"
+
+[[bin]]
+name = "egrep"
+path = "src/main.rs"
+
+[dependencies]
+shims = { package = "secureexec-shims", path = "../../../../../toolchain/crates/libs/shims" }

software/grep/native/crates/cmd-egrep/src/main.rs

@@ -0,0 +1,4 @@
+fn main() {
+    let args: Vec<std::ffi::OsString> = std::env::args_os().collect();
+    std::process::exit(shims::grep_alias::egrep(args));
+}

software/grep/native/crates/cmd-fgrep/Cargo.toml

@@ -0,0 +1,14 @@
+[package]
+workspace = "../../../../../toolchain"
+name = "cmd-fgrep"
+version.workspace = true
+edition.workspace = true
+license.workspace = true
+description = "fgrep launcher for GNU grep"
+
+[[bin]]
+name = "fgrep"
+path = "src/main.rs"
+
+[dependencies]
+shims = { package = "secureexec-shims", path = "../../../../../toolchain/crates/libs/shims" }

software/grep/native/crates/cmd-fgrep/src/main.rs

@@ -0,0 +1,4 @@
+fn main() {
+    let args: Vec<std::ffi::OsString> = std::env::args_os().collect();
+    std::process::exit(shims::grep_alias::fgrep(args));
+}

software/grep/test/grep.test.ts

@@ -0,0 +1,113 @@
+import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises";
+import { tmpdir } from "node:os";
+import { dirname, join } from "node:path";
+import { createWasmVmRuntime } from "@agentos/test-harness";
+import {
+	COMMANDS_DIR,
+	NodeFileSystem,
+	createKernel,
+	describeIf,
+	hasWasmBinaries,
+} from "@agentos/test-harness";
+import type { Kernel } from "@agentos/test-harness";
+import { afterEach, describe, expect, it } from "vitest";
+
+let tempRoot: string | undefined;
+
+async function createTestVFS(): Promise<NodeFileSystem> {
+	tempRoot = await mkdtemp(join(tmpdir(), "agentos-grep-"));
+	await writeFixture(
+		"/project/notes.txt",
+		["Alpha", "beta", "alphabet", "delta"].join("\n") + "\n",
+	);
+	await writeFixture(
+		"/project/other.txt",
+		["gamma", "Beta blocker", "literal a+b"].join("\n") + "\n",
+	);
+	return new NodeFileSystem({ root: tempRoot });
+}
+
+async function writeFixture(path: string, contents: string): Promise<void> {
+	if (!tempRoot) throw new Error("fixture root not initialized");
+	const hostPath = join(tempRoot, path.replace(/^\/+/, ""));
+	await mkdir(dirname(hostPath), { recursive: true });
+	await writeFile(hostPath, contents);
+}
+
+describeIf(hasWasmBinaries, "GNU grep command", { timeout: 10_000 }, () => {
+	let kernel: Kernel;
+
+	afterEach(async () => {
+		await kernel?.dispose();
+		if (tempRoot) {
+			await rm(tempRoot, { recursive: true, force: true });
+			tempRoot = undefined;
+		}
+	});
+
+	async function mountFixture(): Promise<NodeFileSystem> {
+		const vfs = await createTestVFS();
+		kernel = createKernel({ filesystem: vfs });
+		await kernel.mount(createWasmVmRuntime({ commandDirs: [COMMANDS_DIR] }));
+		return vfs;
+	}
+
+	it("reports the upstream GNU grep version", async () => {
+		await mountFixture();
+
+		const result = await kernel.exec("grep --version", {});
+		expect(result.stdout).toContain("GNU grep 3.12");
+	});
+
+	it("prints matching lines from a file", async () => {
+		await mountFixture();
+
+		const result = await kernel.exec("grep alpha /project/notes.txt", {});
+		expect(result.stdout).toBe("alphabet\n");
+	});
+
+	it("supports case-insensitive search", async () => {
+		await mountFixture();
+
+		const result = await kernel.exec("grep -i beta /project/notes.txt", {});
+		expect(result.stdout).toBe("beta\n");
+	});
+
+	it("supports inverted matches", async () => {
+		await mountFixture();
+
+		const result = await kernel.exec("grep -v Alpha /project/notes.txt", {});
+		expect(result.stdout).toBe("beta\nalphabet\ndelta\n");
+	});
+
+	it("counts matches", async () => {
+		await mountFixture();
+
+		const result = await kernel.exec("grep -c a /project/notes.txt", {});
+		expect(result.stdout).toBe("4\n");
+	});
+
+	it("prints file names with matches", async () => {
+		await mountFixture();
+
+		const result = await kernel.exec(
+			"grep -l gamma /project/notes.txt /project/other.txt",
+			{},
+		);
+		expect(result.stdout).toBe("/project/other.txt\n");
+	});
+
+	it("supports egrep extended regex alias", async () => {
+		await mountFixture();
+
+		const result = await kernel.exec("egrep 'Alpha|delta' /project/notes.txt", {});
+		expect(result.stdout).toBe("Alpha\ndelta\n");
+	});
+
+	it("supports fgrep fixed-string alias", async () => {
+		await mountFixture();
+
+		const result = await kernel.exec("fgrep 'a+b' /project/other.txt", {});
+		expect(result.stdout).toBe("literal a+b\n");
+	});
+});

toolchain/Makefile

@@ -24,7 +24,6 @@ COMMAND_PACKAGES := $(addprefix -p cmd-,$(COMMAND_NAMES))
 
 # Alias symlinks: link_name:target_binary
 ALIAS_SYMLINKS := \
-	egrep:grep fgrep:grep \
 	gunzip:gzip zcat:gzip \
 	bash:sh \
 	dir:ls vdir:ls \
@@ -213,7 +212,7 @@ wasm: vendor patch-vendor patch-std wasm-opt-check
 #   codex     the codex fork build    (codex, codex-exec)
 # `just toolchain-cmd <name>` calls this. `make wasm` builds the fast
 # Rust set; `make -C c programs install` builds/installs the fast C set.
-C_COMMANDS := zip unzip envsubst sqlite3 curl wget duckdb vim
+C_COMMANDS := zip unzip envsubst sqlite3 curl wget grep duckdb vim
 
 .PHONY: cmd/%
 cmd/%:

toolchain/c/Makefile

@@ -68,10 +68,10 @@ COMMANDS_DIR ?= ../target/wasm32-wasip1/release/commands
 
 # Fast real commands installed by the default software gate. Slow/heavy commands
 # (duckdb, vim) remain available through explicit parent `make cmd/<name>`.
-COMMANDS := zip unzip envsubst sqlite3 curl wget tree
+COMMANDS := zip unzip envsubst sqlite3 curl wget grep tree
 
 # Programs requiring patched sysroot (Tier 2+ custom host imports)
-PATCHED_PROGRAMS := http_get_test isatty_test getpid_test getppid_test getppid_verify userinfo pipe_test dup_test spawn_child spawn_exit_code pipeline kill_child waitpid_return waitpid_edge syscall_coverage getpwuid_test signal_tests sigaction_self sigaction_behavior delayed_tcp_echo delayed_kill pipe_edge tcp_accept_spawn tcp_echo tcp_server http_server udp_echo unix_socket signal_handler dns_lookup sqlite3 curl wget tree fs_probe
+PATCHED_PROGRAMS := http_get_test isatty_test getpid_test getppid_test getppid_verify userinfo pipe_test dup_test spawn_child spawn_exit_code pipeline kill_child waitpid_return waitpid_edge syscall_coverage getpwuid_test signal_tests sigaction_self sigaction_behavior delayed_tcp_echo delayed_kill pipe_edge tcp_accept_spawn tcp_echo tcp_server http_server udp_echo unix_socket signal_handler dns_lookup sqlite3 curl wget grep tree fs_probe
 
 # Discover all package command and test-program C source files.
 ALL_SOURCES := $(foreach dir,$(C_SOURCE_DIRS),$(wildcard $(dir)/*.c))
@@ -89,7 +89,7 @@ endif
 ifeq ($(wildcard $(PATCHED_SYSROOT)/lib/wasm32-wasi/libc.a),)
   CUSTOM_WASM_PROG_NAMES :=
 else
-  CUSTOM_WASM_PROG_NAMES := curl wget sqlite3 tree
+  CUSTOM_WASM_PROG_NAMES := curl wget grep sqlite3 tree
 endif
 
 WASM_PROG_NAMES := $(sort $(basename $(notdir $(SOURCES))) $(CUSTOM_WASM_PROG_NAMES))
@@ -156,6 +156,9 @@ WGET_URL := https://ftp.gnu.org/gnu/wget/wget-$(WGET_VERSION).tar.gz
 WGET_UPSTREAM_BUILD_DIR := $(BUILD_DIR)/wget-upstream
 WGET_UPSTREAM_OVERLAY_INCLUDE_DIR := overlays/wget/include
 WGET_UPSTREAM_OVERLAY_FILES := $(wildcard $(WGET_UPSTREAM_OVERLAY_INCLUDE_DIR)/*.h)
+GREP_VERSION := 3.12
+GREP_URL := https://ftp.gnu.org/gnu/grep/grep-$(GREP_VERSION).tar.xz
+GREP_UPSTREAM_BUILD_DIR := $(BUILD_DIR)/grep-upstream
 TREE_VERSION := 2.3.2
 TREE_URL := https://gitlab.com/OldManProgrammer/unix-tree/-/archive/$(TREE_VERSION)/unix-tree-$(TREE_VERSION).tar.gz
 MINIZIP_URL := https://github.com/madler/zlib/archive/refs/tags/v1.3.1.zip
@@ -621,6 +624,18 @@ $(BUILD_DIR)/wget: scripts/build-wget-upstream.sh $(WGET_UPSTREAM_OVERLAY_FILES)
 		--ranlib "$(abspath $(WASI_SDK_DIR)/bin/llvm-ranlib)" \
 		--output "$(abspath $@)"
 
+$(BUILD_DIR)/grep: scripts/build-grep-upstream.sh $(PATCHED_SYSROOT)/lib/wasm32-wasi/libc.a $(WASI_SDK_DIR)/bin/clang
+	@mkdir -p $(BUILD_DIR)
+	bash scripts/build-grep-upstream.sh \
+		--version "$(GREP_VERSION)" \
+		--url "$(GREP_URL)" \
+		--cache-dir "$(abspath $(LIBS_CACHE))" \
+		--build-dir "$(abspath $(GREP_UPSTREAM_BUILD_DIR))" \
+		--cc "$(abspath $(CC)) --target=wasm32-wasip1 --sysroot=$(abspath $(SYSROOT))" \
+		--ar "$(abspath $(WASI_SDK_DIR)/bin/llvm-ar)" \
+		--ranlib "$(abspath $(WASI_SDK_DIR)/bin/llvm-ranlib)" \
+		--output "$(abspath $@)"
+
 # duckdb: upstream DuckDB CLI built from source with our patched WASI/POSIX sysroot
 $(BUILD_DIR)/duckdb: libs/duckdb/CMakeLists.txt $(PATCHED_SYSROOT)/lib/wasm32-wasi/libc.a $(WASI_SDK_DIR)/bin/clang $(WASI_SDK_DIR)/bin/clang++ cmake/FindThreads.cmake scripts/build-duckdb.sh include/fcntl.h include/ifaddrs.h include/net/if.h include/sys/ioctl.h
 	@mkdir -p $(BUILD_DIR)