feat: pre-populate devcontainer cache with local OCI Layout bundles

[hbraswelrh]

Summary

Add optional private policy support to the devcontainer by
extending the mock OCI registry to serve mounted Gemara YAML
files. This enables complyctl get, complyctl generate, and
complyctl scan to work with private policy bundles through the
standard workflow — no cache bypass or registry access required.

Problem

Private policy bundles cannot be committed to GitHub or embedded
in the mock OCI registry testdata without exposing their content.
Users evaluating private compliance policies in the devcontainer
need the full get -> generate -> scan pipeline to work with
local bundles that never touch a registry or the repository.

Solution

The mock OCI registry gains a seedFromDirectory() method that
discovers Gemara policy files from a mounted directory and serves
them as OCI artifacts alongside the embedded testdata. The
post-create script adds policy entries to complytime.yaml
pointing at the mock registry, so complyctl get populates the
cache through normal code paths.

User-facing input format

Users mount raw Gemara YAML files (not OCI Layout bundles):

# Mount policies into the devcontainer
/bundles/
└── my-private-policy/
    ├── catalog.yaml
    └── policy.yaml

# After container starts, standard workflow works for all policies
cd ~/test-workspace
complyctl get
complyctl generate --policy-id my-private-policy
complyctl scan --policy-id my-private-policy

How it works

1. Mock registry (seedFromDirectory()): At startup, reads
   MOCK_REGISTRY_CONTENT_DIR (default /bundles/), discovers
   subdirectories containing catalog.yaml + policy.yaml,
   registers them via the existing addArtifact() machinery.
2. Post-create script (Step 4b): Validates bundle directories,
   appends policy entries to complytime.yaml pointing at
   localhost:8765/policies/{name}.
3. Registry startup (Step 5): Passes MOCK_REGISTRY_CONTENT_DIR
   to the mock registry process.

Existing workflow preserved

The mock registry workflow is completely unaffected. Both embedded
testdata and mounted policies are served by the same registry:

Command	Embedded (test-ampel-bp)	Mounted (my-private-policy)
complyctl get	Works	Works
complyctl generate	Works	Works
complyctl scan	Works	Works

Security hardening

• Name validation: Directory names validated against
  ^[a-zA-Z0-9_-]+$ in both Go and shell (consistent)
• Symlink rejection: entry.Type() skips symlinked
  directories; os.Lstat rejects symlinked files
• Resource exhaustion: readFileLimited() caps file reads
  at 10 MB to prevent OOM from oversized or adversarial files
• No path traversal: os.ReadDir returns base names only;
  paths constructed via filepath.Join with hardcoded filenames
• Trust model documented: Code comments explain the
  operator-controlled trust assumption
• gosec clean: All G703, G706, G306 findings addressed with
  validations or //nolint with rationale

Files Changed

File	Change
cmd/mock-oci-registry/main.go	+80: seedFromDirectory(), readFileLimited(), validBundleName regex, maxPolicyFileSize constant, MOCK_REGISTRY_CONTENT_DIR env var
cmd/mock-oci-registry/main_test.go	+263: 12 tests covering happy path, edge cases, and security hardening (symlink, oversize, invalid names)
.devcontainer/scripts/post-create.sh	Replaced 125-line cache bypass (jq/awk/cp) with 15-line registry-based config insertion
docs/TESTING_ENVIRONMENT.md	Updated Private Bundles section: raw YAML input format, standard get workflow, seedFromDirectory() explanation

Comparison to previous approach

Aspect	Previous (cache bypass)	Current (registry serving)
Cache format coupling	Shell must match state.json schema + dir layout	None — complyctl get owns it
complyctl get	Fails for local bundles	Works for all policies
Digest management	Shell copies blobs, jq writes digest to state	addArtifact() handles everything
Maintenance	Script breaks silently if cache internals change	Registry + get flow stays in sync
Testing	Manual validation only	12 unit tests (including security)
Dependencies	Requires jq in container	None new — reuses existing Go code
Code size	~125 lines of shell	~80 lines of Go + ~15 lines of shell
User mental model	"skip get, use generate directly"	Standard workflow for everything

Related Issues

• Closes feat: pre-populate devcontainer cache with local OCI Layout bundles #542

OpenSpec Artifacts

Spec artifacts at openspec/changes/devcontainer-bundle-cache/ —
all complete (proposal, design, specs/bundle-cache-prepopulation,
tasks).

[marcusburghardt]

Suggestion: Extend the mock registry to serve mounted policy files

The current approach works and is fine for unblocking your testing workflow -- I'm good to approve it as-is. This is a suggestion for a follow-up improvement (or you can fold it into this PR if you prefer).

The concern

The shell script in Step 4b replicates cache-internal knowledge that lives in Go code -- the state.json schema (internal/cache/state.go), the directory layout conventions (internal/cache/sync.go), and jq-based JSON manipulation to reproduce what oras.Copy produces. If any of that changes, the script breaks silently. It also creates a split workflow where complyctl get fails for pre-populated policies, requiring users to remember which policies use get and which don't.

The alternative

The mock registry already has all the machinery needed. seedPolicyFromFiles() reads raw Gemara YAML files and addArtifact() wraps them into OCI artifacts with proper manifests, digests, and tags. The only difference between the embedded testdata and user-provided policies is the data source: embed.FS vs the filesystem.

Extending this to read from a mounted directory is ~25 lines of Go:

// seedFromDirectory discovers Gemara policy files from a filesystem
// directory and registers them in the content store, exactly like
// the embedded testdata in seedDefaults().
func (s *contentStore) seedFromDirectory(dir string) {
    entries, err := os.ReadDir(dir)
    if err != nil {
        return // no directory, nothing to do
    }
    for _, entry := range entries {
        if !entry.IsDir() {
            continue
        }
        name := entry.Name()
        policyDir := filepath.Join(dir, name)

        catalog, err := os.ReadFile(filepath.Join(policyDir, "catalog.yaml"))
        if err != nil {
            log.Printf("WARNING: skipping %s: %v", name, err)
            continue
        }
        policy, err := os.ReadFile(filepath.Join(policyDir, "policy.yaml"))
        if err != nil {
            log.Printf("WARNING: skipping %s: %v", name, err)
            continue
        }

        s.addArtifact("policies/"+name, []string{"latest"}, []layerDef{
            {mediaType: gemaraCatalogType, data: catalog},
            {mediaType: gemaraPolicyType, data: policy},
        })
        log.Printf("Seeded policy from directory: policies/%s", name)
    }
}

Then in main(), after store.seedDefaults():

contentDir := os.Getenv("MOCK_REGISTRY_CONTENT_DIR")
if contentDir == "" {
    contentDir = "/bundles"
}
store.seedFromDirectory(contentDir)

What this changes for post-create.sh

The entire Step 4b (125 lines of jq/awk/cp -a cache manipulation) reduces to just appending complytime.yaml entries pointing at http://localhost:8765/policies/{name}. The registry serves the content, complyctl get populates the cache through normal code paths. Roughly ~15 lines of shell instead of 125.

User-facing input format change

This does change what users mount. Instead of pre-built OCI Layout bundles (oci-layout + index.json + blobs/sha256/), users provide raw Gemara YAML files:

/bundles/
└── my-private-policy/
    ├── catalog.yaml
    └── policy.yaml

For a devcontainer testing scenario, this is arguably simpler -- policy authors work with the source YAML files before publishing to a registry. If there are users who only have pre-built OCI Layout bundles (not the source YAML), that's worth considering.

Why this is more robust

Aspect	Current (cache bypass)	Registry serving
Cache format coupling	Shell must match state.json schema + dir layout	None -- complyctl get owns it
complyctl get	Fails for local bundles	Works for all policies
Digest management	Shell copies blobs, jq writes digest to state	addArtifact() handles everything
Maintenance	Script breaks silently if cache internals change	Registry + get flow stays in sync
Testing	Manual validation only	seedFromDirectory() is unit-testable
Dependencies	Adds jq to shell path	None new -- reuses existing Go code
Code size	~125 lines of shell	~25 lines of Go + ~15 lines of shell
User mental model	"skip get, use generate directly"	Standard workflow for everything

Note on bundle format

seedPolicyFromFiles() produces split-layer format artifacts (separate catalog + policy layers). If any private policies use the Gemara bundle format (DetectManifestShape() in internal/policy/loader.go), that would need a separate loading path. For standard Gemara catalog + policy pairs, the existing addArtifact() handles everything.

Timing

This can be a follow-up PR -- the current approach works for immediate unblocking. If done as a follow-up, it replaces the current Step 4b entirely.

[marcusburghardt]

Summary

The registry-serving approach is a strong improvement — seedFromDirectory() cleanly reuses the existing addArtifact() machinery, and the test coverage (12 tests including security edge cases) is thorough. One functional bug must be fixed: the policy URLs inserted into complytime.yaml are missing the http:// scheme prefix, which will cause complyctl get to attempt HTTPS against the plainHTTP mock registry.

Additionally, the OpenSpec artifacts (design.md, proposal.md, spec.md, tasks.md) still describe the original cache-bypass approach (commit 1) and were not updated for the registry-serving refactor (commit 2). Key discrepancies: python3 references, localhost:0 URLs, "no mock registry changes" non-goal, OCI Layout input format, state.json manipulation. These should be updated to reflect the implemented approach.

This review was generated by /review-pr (AI-assisted).

[marcusburghardt]

LGTM. Thanks @hbraswelrh

[hbraswelrh]

@jpower432 once this PR is merged the complyctl release can help with the dependent PRs:

1. PR #540 adding the post-create.sh fixes
2. Another PR that PR #540 depends on. PR 41 updates complytime-providers with a fix to the required opa_bundle_ref.

[jpower432]

LGTM