Shift left learning-path packaging checks (framing, depends, website.yaml description + trailing newline)

.cursor/commands/create-learning-path/README.md

@@ -22,8 +22,8 @@ Follow these phases in order:
 2. **Read feature docs.** Identify the canonical Grafana docs pages for the feature. Read every doc page in full from the local `website` repo first, then WebFetch.
 3. **Propose path options.** Review existing paths in `interactive-tutorials/[slug]-lj` for structural patterns. Propose 2-4 path options with milestones. Target 2-5 minutes per milestone, 6-8 milestones per path (max 10). Wait for user approval before proceeding.
 4. **Scaffold content files.** Create `content.json` for every milestone — interactive blocks for UI steps, markdown blocks for conceptual content.
-5. **Create website metadata files.** Create `website.yaml` for the path and each milestone. Refer to `docs/website-yaml-reference.md`.
-6. **Generate manifests.** Create `manifest.json` for the path (`type: "path"`, milestones array, targeting) and each milestone (`type: "guide"`, depends/recommends chain). Refer to `docs/manifest-reference.md`. Where fields can't be derived, ask the user to provide values before generating.
+5. **Create website metadata files.** Create `website.yaml` for the path and each milestone. Every file needs a non-empty `description` and must end with a trailing newline. Refer to `docs/website-yaml-reference.md`.
+6. **Generate manifests.** Create `manifest.json` for the path (`type: "path"`, milestones array, targeting) and each milestone (`type: "guide"`, depends/recommends chain). Refer to `docs/manifest-reference.md`. Put framing / concept-intro packages on disk for the website but **omit** them from path `milestones`; first hands-on uses `"depends": []`. Where fields can't be derived, ask the user to provide values before generating. Validate with `python3 .github/scripts/validate_learning_path_packages.py --path <slug>-lj` before opening the PR.
 7. **Discover selectors.** Use Playwright at `learn.grafana.net` to find stable CSS selectors for each interactive element. The user must log in through the Playwright browser window (Okta SAML).
 8. **Test in Pathfinder.** Tell the user which `content.json` to import into the Block Editor at `learn.grafana.net/?pathfinder-dev=true`. Wait for their feedback on each "Show me" / "Do it" button. Fix broken selectors based on their reports.
 9. **Verify and wrap up.** Cross-check all factual claims against live docs. Update `.github/CODEOWNERS`. Provide a summary of all files created.

.cursor/skills/review-learning-path/reference-checks.md

@@ -84,7 +84,9 @@ Also apply [review-guide-pr.mdc](../../review-guide-pr.mdc) blocking rules.
 
 Framing packages may exist in the repo for the website Learning Path but must **not** appear in path `manifest.json` `milestones`.
 
-**Common framing:** `business-value`, `advantages`, `welcome`, markdown-only intro milestones.
+**Common framing:** `business-value`, `advantages`, `welcome`, `understand-value` / `understand-baselines`, `value-*` / `value-of-*`, `how-it-works`, markdown-only intro milestones.
+
+**Not framing by prefix alone:** product teach steps such as `understand-alerts` or `understand-dashboards` stay in path `milestones`.
 
 **Flag when:**
 
@@ -94,6 +96,8 @@ Framing packages may exist in the repo for the website Learning Path but must **
 
 **OK:** Framing directories + `website.yaml` remain; `end-journey` and hands-on milestones stay in path `milestones`.
 
+**CI:** `.github/scripts/validate_learning_path_packages.py` (wired in `.github/workflows/validate-json.yml`) enforces framing-out-of-milestones, first-hands-on depends, required `website.yaml` `description`, and trailing newlines on `website.yaml`. Prefer that script over re-deriving patterns during Phase 2.
+
 ---
 
 ## Path root `content.json`
@@ -122,6 +126,8 @@ Path root `{path_dir}/website.yaml` configures the companion Learning Path on gr
 |---|---|
 | Path root file | Missing when path has framing milestone dirs (`business-value`, etc.) or website companion slug exists |
 | `menuTitle` / `description` | Missing or empty on path root `website.yaml` |
+| Step `description` | Missing or empty on any step `website.yaml` present under the package |
+| Trailing newline | Any `website.yaml` missing a final `\n` (breaks deploy-preview Hugo front matter) |
 | `journey` metadata | Missing `group`, `skill`, or layout fields peer LPs include |
 | Slug alignment | Website path slug ≠ `{path_dir}` minus `-lj` when companion exists |
 | Milestone pages | Milestone dir in path `milestones` missing `website.yaml` when peer LPs include one for same step type |

.github/scripts/validate_learning_path_packages.py

@@ -0,0 +1,326 @@
+#!/usr/bin/env python3
+"""Validate Pathfinder learning-path packaging conventions used by Learning Hub.
+
+These checks catch review findings that Pathfinder CLI validate does not cover:
+
+1. Framing packages must not appear in path manifest ``milestones``.
+2. The first hands-on milestone must not ``depends`` on a framing package
+   (use ``"depends": []``).
+3. Path-level and step-level ``website.yaml`` files require a non-empty
+   ``description`` (Learning Hub meta / listings).
+4. Every ``website.yaml`` must end with a trailing newline (deploy-preview
+   concatenates ``type: docs`` after ``cat``; a missing newline merges lines
+   and breaks Hugo YAML unmarshalling).
+
+Framing packages may still exist on disk for the website companion path; they
+must simply be omitted from Pathfinder path ``milestones``.
+
+Usage:
+  python3 .github/scripts/validate_learning_path_packages.py
+  python3 .github/scripts/validate_learning_path_packages.py --path run-first-k6-test-lj
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+from pathlib import Path
+from typing import Iterable
+
+
+# Directory / short-name patterns for website-only framing milestones.
+# Keep in sync with .cursor/skills/review-learning-path/reference-checks.md
+#
+# Do NOT treat every "understand-*" milestone as framing. Product teach steps
+# like understand-alerts / understand-dashboards stay in path milestones.
+# Only known concept-intro short names (and shared value/advantages prefixes)
+# are framing.
+FRAMING_EXACT = frozenset(
+    {
+        "advantages",
+        "business-value",
+        "build-mental-model",
+        "how-it-works",
+        "understand-baselines",
+        "understand-value",
+        "welcome",
+    }
+)
+
+FRAMING_PREFIXES = (
+    "advantages-",
+    "business-value-",
+    "build-mental-model-",
+    "how-it-works-",
+    "value-",
+    "value-of-",
+    "welcome-",
+)
+
+
+class Finding:
+    def __init__(self, path: Path, message: str) -> None:
+        self.path = path
+        self.message = message
+
+    def format(self) -> str:
+        return f"{self.path}: {self.message}"
+
+
+def path_slug(path_id: str) -> str:
+    return path_id[:-3] if path_id.endswith("-lj") else path_id
+
+
+def milestone_short_name(path_id: str, milestone_id: str) -> str:
+    prefix = f"{path_slug(path_id)}-"
+    if milestone_id.startswith(prefix):
+        return milestone_id[len(prefix) :]
+    return milestone_id
+
+
+def is_framing_short_name(short_name: str) -> bool:
+    if short_name in FRAMING_EXACT:
+        return True
+    return any(short_name.startswith(prefix) for prefix in FRAMING_PREFIXES)
+
+
+def is_framing_milestone_id(path_id: str, milestone_id: str) -> bool:
+    return is_framing_short_name(milestone_short_name(path_id, milestone_id))
+
+
+def iter_path_packages(root: Path) -> Iterable[Path]:
+    for manifest in sorted(root.glob("*-lj/manifest.json")):
+        yield manifest.parent
+
+
+def load_json(path: Path) -> dict:
+    with path.open(encoding="utf-8") as f:
+        data = json.load(f)
+    if not isinstance(data, dict):
+        raise ValueError(f"{path}: root must be a JSON object")
+    return data
+
+
+def top_level_description(text: str) -> str | None:
+    """Return the top-level YAML ``description`` value, or None if missing.
+
+    Avoids a PyYAML dependency. Supports single-line and simple folded
+    continuations used in Learning Hub website.yaml files.
+    """
+    lines = text.splitlines()
+    for i, line in enumerate(lines):
+        if not line.startswith("description:"):
+            continue
+        value = line[len("description:") :].strip()
+        if value in {"|", ">", "|-", ">-", ""}:
+            parts: list[str] = []
+            if value not in {"|", ">", "|-", ">-", ""}:
+                parts.append(value)
+            for cont in lines[i + 1 :]:
+                if cont.startswith((" ", "\t")):
+                    parts.append(cont.strip())
+                elif cont.strip() == "":
+                    continue
+                else:
+                    break
+            joined = " ".join(parts).strip()
+            return joined or None
+        return value.strip().strip("\"'")
+    return None
+
+
+def find_guide_dir(path_dir: Path, guide_id: str) -> Path | None:
+    for child in path_dir.iterdir():
+        if not child.is_dir():
+            continue
+        manifest = child / "manifest.json"
+        if not manifest.is_file():
+            continue
+        try:
+            data = load_json(manifest)
+        except (OSError, ValueError, json.JSONDecodeError):
+            continue
+        if data.get("id") == guide_id:
+            return child
+    return None
+
+
+def validate_path_package(path_dir: Path) -> list[Finding]:
+    findings: list[Finding] = []
+    path_manifest = path_dir / "manifest.json"
+    try:
+        path_data = load_json(path_manifest)
+    except (OSError, ValueError, json.JSONDecodeError) as err:
+        return [Finding(path_manifest, f"unable to read path manifest ({err})")]
+
+    if path_data.get("type") != "path":
+        return findings
+
+    path_id = path_data.get("id")
+    if not isinstance(path_id, str) or not path_id:
+        findings.append(Finding(path_manifest, 'path manifest missing string "id"'))
+        return findings
+
+    milestones = path_data.get("milestones")
+    if not isinstance(milestones, list) or not milestones:
+        findings.append(
+            Finding(path_manifest, 'path manifest missing non-empty "milestones" array')
+        )
+        return findings
+
+    for milestone_id in milestones:
+        if not isinstance(milestone_id, str):
+            findings.append(
+                Finding(path_manifest, f"milestone entry must be a string: {milestone_id!r}")
+            )
+            continue
+        if is_framing_milestone_id(path_id, milestone_id):
+            short = milestone_short_name(path_id, milestone_id)
+            findings.append(
+                Finding(
+                    path_manifest,
+                    f'framing milestone "{milestone_id}" (short name "{short}") must not '
+                    f"appear in path milestones — keep the package directory for the website "
+                    f"but omit it from Pathfinder milestones",
+                )
+            )
+
+    first_id = milestones[0]
+    if isinstance(first_id, str):
+        first_dir = find_guide_dir(path_dir, first_id)
+        if first_dir is None:
+            findings.append(
+                Finding(
+                    path_manifest,
+                    f'first milestone "{first_id}" has no matching guide directory/manifest',
+                )
+            )
+        else:
+            first_manifest = first_dir / "manifest.json"
+            try:
+                first_data = load_json(first_manifest)
+            except (OSError, ValueError, json.JSONDecodeError) as err:
+                findings.append(
+                    Finding(first_manifest, f"unable to read first hands-on manifest ({err})")
+                )
+            else:
+                depends = first_data.get("depends") or []
+                if not isinstance(depends, list):
+                    findings.append(
+                        Finding(first_manifest, '"depends" must be an array')
+                    )
+                else:
+                    for dep in depends:
+                        if isinstance(dep, str) and is_framing_milestone_id(path_id, dep):
+                            findings.append(
+                                Finding(
+                                    first_manifest,
+                                    f'first hands-on milestone depends on framing package '
+                                    f'"{dep}" — use "depends": []',
+                                )
+                            )
+
+    website_files: list[Path] = []
+    path_website = path_dir / "website.yaml"
+    if path_website.is_file():
+        website_files.append(path_website)
+    for child in sorted(path_dir.iterdir()):
+        if not child.is_dir() or child.name.startswith(".") or child.name == "assets":
+            continue
+        step_website = child / "website.yaml"
+        if step_website.is_file():
+            website_files.append(step_website)
+
+    for website in website_files:
+        try:
+            raw = website.read_bytes()
+            text = raw.decode("utf-8")
+        except OSError as err:
+            findings.append(Finding(website, f"unable to read website.yaml ({err})"))
+            continue
+        except UnicodeDecodeError as err:
+            findings.append(Finding(website, f"website.yaml is not valid UTF-8 ({err})"))
+            continue
+        if raw and not raw.endswith(b"\n"):
+            findings.append(
+                Finding(
+                    website,
+                    "must end with a trailing newline — deploy-preview cats this "
+                    "file then appends type/title/description; a missing newline "
+                    "merges the last line with 'type: docs' and breaks Hugo YAML",
+                )
+            )
+        description = top_level_description(text)
+        if not description:
+            findings.append(
+                Finding(
+                    website,
+                    'missing required non-empty top-level "description" '
+                    "(Learning Hub meta / listings)",
+                )
+            )
+
+    return findings
+
+
+def validate_repo(root: Path, only: str | None = None) -> list[Finding]:
+    findings: list[Finding] = []
+    packages = list(iter_path_packages(root))
+    if only:
+        packages = [p for p in packages if p.name == only]
+        if not packages:
+            return [Finding(root / only, "learning path directory not found")]
+    for path_dir in packages:
+        findings.extend(validate_path_package(path_dir))
+    return findings
+
+
+def main(argv: list[str] | None = None) -> int:
+    parser = argparse.ArgumentParser(
+        description=(
+            "Validate learning-path framing, depends, website.yaml description, "
+            "and website.yaml trailing newlines."
+        )
+    )
+    parser.add_argument(
+        "--root",
+        type=Path,
+        default=Path("."),
+        help="Repository root (default: current directory)",
+    )
+    parser.add_argument(
+        "--path",
+        dest="only",
+        help="Validate a single *-lj package directory name",
+    )
+    args = parser.parse_args(argv)
+
+    root = args.root.resolve()
+    findings = validate_repo(root, only=args.only)
+    if not findings:
+        scope = args.only or "all *-lj packages"
+        print(f"✅ Learning path packaging checks passed ({scope})")
+        return 0
+
+    print("❌ Learning path packaging check failures:")
+    print("")
+    for finding in findings:
+        rel = finding.path
+        try:
+            rel = finding.path.relative_to(root)
+        except ValueError:
+            pass
+        print(f"  - {rel}: {finding.message}")
+        print(f"::error file={rel}::{finding.message}")
+    print("")
+    print(f"{len(findings)} issue(s) found.")
+    print(
+        "See docs/website-yaml-reference.md and "
+        ".cursor/skills/review-learning-path/reference-checks.md (Framing milestones)."
+    )
+    return 1
+
+
+if __name__ == "__main__":
+    sys.exit(main())

.github/workflows/deploy.yml

@@ -63,7 +63,8 @@ jobs:
       - name: Setup Node.js
         uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
         with:
-          node-version: '22'
+          # pathfinder-app engines.node is >=24 (see validate-json.yml).
+          node-version: '24'
           cache: 'npm'
           cache-dependency-path: pathfinder-app/package-lock.json
 
@@ -141,7 +142,8 @@ jobs:
       - name: Setup Node.js
         uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
         with:
-          node-version: '22'
+          # pathfinder-app engines.node is >=24 (see validate-json.yml).
+          node-version: '24'
           cache: 'npm'
           cache-dependency-path: pathfinder-app/package-lock.json