feat: orbital-based plugin distribution (client-side)#430
Merged
Conversation
Contributor
|
Upgrade -> Update, to stay in line with the new semantics |
JeroenSoeters
added a commit
that referenced
this pull request
Apr 25, 2026
Phase 12 of PR #430 deleted the make targets (install-external-plugins, pkg-external-pkl, publish-external-pkl, EXTERNAL_PLUGIN_REPOS) but left two CI workflows still calling them. This commit closes that loop: - release.yml: drop the two external-pkl steps. - e2e-tests.yml: replace the EXTERNAL_PLUGIN_REPOS-building block with a call to scripts/setup-test-plugins.sh; export FORMAE_PLUGIN_DIR for the test step. - scripts/setup-test-plugins.sh (new): clone, build, and stage each test plugin into <out>/<namespace_lowercase>/v<version>/. Honors PLUGIN_REFS for ref overrides. - tests/e2e/go/setup_pkl.sh: read plugins from FORMAE_PLUGIN_DIR, with $HOME/.pel/formae/plugins fallback for local dev.
JeroenSoeters
added a commit
that referenced
this pull request
Apr 25, 2026
Phase 12 of PR #430 deleted the make targets (install-external-plugins, pkg-external-pkl, publish-external-pkl, EXTERNAL_PLUGIN_REPOS) but left two CI workflows still calling them. This commit closes that loop: - release.yml: drop the two external-pkl steps. - e2e-tests.yml: replace the EXTERNAL_PLUGIN_REPOS-building block with a call to scripts/setup-test-plugins.sh; export FORMAE_PLUGIN_DIR for the test step. - scripts/setup-test-plugins.sh (new): clone, build, and stage each test plugin into <out>/<namespace_lowercase>/v<version>/. Honors PLUGIN_REFS for ref overrides. - tests/e2e/go/setup_pkl.sh: read plugins from FORMAE_PLUGIN_DIR, with $HOME/.pel/formae/plugins fallback for local dev.
JeroenSoeters
force-pushed
the
feat/plugin-distribution
branch
from
ae307d1 to
6b49949
Compare
Add Repository struct and RepositoryType constants to pkg/model/config.go, extend ArtifactConfig with a Repositories field, mirror the PKL-side Repository struct in internal/schema/pkl/model/config.go, and wire up translateArtifactConfig in the PKL→Go translation layer.
…/Upgrade Extend the orbitalClient interface with List, Available, and AvailableFor methods, add domain types (Plugin, AvailableFilter, PackageRef, Operation, Response), and implement the six PluginManager methods that wrap orbital's package manager. Includes 22 unit tests covering all methods, filtering, error propagation, and edge cases (nil header, nil version, refresh failure).
Add 5 REST endpoints for plugin lifecycle management: - GET /plugins (list installed or available plugins) - GET /plugins/:name (get plugin details) - POST /plugins/install (install plugins) - POST /plugins/uninstall (uninstall plugins) - POST /plugins/upgrade (upgrade plugins) The plugin manager is optional (nil when orbital repos are not configured); all endpoints return 503 when it is absent.
Rewrite PluginListCmd to use the new REST plugin API instead of stats, add PluginSearchCmd and PluginInfoCmd commands, and create render.go with human-readable formatters. Also add App.NewClient() helper to expose API client creation.
…al-install Add three new cobra commands that proxy to the agent REST API and perform dual-install/uninstall/upgrade on the CLI host for auth-type plugins via CLIPluginManager. These commands are not yet registered in PluginCmd (deferred to a follow-up task).
Phase 12 of PR #430 deleted the make targets (install-external-plugins, pkg-external-pkl, publish-external-pkl, EXTERNAL_PLUGIN_REPOS) but left two CI workflows still calling them. This commit closes that loop: - release.yml: drop the two external-pkl steps. - e2e-tests.yml: replace the EXTERNAL_PLUGIN_REPOS-building block with a call to scripts/setup-test-plugins.sh; export FORMAE_PLUGIN_DIR for the test step. - scripts/setup-test-plugins.sh (new): clone, build, and stage each test plugin into <out>/<namespace_lowercase>/v<version>/. Honors PLUGIN_REFS for ref overrides. - tests/e2e/go/setup_pkl.sh: read plugins from FORMAE_PLUGIN_DIR, with $HOME/.pel/formae/plugins fallback for local dev.
- agent.go: drop trailing blank line (gofmt). - render.go: replace sb.WriteString(fmt.Sprintf(...)) with fmt.Fprintf(&sb, ...) (staticcheck QF1012). - plugin_manager: move newForTesting to a unit-tagged helpers_test.go file so the unused linter sees it under the build tag the tests use.
PKL's Listing<ClassType> decodes into Go via reflection as a slice of pointers, not values. Declaring the field as []Repository panicked with "reflect.Set: value of type *model.Repository is not assignable to type model.Repository" the first time PKL produced a non-empty repositories listing. Match the pattern used elsewhere (e.g., FilterCondition conditions) and use a pointer slice. The translation layer at pkl.go:720 reads through the pointer transparently.
The agent refuses to start without an auth plugin installed; e2e tests were timing out because aws/azure/compose/grafana are resource plugins only. setup-test-plugins.sh now also stages auth-basic, and branches the destination path on the plugin's type field: auth plugins go to <out>/<name>/v<version>/<name>, resource plugins keep the lowercase namespace layout.
The PKL Config.pkl default for pluginDir is "~/.pel/formae/plugins", which the agent reads at runtime regardless of the FORMAE_PLUGIN_DIR env var (the env var only feeds the PKL evaluator's plugins:/ scheme mount, not the runtime plugin search path). When agent.go generates the test config, inject a top-level pluginDir override pointing at FORMAE_PLUGIN_DIR so the agent finds plugins in the test-scoped tree.
Re-applies the Phase 12 cleanup that was effectively dropped when rebasing onto main, where 055ee2a reintroduced EXTERNAL_PLUGIN_REPOS to keep the dev container build working until aws v0.1.6-dev is published to orbital. Removed: EXTERNAL_PLUGIN_REPOS, PLUGINS_CACHE, fetch/build/install-external-plugins, gen/pkg/publish-external-pkl, the pkg-bin plugin loop, the test-e2e install-external-plugins dep, and the .PHONY entries. Container build (release.yml -> pkg-bin via build-external-plugins) is broken on this branch as a deliberate transitional state. It's restored by the orbital-install switch (Phase G.1 of the execution plan), which publishes aws v0.1.6 to orbital's dev channel and points the container build at "ops install formae standard" instead.
Three findings from the rebased CI run: - internal/cli/app/app.go: findInstalledPlugin (used by formae project init --include) hardcoded ~/.pel/formae/plugins. Read FORMAE_PLUGIN_DIR first. - internal/schema/pkl/pkl.go: GenerateSourceCode (used by formae extract) has the same hardcode for local schema resolution. Same fix. - scripts/setup-test-plugins.sh: TestPluginConfig imports plugins:/Sftp.pkl; add sftp to the plugin set so the file is present in FORMAE_PLUGIN_DIR.
When extracting into an existing IaC codebase, parse the target's PklProject and reuse its declared dep versions for the temp generator project — no version skew between extraction and the user's later evaluation. When extracting a single file, discover deps from the configured plugin dir (Config.PluginDir) instead of FORMAE_PLUGIN_DIR. Threads plugin dir through findInstalledPlugin, formatIncludes, Projects.Init, App.GenerateSourceCode, and SerializeOptions. The only FORMAE_PLUGIN_DIR read that remains is the bootstrap defaultPluginDir in plugins.go (runs before config has been parsed — legitimate). Forward-compat with PR #410: the single pluginsDir threaded here becomes []string{devDir, systemDir} in a follow-up.
Two regression fixes for the config-driven plugin dir refactor: - App.NewApp now defaults Config.PluginDir to "~/.pel/formae/plugins" (matches the PKL Config.pkl default) so CLI commands invoked without a config file still resolve @Local plugin schemas. After PR #410 this dir is semantically the "dev plugin dir" and only used for @Local includes (dev workflow); production users go through the system plugin dir via discovery.SystemPluginDir. - The e2e ProjectInit helper now passes --config so the test harness's injected pluginDir is honored. The earlier "doesn't need --config" comment was true only when findInstalledPlugin read FORMAE_PLUGIN_DIR directly; now that the dir comes from Config, project init needs --config too.
The previous regression fix (e75de07) had ProjectInit pass --config from the e2e harness, but project init has no --config flag — that broke TestProjectInit with "unknown flag: --config". Project init only ever used Config.PluginDir; nothing else from config applies (no agent connection, no auth). Drop the config dependency: - Add --plugin-dir flag to project init, default ~/.pel/formae/plugins. Matches the dev-plugin-dir convention — PR #410 keeps this location as the dev/override path while system plugins move to /opt/pel/formae/plugins. - Project init no longer calls cmd.AppFromContext — calls Projects.Init directly with the flag value. - e2e harness reads FORMAE_PLUGIN_DIR and passes it as --plugin-dir. The App.NewApp default from e75de07 stays — still useful for other commands (extract, eval) that load Config and may run without a config file.
Picks up two fixes that the plugin distribution work depends on: - Version.Parse and Semver now preserve PreRelease and Build, so prerelease tags like 0.1.0-dev.1 round-trip through orbital's cache and the package solver instead of collapsing to 0.1.0 (orbital#9, v0.1.37). - Transaction.remove no longer fails when Objects.Del returns ErrNotFound. Metapackages legitimately have no fs entries; the upstream behaviour caused \`formae plugin uninstall <metapackage>\` to surface a confusing 500 even though the package was actually removed (orbital#10, v0.1.38).
…ast init
Reworks the agent-side plugin manager to support per-query channel
selection, surface curated bundles alongside regular plugins, and
fail loudly at startup when its prerequisites aren't met.
Channel routing
PluginManager holds an orbital factory keyed on channel. Available,
Info, Install, and Upgrade resolve their channel via
clientFor(channel) — empty defaults to DefaultChannel ("stable") so
users see only stable packages unless they pass --channel. Uninstall
stays channel-agnostic since it operates on local state.
AvailableFilter, InstallRequest, and UpgradeRequest gain Channel
fields plumbed end-to-end. GetPlugin gains a channel query param.
Metadata-driven filter
isPluginPackage now checks display.kind ∈ {plugin, metapackage}.
Binary tooling (formae itself, pkl) is filtered out cleanly, and
curated bundles surface as first-class entries with Plugin.Kind set
so the renderer can group them.
Prerelease-safe versioning
versionString builds Major.Minor.Patch[-PreRelease][+Build] ourselves
rather than relying on orbital's Short. uniqueVersions dedupes the
tree-and-repo double-listing that surfaced as "0.1.0, 0.1.0" in info.
InstalledVersion correctness
pluginFromPackage no longer sets InstalledVersion blindly from
Available[0]. List sets it directly from the installed package;
Available/Info derive it from the Installed flag on the candidate
list, so search no longer falsely marks every search hit as installed.
Not-found and refresh hygiene
Info now refreshes the orbital cache before lookup (matches what
Available already did) and translates orbital's "no available
packages for: X" string into nil/nil not-found instead of a 500.
Fail-fast init
agent.Start now constructs the plugin manager before announcing
startup. If plugin_manager.New fails (most often because the orbital
tree isn't initialized at the path derived from the binary location)
the agent logs and exits rather than running with a silently disabled
plugin endpoint — CLI users on a remote agent would otherwise have
seen only 503s.
…ewline Surfaces the plugin manager's channel routing through the CLI and polishes the rendered output of list/search/info. - info, install, upgrade gain a --channel flag that flows through to the agent (search already had it). - install/uninstall/upgrade build the CLI-local plugin manager before printing the agent results. NewCLIPluginManager can trigger orbital's syscall.Exec sudo elevation (when /opt/pel is root-owned and the user is not), and the elevation re-runs the CLI from main. Printing agent results before the elevation produced duplicate "Installed X on agent" lines; deferring the prints lets the original process exit silently and only the privileged re-exec emits output. - list groups output as Resource plugins / Auth plugins / Bundles. Bundles render as "metapackage" internally (the orbital primitive) but the CLI shows them as "Bundles:" with Type: bundle in info, so users get friendlier vocabulary while the wire format stays vendor-neutral. - info shows Description for bundles when it differs from Summary so users can tell what a bundle pulls in. - "No plugins installed." and "No plugins found." end with a newline (zsh % indicator removed). Group headers no longer say "(agent)" — both kinds run on the agent, the qualifier was leftover from when CLI-side plugins were a separate concept.
Replaces the transitional Dockerfile state (which still installed only the formae binary and relied on the legacy bundled-plugin extraction step) with the orbital model: setup.sh installs formae alongside the \`standard\` metapackage, whose requires resolve at install time and pull in the curated default plugin set (aws, azure, gcp, oci, ovh, auth-basic). The plugin-migration RUN step is dropped — there are no bundled plugins to extract once the binary ships without them.
…dy() The blackbox harness builds the formae binary into a temp dir and starts it as a subprocess in that same temp dir's tree. After the agent gained fail-fast plugin-manager initialization, this fails: orbital derives treeRoot from os.Executable() via filepath.Dir(filepath.Dir(binPath)), so a binary at <tmpDir>/formae makes treeRoot = /tmp, which has no .ops/ directory and therefore Ready()=false. Move the binary one directory deeper (<tmpDir>/bin/formae) so treeRoot resolves to <tmpDir>, and create an empty .ops/ marker there. Ready() only checks for the directory's existence — no signed tree, no certs, no state DB needed — so this is a 2-line scaffolding step that costs a single mkdir at startup. Production builds installed via setup.sh land at /opt/pel and follow the same path naturally.
The e2e Makefile target built the binary at <CURDIR>/formae and pointed
E2E_FORMAE_BINARY there. Orbital embedded mode derives treeRoot via
filepath.Dir(filepath.Dir(binPath)), which resolved to the parent of the
repo — no .ops/ marker there, so the agent's fail-fast plugin-manager
init refused to start with "orbital tree not initialized" and every e2e
test failed at agent health-check.
Stage the binary into dist/e2e/bin/formae and create dist/e2e/.ops/
before running tests. This mirrors the layout that setup.sh / pelmgr
lay down at the install path (verified locally: pelmgr install creates
<install-path>/bin/formae + <install-path>/.ops/{cache,pki.db,state.db}),
so treeRoot resolves to dist/e2e/ and Ready() passes. dist/ is already
in .gitignore.
JeroenSoeters
force-pushed
the
feat/plugin-distribution
branch
from
b4f76d2 to
3b91368
Compare
The e2e workflow used to clone each plugin's main branch, build it,
and stage binaries in $RUNNER_TEMP/test-plugins exposed to the test
agent via FORMAE_PLUGIN_DIR. That side-channel skipped orbital
entirely — the agent's PluginManager initialized but never exercised
the install path real users take.
Replace it with the production install flow:
- Build formae from source, stage at dist/e2e/bin/formae +
dist/e2e/.ops so orbital's Embedded mode resolves treeRoot to
dist/e2e.
- Start a bootstrap agent on the test runner with the default
artifacts.repositories (pel#stable + community#stable, no auth,
no discovery, no sync).
- formae plugin install <per-test plugin list> --channel stable.
Plugins land at dist/e2e/formae/plugins/<name>/v<version>/, which
the agent's multi-source discovery (added in the prior plugin-
discovery refactor) then finds via SystemPluginDir(binPath).
- Stop the bootstrap agent so the test harness can spawn its own.
The matrix now uses include: blocks declaring the plugin set per
test, so each runner only installs what it needs (~7-30s per
runner, e.g. aws+azure) instead of building all 6 plugins.
Drop scripts/setup-test-plugins.sh and the now-unused plugin_refs
workflow_dispatch input. setup_pkl.sh's error message updated to
point at \`formae plugin install\` rather than the deleted helper.
The Makefile's test-e2e target no longer rm -rf's dist/e2e so the
orbital install survives the rebuild step.
Local dev path unchanged: setting FORMAE_PLUGIN_DIR or doing
\`make install\` in plugin repos still works for ad-hoc runs.
`formae plugin install` against a fresh tree fails with "no install candidates found" because the agent's install endpoint doesn't auto-refresh repository metadata — only the Available endpoint does. Prepend `formae plugin search --channel stable` to populate orbital's cache before installing. Tracked as a follow-up in the install endpoint itself; this is the workflow-side workaround.
A user running \`formae plugin install foo\` against a fresh orbital tree shouldn't have to know to call \`formae plugin search\` first. But the install endpoint doesn't auto-refresh — only Available does — so a fresh tree fails with "no install candidates found". Refresh once at PluginManager construction, before the agent finishes starting. Best-effort: a refresh failure (hub unreachable, etc.) logs a warning rather than blocking startup; subsequent operations rely on cached data and refresh on their own when they can. Drop the workaround \`formae plugin search\` step from the e2e workflow now that startup handles this. Follow-up: an explicit user-facing "refresh metadata" command (apt update analog) for stale-cache cases between agent restarts. Naming TBD — \`plugin update\` is taken by the upcoming \`plugin upgrade\` rename in the hub design.
Per-test orbital install (matrix.plugins) means setup_pkl.sh runs
with only a subset of plugins available — TestAuthBasic installs
auth-basic only, TestPluginConfig installs sftp only, etc. The
script previously required AWS+Azure unconditionally, exiting 1 in
hub_uri's $(...) subshell with the error swallowed by command
substitution; the failure surfaced only as a bare "Error 1" from
make.
Make all hub_uri calls non-required; fixtures that import a plugin
not installed will fail to evaluate with a clear PKL error at the
right layer (test mismatch with declared deps).
Also:
- Send hub_uri's stderr ERROR/WARN to >&2 so failures appear in
workflow logs even when the function output is captured by
$(hub_uri ...).
- Fix compose plugin dir name: setup-test-plugins.sh staged by
namespace (DOCKER → "docker/"), but orbital installs by package
name ("compose/"). The script now matches orbital's layout.
…on against same-path
Two related fixes for the orbital-flow e2e regression where
TestAuthBasic (and others) failed with "Auth plugin not installed —
refusing to start without auth" right after \`formae plugin install
auth-basic\` reported success.
Root cause: the test workflow set FORMAE_PLUGIN_DIR=
\$WORKSPACE/dist/e2e/formae/plugins so setup_pkl.sh could find the
orbital-installed plugins. agent.go injected that into the test
agent's cfg.PluginDir. The plugin-discovery refactor's
SystemPluginDir(binPath) for binaries at dist/e2e/bin/formae also
resolves to dist/e2e/formae/plugins. dev dir == system dir.
CleanStaleDevPlugins then matched every "system" plugin against an
identical "dev" entry and deleted the only copy.
- tests/e2e/go/agent.go: drop the FORMAE_PLUGIN_DIR override.
cfg.PluginDir defaults to ~/.pel/formae/plugins (empty in CI);
the multi-source scan finds orbital-installed plugins via
SystemPluginDir(binPath) without further help. setup_pkl.sh
keeps reading FORMAE_PLUGIN_DIR independently.
- pkg/plugin/discovery/migrate.go: defensive guard. When devDir
and systemDir resolve to the same path on disk
(os.SameFile-aware so symlinks, ./ prefixes, and absolute-vs-
relative variants are treated correctly), skip the migration
entirely — every "stale dev" match would be deleting the only
copy. Production deployments always have separate dev and
system dirs; this only fires in deliberate test setups that
point cfg.PluginDir at the system directory.
The test invokes \`formae project init --include aws --include azure\` which expects both plugins installed. Matrix was \`plugins: azure\` only, so AWS wasn't installed and the PKL evaluator surfaced "Element index 1 is out of range" when the plugin manifest list came up one short.
Reverts the previous \"drop pluginDir override\" change. Agent runtime found plugins fine via SystemPluginDir, but the CLI's extract command reads pluginDir from the same formae.conf.pkl and uses it as LocalPluginDir for schema resolution. Without it, the PackageResolver scans an empty ~/.pel/formae/plugins, finds nothing, falls through to the remote-package code path with empty Version, and PklProjectTemplate.pkl crashes splitting "aws.aws" on "@" (no @-delimiter present). Mirror FORMAE_PLUGIN_DIR into the agent config again. The CleanStaleDevPlugins same-path safety check (in the prior commit) prevents the devDir==systemDir collision this re-introduces. Once #410's multi-source pattern is propagated into the PackageResolver / CLI's a.Config.PluginDir, this override can be dropped permanently.
…ect init
`formae extract` and `formae project init` previously scanned the CLI
machine's local plugin dir to look up plugin versions for the
generated PklProject's dependency URIs. After the multi-source
plugin-discovery refactor, orbital-installed plugins live with the
agent — not the CLI. On any deployment where the two are separate
(or in CI, where plugins land at the agent's binary-derived system
path), the local scan finds nothing and the generated PklProject
contains malformed dep strings (`aws.aws@`, no version), crashing
PklProjectTemplate.pkl with "Element index 1 out of range" when
`pkl project resolve` evaluates it.
Make the agent the source of truth:
- App.InstalledResourcePluginVersions() wraps the existing
GET /api/v1/plugins?scope=installed call and returns a
namespace→version map.
- App.GenerateSourceCode (the extract path) drops LocalPluginDir
entirely. It collects namespaces from the forma's resources,
pairs them with versions from the agent, and pre-populates
options.Dependencies so the schema plugin's resolveIncludes
short-circuits the local-scan logic.
- Projects.formatIncludes (the project-init path) uses the same
map for non-@Local includes. @Local includes still resolve
against --plugin-dir on disk, since their purpose is to point
at a developer's freshly-built plugin schema. ProjectInitCmd
only fetches the version map when the include set actually
needs it (skip the agent round-trip if every include is @Local).
Test changes:
- TestProjectInit now starts an agent. The CLI helper passes
--config so project init can talk to it. Out-of-scope previously
because project init was CLI-only; that's no longer true once
plugin versions live on the agent.
- tests/e2e/go/agent.go drops the FORMAE_PLUGIN_DIR-into-pluginDir
mirror. cfg.PluginDir defaults to ~/.pel/formae/plugins (empty
in CI) and the multi-source discovery finds orbital-installed
plugins via SystemPluginDir without help. setup_pkl.sh still
reads FORMAE_PLUGIN_DIR independently of the agent config.
Follow-ups out of scope:
- eval (App.SerializeForma) has the same local-scan dependency.
Not touched here because no e2e test exercises it. Same
refactor applies whenever it surfaces.
- greenfield `formae project init` (no agent yet) currently
errors out asking the user to install plugins via the agent
first. A future change could let `--include aws@0.1.6` pin
explicit versions without an agent round-trip.
The previous commit made project init talk to the agent for non-@Local plugin versions. Without --config, the App's API client uses the default port (49684), which doesn't match the test agent (random port) or any non-default deployment. Add --config like every other agent-touching CLI command.
App.SerializeForma previously scanned the CLI machine's local plugin dir to look up plugin versions for the temp PklProject's dependency URIs. After the multi-source plugin-discovery refactor and the extract path's switch to agent-source-of-truth in #430, eval is the last surface that still scanned locally. On any deployment where the CLI and agent are separate (or in CI where plugins land at the agent's binary-derived system path), the local scan finds nothing and the temp PklProject contains malformed dep strings (`aws.aws@`, no version), which fails to evaluate. Mirror the pattern used by App.GenerateSourceCode: query the agent's InstalledResourcePluginVersions(), force SchemaLocation=Remote, and pre-populate options.Dependencies via buildRemoteDependencyStrings. The schema plugin's resolveIncludes already short-circuits the local scan when Dependencies is non-empty, so no plugin-side change.
After f3646b6 (extract uses agent versions) and the eval fix above, no production code path calls PackageResolver.HasRemotePackages — the classification 'is this dep remote?' is now determined upstream by the caller building the dep strings, not by the resolver. Drop it.
Three new e2e tests drive the --schema-location flag work: - TestEvalDefault: regression coverage for the post-fix eval path. Locks down that 'formae eval' against a forma using a resource plugin produces valid JSON. Should pass today. - TestEvalSchemaLocationLocal: invokes 'formae eval --schema-location local'. Today: the flag is unknown so the command fails. After implementation: the agent serves localPath, eval resolves plugin schemas from the agent's local filesystem paths. - TestExtractSchemaLocationLocal: applies a small fixture, extracts with --schema-location local, asserts the generated PklProject contains import() calls and no package:// URIs. Today: red on the unknown flag. Adds an extract_schema_local_aws.pkl fixture with unique resource names so it does not collide with TestExtractAndReapply in the parallel matrix. Wires Eval and a variadic ExtractToFile into the formae CLI helper.
IsDynamicCommand (cmd.go:178) iterates os.Args looking for the first argument containing a supported file extension and treats it as the forma file. With --config /path/to/formae.conf.pkl appearing before the fixture, the dynamic-command pre-eval picked up the config path as the forma, and PKL refused to load formae:/Config.pkl because the Properties evaluator was set up without the formae: scheme on its allowlist (only FormaeConfig adds that scheme). Apply's helper already uses fixture-before-config ordering for the same reason. Match the convention. Filed for follow-up: IsDynamicCommand should skip flag values when scanning for the forma file; today it's order-dependent and any .pkl path appearing before the forma file (--config, --plugin-dir in some shapes, etc.) breaks the dynamic-command discovery.
Eval JSON output uses Stacks, Targets, Resources (all plural). Fix the assertion in TestEvalDefault to match.
Adds an opt-in --schema-location {remote|local} flag to formae eval
and formae extract. Default remote (today's behavior post-cb25fd6c):
emit package:// URIs that PKL fetches from the hub. Opt-in local:
emit local file imports against the agent's on-disk PklProject
paths; same-box only.
Plumbs the on-disk path through the API:
- apimodel.Plugin gains a LocalPath field surfacing the absolute
path to each plugin's PklProject on the agent's filesystem.
- plugin_manager.New takes pluginDirs and re-runs the discovery
scan in List() so LocalPath comes straight from
discovery.DiscoverPluginsMulti, the same source the agent uses
at startup to populate the PluginProcessSupervisor.
- agent.go passes the existing devPluginDir + systemPluginDir.
CLI side:
- App.InstalledResourcePlugins returns versions + LocalPath keyed
by lowercase namespace.
- App.buildDependencyStrings emits either remote (`<plugin>.<name>@<v>`)
or local (`local:<name>:<path>`) dep strings based on the caller's
SchemaLocation. Same-box check stat()s each agent-reported path
and fails fast with a clear error pointing at the constraint.
- App.GenerateSourceCode now takes a SchemaLocation argument.
- eval.go and extract.go register --schema-location and pass it
through.
Formae core stays remote in both modes — the agent doesn't surface
its own PKL schema as a local path, and dev workflows typically test
against a published formae version.
Resolves the dev-workflow regression introduced when the eval/extract
paths switched to agent-source-of-truth in 79063e5 and f3646b6.
The remote default keeps the cross-box deployment shape working;
local opt-in restores the "iterate on a plugin without publishing"
flow for users running CLI and agent on the same host.
The PklProject under --schema-location local contains both a local import() for aws and a remote package:// URI for formae core (the agent does not surface its own PKL schema as a local path). Tighten the assertion to check that aws specifically resolves locally, and that aws is not also resolved remotely. The presence of 'package://hub...' for formae is expected.
Replaced by buildDependencyStrings in the --schema-location refactor; golangci-lint flagged the leftover as unused.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Migrates plugin distribution from "bundled with the formae binary" to "served via orbital from the platform.engineering Hub", and reshapes the CLI/agent surface to match. Net change: 102 files, +6.9K/-744 across 67 commits.
What ships in this PR
Plugin distribution
standardmetapackage's manifest, and resolves plugins on first install via orbital from thecommunityrepo onhub.platform.engineering. The container image (Dockerfile),setup.shflow, andmaketargets reflect this.formae plugin install / uninstall / upgrade / search / info / list / init(internal/cli/plugin/*.go). Auth plugins (e.g.auth-basic) are dual-installed on agent and CLI in a single command.formae extract,formae project init, andformae evalall query the agent'sGET /api/v1/plugins?scope=installedendpoint instead of scanning the CLI box's local plugin dir. Cross-box deployments (CLI on a different host than the agent) work correctly; eval no longer generates malformedaws.aws@dependency strings.--schema-location {remote|local}flag onformae extractandformae evalfor the same-box developer workflow. Defaultremoteemitspackage://...URIs; opt-inlocalemits local file imports against the agent's on-disk PklProject paths and validates the path is readable from the CLI host.pkg/plugin/discovery/): the agent scans both~/.pel/formae/plugins(dev installs) andSystemPluginDir(binPath)(orbital installs); dev wins on collision.internal/metastructure/plugin_manager/).Configuration
artifacts.repositoriesconfig block. NewListing<Repository>withpel(binary) andcommunity(formae-plugin) defaults pointing athub.platform.engineering. The flatartifacts.url/artifacts.username/artifacts.passwordfields are kept as@Deprecatedfor backward compat with synthesis + warnings viaemitArtifactDeprecationWarnings(internal/schema/pkl/pkl.go).apimodel.Plugingains aLocalPathfield surfacing the on-disk PklProject path; the agent populates it from the discovery scan inplugin_manager.List().Test coverage
tests/e2e/go/eval_test.go):TestEvalDefault(regression),TestEvalSchemaLocationLocal,TestExtractSchemaLocationLocal.e2e-tests.ymlmatrix updated to install only the plugins each test needs (per-testplugins: aws-style declarations) instead of cloning + building all of them upfront.formae plugin installagainst an ephemeral agent before the test harness spins up its own.Other notable changes
pkg/plugin/discovery/migrate.go— one-time migration of plugin layouts from the legacy~/.pel/formae/pluginsshape to the<name>/v<version>/shape orbital expects.pkg/plugin/descriptors/— extract-schema improvements with new dependency-staging tests.pkg/plugin-conformance-tests/) updated to drive the CLI's plugin install path instead of its own bundled-plugin extraction.Adjacent repos that had to change
Every change required outside this PR is on the corresponding repo's
main. Linked below.platform-engineering-labs/orbital— bumped from v0.1.35 → v0.1.38 in both formae and pel-manager. Earlier releases silently dropped the second repo's metadata in multi-repo configurations; v0.1.38 fixes the refresh path. Repo: https://github.com/platform-engineering-labs/orbital.platform-engineering-labs/pel-manager— added thecommunityrepo to the embedded defaultTreeConfigsopelmgr install standardfinds the metapackage. Pelmgr was hard-coded to know only thepelbinary repo. Also fixedSetup()to apply--channelto all configured repositories. PR: Add community formae-plugin repo to default tree config pel-manager#1 (merged, released as v0.1.17 to S3). Repo: https://github.com/platform-engineering-labs/pel-manager.platform-engineering-labs/formae-actions— two CI changes onmain:schema/pkl/VERSIONfrom the release tag before opkg build, not after, so the published opkg always carries the right version. Surfaced when sftp@0.1.0 shipped a broken opkg.kind=metapackagematrix across all 4 platforms (linux x86_64/arm64, macos x86_64/arm64). Previously the metapackage was built once onubuntu-latestand ended up inlinux-x8664/metadata.dbonly; arm64 / macOS users couldn't resolvestandard. The proper fix is forops publishto handle noarch metapackages; tracked as a follow-up against orbital.Repo: https://github.com/platform-engineering-labs/formae-actions.
All 10 plugin repos (
formae-plugin-{aws,azure,gcp,oci,ovh,compose,grafana,sftp,auth-basic,k8s}) andformae-plugin-template:schema/pkl/VERSIONis now generated fromformae-plugin.pkl'sversionfield atmake buildtime, eliminating drift. The committed file was previously stale (e.g. AWS had0.1.5in VERSION while the manifest said0.1.6). New plugins scaffolded from the template inherit the generated-VERSION shape.platform-engineering-labs/formae-plugin-standard— re-tagged0.1.0after the formae-actions fan-out fix so the metapackage now exists in every platform's metadata.platform-engineering-labs/formae-docs— release notes and configuration docs additions for the 0.85.0 cut. Lives on therelease/0.85.0branch in formae-docs and ships on its own cadence with the release.RFCs
Deliberately out of scope (tracked as follow-ups)
artifacts.username/artifacts.password.ops publishto register noarch metapackages into every platform's metadata index (today's metapackage fan-out is the workaround).formae evalandformae extractagainst an unpublished plugin in cross-box deployments —--schema-location localcovers same-box; cross-box would need agent-served PKL packages over HTTP.