Delayed release of named dependencies as "sub" catalogs.

[TripleDogDare]

@warpfork We talked about this a bit but I'm trying to figure out how this relates to workspaces and candidates.

I have a project that's pulling in some non-reproducible input. Let's say some variant of a base image that I want to name "ubuntu" and release with the catalog of my primary module as a dependency.

In final release this might result in something along the lines of
.timeless/catalog/github.com/tripledogdare/<project>/ubuntu/catalog.tl
But updates to those dependencies might be done manually or maybe by another module at some point. But I still would like to track them as releases separately from the primary release. This would allow easy roll forward and back for various dependencies being mounted if needed.

We also discussed putting workspace config into .timeless/config.json which might keep some config for a workspace name. This would allow delayed resolve of a final name during build/release. E.g. I would build things and records would populate with $workspace where the domain github.com/tripledogdare/<project> would go. When a release is specified the tools will force you to choose a host location. Perhaps populating automatically with the git origin remote url.

Ultimately I want something where stellar init doesn't need a remote repository or other hosting info to be required until releases occur. Similar to how git doesn't require user config until commits are made.

[warpfork]

Something following that .timeless/catalog/github.com/tripledogdare/<project>/ubuntu/catalog.tl pattern should just work.

There would be nothing particularly special about having that name prefix. It might suggest to a human reader that this is "the ubuntu instance I packed for this project", but in actionable considerations to the tooling, it's just another name.

But updates to those dependencies might be done manually

Should always be fine.

or maybe by another module at some point.

Should also be almost a non-event if we get everything right. The steps would be roughly:

• make another directory in your source tree (e.g. not under .timeless/catalog) to put the new module.tl
• write the actual module
• evaluate it; it should write a new release of version "candidate" into .timeless/catalog/...
• (... profit!)
• when publishing releases later, every manifest in the catalog with a release named "candidate" will have to be rewritten to use some other release name (but the tooling should help automate that and give nice prompts).

When a release is specified the tools will force you to choose a host location.

(where "host location" is aka "module name" or maybe will be aka "import path"...)

Right. This is easy -- for a single module. At no point do we need a module name for it when eval'ing a single module. Right now, the stellar eval tool already does this: it simply won't write candidate releases into the catalog if it can't see a name for the module.

The implication here is that the "candidate" release... though not being published... is for at least some purposes, still a "release". If we want to use a candidate release in another module... we still need module name to import it by. So, the ability to defer picking a module name is limited: it only works as long as you don't have anything importing it.

[warpfork]

So harkening back to the issue title... """Delayed release of named dependencies as "sub" catalogs"""

I see you already put air-quotes around 'sub' there, but for clarity on the record, "subcatalogs" is definitely not a thing (in API terms). But let's say that's a general term for the human-level behavior of having common prefixes in module names aka import names.

So we want to defer the choice of naming a whole bunch of modules and catalogs at once, and give them all a (presumable shared prefix) name at once later? Similar to how we have defered naming of releases themselves with the 'candidate' system?

One main concern is that that non-fully-qualified import names will definitely not jive with the goals of published releases: we need fully-qualified names for recursive audit to be tractable. However, there's precedent for this, in both 1) the candidate system, and 2) ingests: Both are things which are useful for specifying data flow in development, and both need to be rewritten before having a release that's valid to publish; and we did it and it works. So, this concern is surmountable.

Then the other concern is... syntactically, how?

And a third concern is the usual "will this flexibility provide more value than it costs in user-facing complexity?". (And I have no fully formed opinion on that yet. We should think about it after coming up with syntax ideas though.)

[warpfork]

how this relates to [...] candidates.

In tl;dr: the deferrability of naming comes up in two situations. Candidates solve it for release naming. Module naming / import path choice can currently be deferred for a single module as well... but only up until the point that something wants to depend on it. We currently have no system for deferring the naming of multiple modules (and the candidate system is totally orthogonal).

[warpfork]

It's interesting to note that while the release name issue and candidates comes up every release -- and it's correct and normal to provide a new value every time -- with a system for deferred module naming, this isn't true. A module name is assigned once.

(Doing a bulk-rename operation later is... very much not a case we should optimize for. It's possible, in the sense of "anything's possible", but should be avoided: in addition to all the usual reasons that renaming things causes communication needs and changes that are harder to propagate around large and multi-author systems... when renaming a module in the Timeless Stack, we can only ever copy the existing release manifests to the new name and go from there. We can't discard the old manifests under their old name because then "recursive explain" of any other replays which refer to that old name would fail.)