diff --git a/Cargo.lock b/Cargo.lock index 7a73c71..f40cc43 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -928,7 +928,7 @@ checksum = "384b8ab6d37215f3c5301a95a4accb5d64aa607f1fcb26a11b5303878451b4fe" [[package]] name = "openresearch-cli" -version = "0.1.61" +version = "0.1.62" dependencies = [ "anyhow", "async-trait", diff --git a/Cargo.toml b/Cargo.toml index 93ad78a..c06888f 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "openresearch-cli" -version = "0.1.61" +version = "0.1.62" edition = "2021" description = "OpenResearch CLI (orx) — Rust port" repository = "https://github.com/alphaXiv/openresearch-cli" diff --git a/SKILL.md b/SKILL.md index 1e3d3d1..bd6e04d 100644 --- a/SKILL.md +++ b/SKILL.md @@ -11,21 +11,26 @@ tree, runs, logs, artifacts, evidence database) and **write actions** (create experiments, launch/cancel runs on GPU compute). Code is the one thing `orx` does not serve: every experiment is a git branch on the project's GitHub repo, and the **local clone in `~/.cache/openresearch/repos//` is -the standard way to read, diff, and edit it** (see "Reading & editing a node's -code"). Use `orx` when you need to inspect or drive project state from a shell -instead of the web UI. +the standard way to read, diff, and edit it** (see the `orx-git` module). Use +`orx` when you need to inspect or drive project state from a shell instead of the +web UI. + +This overview is deliberately short: it carries the cardinal rules and a command +quick-reference, then points at focused **modules** for everything else. Load a +module with `orx skill ` (the live index is printed at the end of `orx +skill` output). ## Cardinal rules — read before doing anything else These four govern everything below. Breaking any one silently invalidates your -results — they are not style preferences. The detailed "experiment-tree model" -section expands on the why; these are the non-negotiables. +results — they are not style preferences. The `orx-experiment-tree` module +expands on the why; these are the non-negotiables. 1. **Never edit a baseline (root) once it holds real code.** A root is the control its variants are measured against. Projects start with an empty tree — **you create the baseline** (the first `orx create-experiment`, no `--parent`) and, on a blank repo, seed it with starting code before its first - run (see "Seeding an empty baseline"). That setup window is the one + run (see the `orx-create` module). That setup window is the one exception, like rule 2's single legitimate `--set`. From the moment a root holds real code this rule is absolute: to try an idea, **branch a child** and edit the child. Editing a root moves the goalposts and destroys every @@ -44,7 +49,7 @@ section expands on the why; these are the non-negotiables. 4. **Grow the tree downward, not sideways.** Fan a little *within* a round (the options of one decision), then **descend onto that round's winner** for the next round. A root with a long row of direct children and no grandchildren is - the failure mode. See "Shape the tree" below. + the failure mode. See "Shape the tree" in the `orx-experiment-tree` module. If you're ever tempted to change the command, pass an env var, or pile another node onto the root instead of branching a child, editing its branch, and @@ -61,7 +66,12 @@ orx logout # remove the stored token default. Set `OPENRESEARCH_API_URL` for non-local use. - Every command except `login`, `lit`, and `paper` needs a token; if you see `Not logged in`, run `orx login`. (`lit` and `paper` hit alphaXiv's public hosts and work without one.) -## Commands +## Command quick-reference + +Project-scoped commands take a **project id**; experiment-scoped commands take an +**experiment id**; run-scoped commands take a **run id**. Don't mix them — get +ids from `orx projects`, `orx experiments`, and `orx runs` respectively. Each +group below has a module (`orx skill `) with the full flags and rules. ### Auth | Command | What it does | @@ -72,1027 +82,66 @@ orx logout # remove the stored token ### Discover (project- and experiment-scoped) | Command | What it does | |---|---| -| `orx projects [--all] [--json]` | List your projects (id + name + GitHub `owner/repo`), grouped by org (name + org id). `--all` includes archived; `--json` emits a flat array (incl. each project's `paperId`) for scripts. **Project ids, org ids, and the repo to clone come from here.** | -| `orx explore [--json]` | List the **public** project directory (id + name + repo) — projects anyone can view, not just yours. Use it to discover others' work; then drill in with `orx project view`, `orx experiments`, `orx runs`, `orx report show` on the id. | -| `orx project view ` | Overview of one project: details (incl. public/private), its experiment tree, and its reports — a single-screen jumping-off point. Works on any public project or any private one in your orgs. | -| `orx experiments ` | Print the project's experiments as an indented tree (nested by parent). **Experiment ids come from here.** | -| `orx runs [--experiment ]` | List runs as a table (status, experiment, commit, updated), newest first. `--experiment` filters to one experiment. **Run ids come from here.** | -| `orx env ` | List the **names** of the environment variables a run in this project will see (merged org + project vars plus your own per-user vars), each tagged with its source. **Names only — values are never returned.** Use this to check whether a secret a run needs (e.g. `WANDB_API_KEY`) is set, without ever seeing its value. | - -### Run evidence (run-scoped) +| `orx projects [--all] [--json]` | List your projects (id + name + GitHub `owner/repo`), grouped by org. `--all` includes archived; `--json` emits a flat array (incl. each project's `paperId`) for scripts. **Project ids, org ids, and the repo to clone come from here.** | +| `orx explore [--json]` | List the **public** project directory (id + name + repo) — projects anyone can view. Drill in with `orx project view` / `orx experiments` / `orx runs`. | +| `orx project view ` | Overview of one project: details, its experiment tree, and its reports. Works on any public project or any private one in your orgs. | +| `orx experiments ` | Print the project's experiments as an indented tree. **Experiment ids come from here.** | +| `orx runs [--experiment ]` | List runs as a table, newest first. **Run ids come from here.** | +| `orx env ` | List the **names** of the env vars a run will see (merged org + project + per-user), each tagged with its source. **Names only — values never returned.** | + +### Run evidence (run-scoped) — module `orx-evidence` | Command | What it does | |---|---| -| `orx logs [--head] [--bytes ] [--range :]` | Read a run's terminal log. See below. | -| `orx search-logs "" (--run \| --experiment ) [--max ]` | Grep run logs for a literal pattern. See below. | -| `orx artifacts ` | List the text artifacts a run produced (key + size). | -| `orx artifact [--head] [--bytes ]` | Read a run's text artifact (tail by default). Also caches it for `orx query` SQL search. | -| `orx wandb ` | List the W&B runs linked to a run (with dashboard URLs). | -| `orx chart wandb --metric "" --run [:label] ...` | Render a W&B metric across runs to a PNG line chart. See below. | -| `orx query ""` | Run **one read-only DuckDB SQL statement** against the project's evidence schema. See below. | - -### Create and run experiments (write) +| `orx logs [--head] [--bytes ] [--range :]` | Read a run's terminal log. | +| `orx search-logs "" (--run \| --experiment ) [--max ]` | Grep run logs for a literal pattern. | +| `orx artifacts ` / `orx artifact [--head] [--bytes ]` | List / read a run's text artifacts. | +| `orx wandb ` | List the W&B runs linked to a run. | +| `orx chart wandb --metric "" --run [:label] ...` | Render a W&B metric across runs to a PNG. | +| `orx query ""` | Run one read-only DuckDB SQL statement against the evidence schema. | + +### Create and run experiments (write) — modules `orx-create`, `orx-compute`, `orx-git` | Command | What it does | |---|---| -| `orx create-project --name "" [--repo ]` | Create a project bound to a GitHub repo (or a fresh blank repo when `--repo` is omitted). Starts with an **empty experiment tree** — create the baseline next. See below. | -| `orx project edit [--name ""] [--description "" \| --description-stdin]` | Edit a project's name and/or description (pass at least one). `--description-stdin` overwrites the description from stdin (long markdown). | -| `orx create-experiment --title "" [...]` | Add an experiment node; prints its git branch. See below. | -| `orx compute [--gpu ] [--count ] [--provider ] \| --cpu]` | List the GPU compute catalog across all providers, or CPU-only offers with `--cpu` (price-sorted). See below. | -| `orx instance create (--gpu [--count ] [--disk ] [--provider ] \| --cpu [--vcpus ])` | Spin up a standalone instance in an org (not tied to an experiment), like the dashboard's "Spin up". See below. | -| `orx exp status/cmd/run/cancel/wait ` | Inspect, run, cancel, and wait on a single experiment node. `status` prints the node's branch, its parent's branch, the latest run's full commit SHA, and a ready-to-paste local `git diff` recipe. See below. | -| `orx exp desc [--set "" \| --stdin]` | Read or overwrite the experiment's description (free-form notes). See below. | -| `orx report upload [--title ""]` | Upload a report folder (`report.md` + `images/`) to the project. Appears on the project page and its public view. `orx report list ` lists them; `orx report show ` prints a report's markdown to stdout (works on any public project); `orx report download ` writes `report.md` + referenced images back to a local folder (the inverse of `upload`). See below. | +| `orx create-project --name "" [--repo ]` | Create a project bound to a GitHub repo (or a fresh blank repo). | +| `orx project edit [--name ""] [--description "" \| --description-stdin]` | Edit a project's name and/or description (pass at least one); `--description-stdin` overwrites the description from stdin (long markdown). | +| `orx create-experiment --title "" [...]` | Add an experiment node; prints its git branch. | +| `orx compute [--gpu ] [--count ] [--provider ] \| --cpu]` | List the GPU/CPU compute catalog. | +| `orx instance create (--gpu … \| --cpu …)` | Spin up a standalone instance in an org. | +| `orx exp status/cmd/run/cancel/wait ` | Inspect, run, cancel, and wait on a single experiment node. | +| `orx exp desc [--set "" \| --stdin]` | Read or overwrite the experiment's description. | +| `orx report upload/list/show/download …` | Publish and read project reports (module `orx-reports`). | To **read or edit** a node's code — including diffing what a run changed — use -plain git in the cache-dir clone; there is no `orx` code command. See "Reading & -editing a node's code" below. +plain git in the cache-dir clone; there is no `orx` code command. See the +`orx-git` module. -### Literature & papers — alphaXiv (no login required) +### Literature & papers — alphaXiv (no login required) — module `orx-lit` | Command | What it does | |---|---| -| `orx lit "" [--limit ] [--json]` | Full-text search alphaXiv's paper corpus; returns ranked hits (id, title, date, votes, abstract). See below. | -| `orx paper ` | Fetch a paper's **machine-readable report** (structured LLM-oriented analysis). Prints the paper's linked **GitHub repo** at the top when one exists. See below. | -| `orx paper --full` | Fetch the paper's **full extracted text** instead — fallback when the report lacks a specific detail. | +| `orx lit "" [--limit ] [--json]` | Full-text search alphaXiv's paper corpus. | +| `orx paper [--full]` | Fetch a paper's machine-readable report (or full text with `--full`). | ### Meta | Command | What it does | |---|---| -| `orx skill [path]` | Print this overview (no args), or fetch a deeper skill/reference doc by path. | - -Project-scoped commands take a **project id**; experiment-scoped commands take an -**experiment id**; run-scoped commands take a **run id**. Don't mix them — get -ids from `orx projects`, `orx experiments`, and `orx runs` respectively. - -## The experiment-tree model — read this before driving a project - -A project is a **tree of experiment nodes**. The root (**baseline**) holds the -starting code and a **run command** — the single shell command that trains or -evaluates the node and writes an `EVAL.md` with its results. Every other node is a -**child** branched off a parent, inheriting its code and its run command. The two -rules this depends on — **never edit the baseline** and **the run command + env is -a fixed contract** — are the cardinal rules at the top of this skill; everything -below assumes them. - -### Shape the tree — stacked bushes, not a flat fan or a noodle - -The single most common way to drive a project badly is to get the **shape** wrong. -There are two opposite failures, and the right shape sits between them: - -``` -FLAT FAN (wrong) NOODLE (wrong) STACKED BUSHES (right) -root root root -├ a ├ b ├ c ... ├ n └ a └ lr-head ┐ round 1: - └ b ├ lr 2e-5 │ a small fan of - └ c └ lr 3e-5 ┘ co-equal options - └ d ... └ winner ── arch-head ┐ round 2 - ├ arch-A │ descends onto - └ arch-B ┘ round 1's winner -``` - -- **Flat fan** (your whole sweep hanging off the root): every result is measured - against the *start*, so wins never accumulate and the tree never makes progress. -- **Noodle** (a long single-child chain): depth manufactured for its own sake — - each step doesn't actually build on the one above it. -- **Stacked bushes** (correct): a *small fan within a round* (the options of one - decision), then **descend onto that round's winner** for the next round. - -**The one rule that produces this shape.** Before you make X a child of Y, name -what Y established that X builds on: - -- **You can name it** ("Y is the LR winner; X keeps that LR and changes the - architecture") → real depth. X is a **child** of Y. Descend. -- **You can't — X and Y are co-equal options you're trying at the same time** - (lr 2e-5 vs lr 3e-5) → they don't build on each other. They're **siblings** in - the same bush. Fan, don't chain. - -So: **width = the open options of one decision** (fan freely — a 3-way LR sweep -*should* be three siblings under a common head); **depth = decisions already -resolved, stacked** (one level down per winner kept). A new *round* never hangs off -the root — it hangs off the previous round's winner. That keeps the tree moving -**downward** as research progresses, without stringing unrelated nodes into a line. - -Re-read `orx experiments` each round and check the shape: a wide row of direct -children off the root with no grandchildren means you're fanning when you should be -descending; a long depth-N chain with no branching means you're chaining co-equal -variants that should have been siblings. - -### The auto-research loop - -To drive a project toward a goal (e.g. "best convergence for d=8") under a fixed -GPU budget, this is the intended flow — do **not** edit the baseline or rewrite the -run command: - -1. **Read the baseline's code.** Clone the project's repo into the cache dir and - read it with your normal tools (see "Reading & editing a node's code" for the path). - See its run command with `orx exp cmd ` and find where the knobs live - (config files, hyperparameters, model defs). -2. **Form one round's worth of hypotheses** — the co-equal options of a *single* - decision (which LR? which schedule? which init?), each a concrete change you can - make and measure against the others in this round. Don't mix decisions from - different rounds into one batch — that's what produces the flat fan. -3. **Create the round as a bush, and pick its parent deliberately.** All of this - round's options are **siblings under one parent** — the title is the idea, the - description is the concrete change you'll make on that node's branch. The parent is: - - the **baseline**, only for the very first round (nothing has been won yet); or - - the **previous round's confirmed winner**, for every round after — so this - round's changes build *on top of* the last gain instead of resetting to the - start. This is what walks the tree downward (see "Shape the tree" above). - - ```sh - # Round 1 — one decision (the LR), its options fanned off the baseline: - orx create-experiment --parent --title "LR 2e-5" \ - --description "Set the LR in config.yaml to 2e-5; change nothing else." - orx create-experiment --parent --title "LR 3e-5" \ - --description "Set the LR in config.yaml to 3e-5; change nothing else." - - # Round 2 — LR 3e-5 won → the next decision (architecture) descends onto it: - orx create-experiment --parent --title "Wider MLP" \ - --description "On top of the LR-3e-5 winner, widen the MLP hidden dim 1024→2048 in model.py." - ``` - The child inherits its parent's run command automatically — you don't set it, - and you never give siblings different commands or env vars (cardinal rule 2). -4. **Implement each child's change on its git branch** — `orx create-experiment` - prints the child's branch (`orx/`); sync the project's clone (in the - openresearch cache dir — see "Reading & editing a node's code"), check the branch out, - edit only the files that idea touches, commit, and push. **Leave the run - command alone:** - ```sh - DIR=~/.cache/openresearch/repos// # owner/repo from `orx projects` - [ -d "$DIR" ] || git clone https://github.com// "$DIR" - git -C "$DIR" fetch origin && git -C "$DIR" checkout -B orx/ origin/orx/ - # …edit config.yaml under "$DIR": schedule: constant → cosine … - git -C "$DIR" commit -am "cosine LR + warmup" && git -C "$DIR" push - ``` - While you're in the code, **make the run emit the evidence you'll need to judge - it.** Have it write rollout transcripts, per-sample eval breakdowns, generated - text, or summary tables to `.openresearch/artifacts/` (as text — JSONL/CSV/md) — - a directory at the **repo root**, where the run command's working dir points, so - `.openresearch/artifacts/foo.jsonl` is a plain relative path (if your code `cd`s - into a subdir or writes to `/tmp` first, resolve it from the repo root instead). - Those files sync to storage and become readable in step 7 via `orx artifacts` / - `orx artifact`. A run you can only read the tail-log of is far weaker evidence - than one that dumped its rollouts. See "Run artifacts" below. -5. **Launch up to your GPU budget** — one run per ready child, in parallel: - ```sh - orx exp run --gpu H100_SXM --count 1 - ``` -6. **Keep the budget saturated — drive a per-completion loop, not a wait-for-all - barrier.** With a cap of N concurrent runs, you want control back the moment - *any one* run finishes so you can analyze the state of experiments and either refill its slot or stop if no experiment further is needed — not after the whole batch - drains. `orx exp wait --project ` is built for exactly this: it - returns on the **first** completion. Treat it as one **tick** of a loop, where - *you* are the loop body: - - ``` - # after launching your runs, loop until the project is drained: - loop: - orx exp wait --project # sleeps; returns on the first completion - orx runs # SOURCE OF TRUTH: re-read all run states - # for each run now terminal that you haven't handled yet: - # - read its results (step 7) and decide: launch a refill? promote it? stop? - # - launch the next queued child to refill the freed slot (step 5) - # if `exp wait` printed "drained: no runs in flight" → batch is done, break - ``` - - Three things make this robust — follow all of them: - - **`exp wait --project` is a sleep-until-change signal, not the source of - truth.** It only reports completions it observed *during that one call*. A - run that finishes while you're analyzing the previous one is already terminal - by the next call and **won't be reported**. So on every wake, re-read - `orx runs ` and reconcile against the set of runs you've already - handled — act on *every* newly-terminal run, not just the line `exp wait` - printed. (This is the one time you do look at `orx runs` in a loop — as the - reconcile after each wake, **not** as a tight poll in place of `exp wait`.) - - **Re-issue `exp wait` each tick.** One completion → one return → you decide → - you call it again. Don't expect a single `exp wait` to block until everything - is done; that's the failure mode this loop avoids. - - **Terminate on drained.** When no runs are in flight, `exp wait --project` - returns immediately printing `drained: no runs in flight`. That — or seeing - every run terminal in `orx runs` with no more children to launch — is your - exit condition. Don't keep calling it into a timeout. -7. **Analyze each finish as it lands, then iterate.** Do the per-completion read - *inside the loop above*, not deferred to the end — when a run finishes, - **actually read its results** before deciding: `orx artifact EVAL.md`, - `orx chart wandb …`, `orx query …`. To see exactly what a finished node - changed, use the local git diff recipe `orx exp status ` prints (see - "Code diffs — local git"). Don't infer from status alone. Each - completion is a decision point with three moves: - - **Refill** — result is mediocre or inconclusive: launch the next queued child to - keep the GPU budget saturated (step 5). - - **Promote** — result is a clear win: this node becomes the **parent for the next - round**. The next batch of children branch off *it*, not the baseline, so the win - carries forward and the next ideas stack on top of it. This is the move that makes - the tree grow deeper; skipping it is what produces a flat, sweep-only tree. - - **Stop** — goal met, or the branch is exhausted. - - The baseline stays untouched throughout — promotion moves the *focal parent* down the - tree, it never edits the root. - -Stop when the goal is met, or after ~3 consecutive failed or regressed runs. -When you stop, consider writing up the tree as a local markdown report — -fetch `orx skill report` for the folder layout and section structure. Once -written, publish it with `orx report upload ` so it -shows up on the project page (and its public view) with images inline. The -folder should hold `report.md` plus an `images/` subfolder; the markdown -references images by relative path (`![](images/foo.png)`). - -## `orx create-project` — start a new project - -Creates a project in an organization (org ids are printed next to the org names -in `orx projects`), bound to a git repo. The project starts with an **empty -experiment tree** — no baseline yet. Every project is backed by exactly one git -repo; `--repo` picks where that repo comes from: - -```sh -# From an existing repo — yours (bound directly) or any readable repo -# (copied into a fresh repo the platform can write to): -orx create-project --name "NanoGPT speedrun" --repo karpathy/nanoGPT - -# From scratch — a fresh blank repo (just a stub root commit on main): -orx create-project --name "My new idea" -``` - -- `--repo` takes `owner/repo` or a github.com URL. `--branch` (only with - `--repo`) binds a non-default branch — the baseline will branch off it. - `--description` is optional. -- The command prints the project id + repo. **Next step: create the baseline** - (the root node, the control every variant is measured against): - `orx create-experiment --title "Baseline"` (no `--parent`). - Run it once for reference numbers, then hang children off it with - `orx create-experiment --title "" --parent `. -- For a **blank** project the baseline you create starts empty (a stub README): - seed it from existing code before launching runs — for a paper or a known idea - there is almost always a repo that implements it, and starting from that is - faster and a far better control than code written from scratch. See "Seeding - an empty baseline" below. - -## Seeding an empty baseline — start from existing code, not from scratch - -On a **blank** project, the baseline you create (`orx create-experiment`, no -`--parent`) starts as an empty stub (just a `README.md`) — there's no code to -run yet. The right move is almost never to **write the implementation by -hand.** For nearly any paper or idea there is already a repo that implements it, -and seeding the baseline from that repo is faster, more faithful, and a far -better control than something typed from memory. Reproductions should start from -the authors' (or a strong community) implementation, not a blank file. - -This is the one legitimate time you put code *on the baseline itself* (cardinal -rule 1's only exception): it applies **only while the baseline is still the empty -stub.** Once it holds real code, the baseline is frozen — vary code on children -from then on. - -**Find the code to seed from, in priority order:** - -1. **The paper's own repo.** If the project has a paper (`orx project view` shows - it, or you were given an arXiv id), run `orx paper ` — when alphaXiv has a - repo linked it prints `GitHub: ` as the first line. That repo is the - default seed. (Sanity-check the name: the linked repo is the most-starred one - and is occasionally a big framework rather than the paper's own code.) -2. **No repo line, or the wrong repo? Search for one** — a missing `GitHub:` line - means alphaXiv didn't have one indexed, *not* that none exists. Before falling - back to scratch: - - skim the paper's full text for a code/project URL: `orx paper --full` - (authors often link a repo or project page in the body or a footnote); - - search the literature for the method and check related papers' repos: - `orx lit ""` → `orx paper ` on the strongest hits and read - their `GitHub:` lines. -3. **No paper at all (a free-form idea project)?** Treat the idea as the query: - `orx lit ""`, read the most relevant report with `orx paper`, and - seed from the best implementation it points to. Only if a genuine search turns - up nothing usable do you start from a minimal scaffold of your own — and say so - in the baseline's description. - -**Seed the baseline branch from the chosen repo.** Work in the cache-dir clone -(see "Reading & editing a node's code"); replace the stub with the source's code -as one squashed commit, so the baseline keeps clean provenance and stays rooted -on the project repo: - -```sh -DIR=~/.cache/openresearch/repos// # the PROJECT's repo, from `orx projects` -[ -d "$DIR" ] || git clone https://github.com// "$DIR" -git -C "$DIR" fetch origin -git -C "$DIR" checkout -B orx/ origin/orx/ # the baseline's branch - -src=$(mktemp -d) && git clone --depth 1 https://github.com// "$src" -SHA=$(git -C "$src" rev-parse --short HEAD) && rm -rf "$src/.git" -find "$DIR" -mindepth 1 -maxdepth 1 ! -name .git -exec rm -rf {} + # drop the stub -cp -a "$src/." "$DIR/" # lay down the source tree -git -C "$DIR" add -A -git -C "$DIR" commit -m "Seed baseline from /@$SHA" -git -C "$DIR" push -``` - -Then make the baseline runnable and proceed normally: - -- read the seeded code, find its entry point, and set the run command **once**: - `orx exp cmd --set "bash run.sh"` (rule 2's one legitimate `--set`); -- run the baseline first for a control `EVAL.md`, then branch children and vary - code per the auto-research loop. The baseline is **frozen** the moment it holds - real code — shrink to the smallest config that still shows the paper's claim by - editing a **child**, never by trimming the root. +| `orx skill [name]` | Print this overview + the live module index (no args), or print one module / fetch a deeper reference doc by name. | -## `orx create-experiment` — add a node to the tree +## Modules -Adds a node to the experiment tree. `--title` is always required. The node shape -is chosen by flags: - -```sh -# Child node, branched off an existing experiment: -orx create-experiment --title "Larger batch size" --parent - -# Baseline (root) node on the project's bound repo: -orx create-experiment --title "Baseline" - -# Additional baseline (another root) when the project already has one: -orx create-experiment --title "Baseline v2" --baseline -``` - -- `--parent` selects the shape: with `--parent` ⇒ child; without it, on an - **empty project**, ⇒ the baseline (root) on the repo the project is already - bound to. Once a root exists, a parentless create attaches under the oldest - root on local projects (server projects create another baseline); pass - `--baseline` to explicitly add another root — projects may hold multiple - baselines, each the control for its own subtree. The repo a project works - on is chosen when the **project** is created (`orx create-project` or the - web), so there is no `--repo` flag here. -- **A `--parent` child inherits the parent's run command** (and branches off its - code). You do **not** set a run command on the child — keep it and vary the code - on the child's git branch (see "the experiment-tree model" above). -- **Choose the parent to keep the tree descending, not the root.** Before you pass - `--parent`, name what that parent established that this node builds on. The root - is the right parent only for the *first* round; every later round's siblings hang - off the **previous round's winner** (`orx experiments` shows the current shape). - Piling round after round of children onto the root is the flat-fan failure (see - "Shape the tree"). Co-equal options of the same decision are siblings under one - parent — don't chain them into a line either. -- `--description` is optional but **strongly recommended for children**: use it to - record the hypothesis / the concrete change this node makes. It's the node's - free-form notes field (the same one `orx exp desc` reads/writes), and it's how - you and the analysis tools tell sibling variants apart. - -## Running an experiment — `orx exp` + `orx compute` - -Each experiment node has a **run command** (the shell command that trains/evaluates -it) and is launched on **compute** you choose at run time. Compute is *not* stored -on the node — you pick a GPU, a CPU-only instance, or an existing sandbox each time -you launch (local projects may instead carry a default target — see "The default -compute target" below). - -```sh -orx exp status # status, branch, parent, run command, latest run + commit, local diff recipe -orx exp cmd # print the current run command -orx exp cmd --set "bash run.sh" # set it ONCE on the baseline; children inherit it -orx compute # browse GPU offers across all providers (price-sorted) -orx compute --gpu H100_SXM --count 1 # filter by gpu / count -orx compute --provider vast # filter by provider -orx compute --cpu # browse CPU-only offers (price-sorted) -orx exp run --gpu H100_SXM --count 1 [--disk 100] # launch on a NEW GPU instance (RunPod) -orx exp run --gpu H100_SXM --provider vast # launch on another provider's GPU -orx exp run --cpu cpu5c --vcpus 8 # launch on a NEW CPU-only instance -orx exp run --sandbox # launch on an EXISTING node -orx exp cancel # cancel the in-flight run -``` - -Rules and notes: -- **The run command is a fixed contract — set it once on the baseline, then leave - it alone.** Children inherit it (see "the experiment-tree model" above). Don't - `--set` a different command per child, and don't bake swept hyperparameters into - it — vary the **code/config** on a child's git branch instead, so every variant - runs the same command and their `EVAL.md`s stay comparable. The normal reason to - touch a command is the baseline having none yet. -- **Set a run command before launching.** `orx exp run` fails with a pointer to - `orx exp cmd --set` if the node has none. -- **Push your edits before launching.** A run trains the branch's tip **as it is - on GitHub** — so commit and push first (see "Reading & editing a node's code"). As a - safety net, `orx exp run` refuses a child whose branch has **no changes over its - parent** (the tell-tale of "queued before pushing") — push and retry, or pass - `--force` to run the unchanged code deliberately. -- **Pick compute with exactly one of `--gpu`, `--cpu`, or `--sandbox`.** With - `--gpu`, `--count` defaults to `1` and `--disk` to `100` (GB). A new GPU - instance defaults to **RunPod** (the cheapest matching RunPod offer for the - chosen gpu/count); pass `--provider ` to launch from another provider - shown in `orx compute` (e.g. `vast`, `lambda`). New CPU instances are - RunPod-only. Browse valid gpu ids, providers, and prices with `orx compute`. -- **GPU ids are exact enum strings, not family names.** `--gpu H100` is invalid — - the variant suffix is part of the id (`H100_SXM`, `H100_PCIE`, `A100_SXM_80GB`, - `RTX_4090`, …). Use the exact `GPU` column value from `orx compute`; run it - first if unsure. -- **Use `--cpu` for GPU-less work** (data prep, eval harnesses, CPU-bound papers). - The flavor is one of `cpu5c` (compute), `cpu5g` (general), or `cpu5m` (memory); - `--vcpus` is one of `2`, `8`, `32` (default `8`). Browse offers with - `orx compute --cpu`. CPU instances size their disk from the vCPU count, so there - is no `--disk` knob. -- `orx exp run` **queues** the run and returns immediately — it does not wait. - Follow progress with `orx runs ` and `orx logs `, or block - with `orx exp wait` (below). - -### The default compute target (local projects) - -The user may set a **default compute target** in the `orx up` dashboard -(Settings → Compute → Make default); it is machine-wide and applies to every -local project. When one is set, `orx exp run ` with no `--backend` -launches there, with the saved default flavor — omitting the flag is how you -use it (flavor-required backends still need `--flavor` if no default flavor -is saved). When none is set, local launches require an explicit `--backend`; the -compute choice is the user's, so ask. Server projects are unaffected: managed -compute (`--gpu`/`--cpu`/`--sandbox`) stays their default. - -### Running on Hugging Face Jobs — `--backend hf` - -**Managed compute (`--gpu`/`--cpu`/`--sandbox`) is the default. Use -`--backend hf` ONLY when the user explicitly asks for Hugging Face Jobs** -(e.g. "run this on HF", "use my huggingface account"), it is the configured -default target (orx up Settings → Compute), or the project context says to -prefer it. A connected HF token by itself is NOT a signal to switch — it just -means the option exists. When in doubt, launch on managed compute. - -With `--backend hf`, the job runs on the user's own HF account (requires -`HF_TOKEN` in the environment — orgs that connect their HF account in compute -settings get it synced automatically) and is billed there per minute; no -OpenResearch balance is spent. - -```sh -orx exp run --backend hf --flavor a10g-small # one GPU job -orx exp run --backend hf --flavor a100-large --timeout 8h -orx exp run --backend hf --flavor cpu-upgrade --image python:3.12 -``` - -Rules and notes: -- **`--flavor` is required** and replaces `--gpu`/`--cpu`/`--sandbox`. Common - flavors: `t4-small`, `a10g-small`, `a10g-large`, `l4x1`, `l40sx1`, - `a100-large`, `h200` (and `x2/x4/x8` multiples); CPU: `cpu-basic`, - `cpu-upgrade`. Prefer the smallest flavor that fits — same sizing discipline - as managed GPUs. -- **Set `--timeout` to cover the whole run** (default `4h`). HF kills the job - at the timeout; a killed job reads as a failed run. -- The job clones the experiment branch's **GitHub tip** and runs the fixed run - command, same contract as managed runs — commit and push first. Private - repos work automatically: the platform mints a repo-scoped clone token from - the project's connected GitHub app and passes it to the job as a secret. - Never ask the user to provision a `GITHUB_TOKEN`; setting one (env or - project env var) is only an override for repos outside the connected app. -- `--image` overrides the container (default: a CUDA pytorch image on GPU - flavors, `python:3.12` on cpu flavors). Pick an image with your deps baked - in when pip-install time dominates the run. -- Everything downstream is identical: the run appears in the tree, `orx exp - wait` / `orx runs` / `orx logs` work unchanged, and cancel from the web or - `orx exp cancel` reaches the job within a few seconds. A detached - `orx supervise` process mirrors status and logs; don't kill it. - -### Running on Modal — `--backend modal` - -**Same rule as HF: managed compute is the default. Use `--backend modal` ONLY -when the user explicitly asks for Modal** ("run this on Modal", "use my Modal -account") or it is the configured default target. Modal runs on the user's -own Modal account, billed there per second; no OpenResearch balance is spent. -It runs the job in a Modal **Sandbox** (an ephemeral container that scales to -zero when the run ends). - -orx auto-provisions a managed `modal` environment on the first Modal launch (no -pip-install needed). You only need a Modal token — `MODAL_TOKEN_ID` + -`MODAL_TOKEN_SECRET` in the environment (set them as org or project env vars and -they sync to the box automatically), or `modal token new`. - -```sh -orx exp run --backend modal --flavor a10g # one GPU sandbox -orx exp run --backend modal --flavor a100-80gb --timeout 8h -orx exp run --backend modal --flavor h100:2 # 2× H100 -orx exp run --backend modal --flavor cpu --image python:3.12 -``` - -Rules and notes: -- **`--flavor` is required** and replaces `--gpu`/`--cpu`/`--sandbox`. It's a - Modal GPU: `t4`, `l4`, `a10g`, `a100`, `a100-80gb`, `l40s`, `h100`, `h200` - (append `:N` for a count, e.g. `h100:2`); or `cpu` / `cpu-large` for CPU-only. - Prefer the smallest flavor that fits. -- **Set `--timeout` to cover the whole run** (default `4h`). Modal kills the - sandbox at the timeout; a killed sandbox reads as a failed run. -- Same clone contract as HF/managed: the sandbox clones the experiment branch's - **GitHub tip** and runs the fixed command — commit and push first. Private - repos work automatically via the platform's repo-scoped clone token. -- `--image` overrides the container (default: a CUDA pytorch image on GPU - flavors, `python:3.12` on cpu). Pick one with your deps baked in when - pip-install time dominates. -- Everything downstream is identical (`orx exp wait` / `orx runs` / `orx logs`, - cancel from web or `orx exp cancel`). A detached `orx supervise` mirrors - status and logs; don't kill it. - -### Running on your Kubernetes cluster — `--backend k8s` - -**Same rule: use `--backend k8s` ONLY when the user explicitly asks to run on -their cluster** ("run this on k8s", "use our cluster") or it is the -configured default target. Local projects (`orx up`) only for now. Auth -comes from the local kubeconfig — orx never stores cluster credentials; the -context/namespace live in `orx up` Settings → Compute. - -**There are no flavors: the run's shape is a Kubernetes manifest you commit -on the experiment branch** (default `.orx/k8s.yaml`, or `--manifest `). -Inspect the cluster yourself (`kubectl get nodes`) and write whatever the run -needs — a single-pod GPU Job, an Indexed Job spanning nodes with a headless -Service, an auxiliary inference Deployment. The manifest inherits through the -experiment tree like all code; changing it is a commit, visible in the diff. - -```sh -orx exp run --backend k8s # runs .orx/k8s.yaml from the branch tip -orx exp run --backend k8s --manifest infra/run.yaml --timeout 8h -``` - -A minimal manifest: - -```yaml -apiVersion: batch/v1 -kind: Job -metadata: - name: train-{{ORX_RUN}} -spec: - template: - spec: - restartPolicy: Never - containers: - - name: run - image: pytorch/pytorch:2.6.0-cuda12.4-cudnn9-runtime - command: ["bash", "-c", "$ORX_SCRIPT"] - resources: - requests: { nvidia.com/gpu: "4", cpu: "32", memory: "128Gi" } - limits: { nvidia.com/gpu: "4" } -``` - -The contract orx enforces at submit (loud, before anything runs): -- **Exactly one Job** — its completion/failure is the run's outcome. With - several Jobs, label the primary `orx-primary: "true"`. Other resources - (Services, Deployments, ConfigMaps) ride along; cancel deletes exactly what - the manifest created. -- **Some container of that Job must run `$ORX_SCRIPT`** — the env var orx - injects with the clone-and-run script (branch tip + the experiment's fixed - run command). The manifest shapes *where* the command runs, never *what* - runs. -- Every resource needs `metadata.name` (no `generateName`) and no foreign - `metadata.namespace`. Use `{{ORX_RUN}}` in names — orx substitutes a - run-unique token so re-runs don't collide. -- orx injects run labels, the `orx-env` Secret (synced env + `HF_TOKEN` / - `GITHUB_TOKEN`) on the primary Job's containers, and defaults for - `activeDeadlineSeconds` (from `--timeout`, default 4h; a manifest-set value - wins), `ttlSecondsAfterFinished`, and `backoffLimit: 0`. Auxiliary - resources that need the env reference the `orx-env` Secret themselves. -- The run log follows the primary Job's **leader pod** (completion index 0 - for Indexed Jobs, else its sole pod) — make it print the evidence; other - pods stay reachable via `kubectl logs`. -- Everything downstream is identical (`orx exp wait` / `orx runs` / - `orx logs`, cancel via `orx exp cancel`). A detached `orx supervise` - watches the Job via kubectl; don't kill it. - -### Running on your own box — `--backend ssh` - -**Same rule: use `--backend ssh` ONLY when the user explicitly asks to run on -their own machine/server** ("run this on my box", "use my GPU server") or it -is the configured default target (`--host ` is still required per -launch). Local projects (`orx up`) only for now. It runs the experiment as a -detached background process on a host from your `~/.ssh/config`, over `ssh` — -no scheduler, no container, the host's own environment. - -```sh -orx exp run --backend ssh --host my-gpu-box # ~/.ssh/config alias -``` - -Rules and notes: -- **`--host` is the ssh host alias** (from `~/.ssh/config`) — a machine, not a - hardware shape, so there is no `--flavor` here. See `orx up` Settings → - Compute → SSH (each host has a "Test" button that checks reachability + git). -- Auth is your ssh keys/agent — orx never reads a key, it just shells out to - `ssh `. The host needs `git` and `bash`; it clones the experiment - branch's GitHub tip (private repos via the `GITHUB_TOKEN` passed in the run's - env) and runs the fixed command. Commit and push first, same as the others. -- No `--image` (the host's environment is used as-is) and no `--timeout` (the - process runs until it exits or you cancel). -- The run lives under `~/.orx/runs//` on the host (`run.sh`, `log`, - `pid`, `exit_code`). Cancel from the web or `orx exp cancel` kills the remote - process group. Everything downstream (`orx exp wait` / `runs` / `logs`) is - identical; a detached `orx supervise` polls it over ssh — don't kill it. - -### Running on this machine — `--backend local` - -**Same rule: use `--backend local` ONLY when the user explicitly asks to run -locally** ("run it on this machine", "just run it here") or it is the -configured default target. Local projects (`orx up`) only. It runs the -experiment as a detached background process on the machine running orx — no -scheduler, no container, this machine's own environment. - -```sh -orx exp run --backend local -``` +The detail lives in focused modules — load one with `orx skill ` (the live +list, with one-line descriptions, is printed at the end of `orx skill` output): -Rules and notes: -- **Nothing to pick** — no `--flavor`, `--host`, `--image`, or `--timeout` - (the process runs until it exits or you cancel). The hardware is whatever - this machine has; prefer it for small or CPU-scale runs and use a remote - backend for anything heavy — it shares CPU/RAM/GPU with everything else on - the machine. -- Same clone contract as every backend: the run clones the experiment - branch's GitHub tip into its own run dir (never your checkout) and runs the - fixed command — commit and push first. Never run training directly in your - shell instead: that would be unsupervised and invisible to the dashboard. -- The run lives under `/local-runs//` (`run.sh`, `log`, - `pid`, `exit_code`). Cancel from the web or `orx exp cancel` TERMs the - process group. Everything downstream (`orx exp wait` / `runs` / `logs`) is - identical; a detached `orx supervise` watches it — don't kill it. - -## Spinning up a standalone instance — `orx instance create` - -Provision a persistent instance in an **organization**, not tied to any -experiment — the CLI equivalent of the dashboard's org "Spin up" panel. Use this -for ad-hoc/manual compute (you SSH in yourself); experiment runs use `orx exp run` -instead. - -```sh -orx instance create --gpu H100_SXM --count 1 [--disk 100] # GPU box (cheapest provider) -orx instance create --gpu H100_SXM --provider runpod # pin a provider -orx instance create --cpu cpu5g --vcpus 8 # CPU-only box -``` - -- `` comes from `orx projects` (the `org:` line). The flags mirror - `orx exp run`: exactly one of `--gpu` or `--cpu`; `--count`/`--disk` apply to - `--gpu`, `--vcpus` to `--cpu`. -- Unlike `orx exp run`, omitting `--provider` picks the **cheapest** matching - offer across all providers; pass `--provider ` to pin one. -- The box provisions asynchronously — the command prints its id and current - status; its SSH host appears once it's online. - -## Waiting on runs — `orx exp wait` - -Block until a run changes state — useful when driving a research loop and you want -to act as soon as a run finishes. Two modes, picked by argument: - -```sh -orx exp wait # level trigger: poll this experiment's latest run - # until it reaches a terminal state (done/failed/cancelled) -orx exp wait --project # edge trigger: return when the FIRST run in the - # project COMPLETES (transitions into done/failed/cancelled) -orx exp wait --interval 10 --timeout 3600 # tune polling -``` - -- Pass **exactly one** of `` or `--project` (not both, not neither). -- `--project` is the **budget-loop** primitive: it wakes only on a **completion** - (a run reaching `done`/`failed`/`cancelled`) — i.e. a freed slot. Run *starts*, - new queued runs, and `queued→running` transitions are intentionally ignored, so - it won't wake you on non-events. It returns on the **first** completion — call - it in a loop, one return per tick, and you (not the CLI) decide what to do with - each freed slot. See the per-completion loop under "the experiment-tree model". -- **It's a sleep-until-change signal, not the source of truth.** It reports only - completions it saw *during that one call*; a run that finishes between calls is - already terminal next time and won't be reported. On every return, re-read - `orx runs ` and act on *all* newly-terminal runs — don't treat the - printed line as the complete set, and don't replace `exp wait` with a tight - `orx runs` poll either (use `exp wait` to sleep, `orx runs` to reconcile). -- Call `--project` **while runs are in flight** (right after launching). If every - run is already terminal, there's nothing left to complete, so it returns - immediately printing `drained: no runs in flight` (exit 0) — your loop's - termination signal. -- `--interval` is seconds between polls (default `5`); `--timeout` gives up after - N seconds (default `1800`) and exits **non-zero** so callers can branch on it. - For long training runs, raise `--timeout` (or treat a timeout exit as "nothing - changed yet, loop again") so a wait that simply outlasts the interval isn't - mistaken for an error. -- Progress lines go to **stderr**; the final completion line(s) go to **stdout**, - each as ` -> ` (or ` (new)`), or the - single line `drained: no runs in flight` when nothing was in flight. -- **When a run is `failed`, a `reason:` line follows it.** Compute spin-up - failures (no GPU capacity, provider quota/limit hit, transient provider error) - carry the provider's own message here — the same text the website shows as a - toast. These are usually **transient and retryable**: wait and re-launch the - same run, or pick a different GPU/provider, rather than treating the experiment - as a dead end. If the run instead failed *after* the box came up, the `reason:` - line points at `orx logs `, where the traceback/OOM/setup error lives. - The same `reason:` line appears under `orx exp status ` and beneath the - `orx runs ` table. - -## Experiment description / notes — `orx exp desc` - -Each experiment node carries a free-form **description** (markdown) — the same -field set by `create-experiment --description`. Use it for notes: observations, -hypotheses, or a running summary. It is a whole-document field: writing -overwrites whatever was there. - -```sh -orx exp desc # print the description to stdout (empty → hint on stderr) -orx exp desc --set "tried lr=3e-4, diverged at step 4k" # overwrite with a short note -cat notes.md | orx exp desc --stdin # overwrite from stdin (long markdown) -``` - -- **Read** prints the text to **stdout** (pipe/redirect-friendly); when empty, a - hint is printed to **stderr** and stdout stays empty. -- **Write** with exactly one of `--set` (inline) or `--stdin` (whole of stdin). - Passing both is an error. Writing **replaces** the entire description — to - append, read first, edit, and write back. -- `` comes from `orx experiments ` (the experiment id, not a run - or project id). - -## Reading & editing a node's code — plain git in the cache-dir clone - -Every experiment node **is a git branch** (`orx/`) on the project's GitHub -repo — `orx create-experiment` prints it. There is no dev box and no `orx` code -command: the **local clone in the cache dir is the standard way to interface -with code** — reading a node's files, diffing what a run changed, and editing — -all with plain git and your own tools. - -**Clone into the openresearch cache dir, not your cwd.** The canonical location, -keyed by repo so the same clone is reused across all of a project's experiments: - -``` -~/.cache/openresearch/repos// -``` - -`/` comes from `orx projects`. **Never** clone into your current -directory or the user's project folders — clones accreting in `~/projects` is the -failure mode this avoids. - -This is how you **realize a child's hypothesis**: after `create-experiment ---parent`, check out the child's branch and make the specific code/config edits -its description calls for — then commit, push, and run. Edit only the files that -idea touches, and **don't touch the run command** (it's inherited; see "the -experiment-tree model" above). Edit children, never the baseline. - -The sync recipe is **idempotent** — run it verbatim whether or not the clone -already exists from a previous session. Always fetch + reset before editing, so a -reused clone is never stale (and the experiment's branch, created server-side, is -fetched even when it's brand-new and not in your local clone yet): - -```sh -DIR=~/.cache/openresearch/repos// - -# Clone once (skips if it already exists), then ALWAYS sync before touching a branch: -[ -d "$DIR" ] || git clone https://github.com// "$DIR" -git -C "$DIR" fetch origin -git -C "$DIR" checkout -B orx/ origin/orx/ # create, or reset to origin if it exists - -# …edit files under "$DIR" with your normal tools… -git -C "$DIR" commit -am "tune lr" # one or more commits — your call -git -C "$DIR" push # push so runs and the tree see the change -``` - -Rules and notes: -- **Always sync first.** `git -C "$DIR" fetch origin && git -C "$DIR" checkout -B - orx/ origin/orx/` is mandatory every time — `-B …origin` creates the - branch or resets an existing local one to the GitHub tip, so a persistent clone - never edits a stale base. It discards uncommitted/unpushed local work on that - branch, which is exactly what you don't want to carry across sessions (the - contract is commit + push before moving on). -- **Auth is your own git.** Clone/push use whatever GitHub credentials your `git` - already has — the repo lives under your account or your org, so access is the - same as any of your repos. If a clone or push fails on auth, authenticate git - for github.com (e.g. `gh auth login` or an SSH key) and retry. -- **Push before you run.** `orx exp run` launches from the branch's pushed tip on - GitHub — uncommitted or unpushed edits won't be in the run. Commit and push - first. -- **Reading another node's code** without disturbing your checkout: that branch is - already in the clone after a fetch — `git -C "$DIR" show origin/orx/:`. - -### Code diffs — local git - -What did a run change vs. its parent experiment? `orx exp status ` prints -the parent's branch, the latest run's full commit SHA, and this exact recipe — -compute the diff locally in the same clone: - -```sh -DIR=~/.cache/openresearch/repos// # owner/repo from `orx projects` -[ -d "$DIR" ] || git clone https://github.com// "$DIR" # cold cache → clone first -git -C "$DIR" fetch origin # ALWAYS fetch first — the commit and parent tip live on GitHub -git -C "$DIR" diff origin/... -``` - -- The **three-dot** form diffs from the merge-base — what the run's branch - changed, not what the parent gained since the fork. That's the cumulative - "what this experiment did to the code" view. -- Fetch first is mandatory: the run's commit and the parent's tip exist on - GitHub and may not be in your clone yet. -- Root experiments have no parent — there is no diff base, by definition. - -## Reading & searching run logs — `orx logs` / `orx search-logs` - -A run's terminal output (the PTY stream) is captured live while it runs and -persisted afterwards. These two commands read and grep it the same way the -OpenResearch assistant's "Read run log" / "Search run log" tools do — byte-range -reads against the persisted log, the live buffer for an in-flight run. - -```sh -orx logs # tail (the end — usually what you want) -orx logs --head # read from the start instead -orx logs --bytes 200000 # raise the byte cap (default 64 KB, max 1 MB) -orx logs --range 4096:8192 # exact byte window [start, end) -``` - -- The log goes to **stdout** (pipe/redirect-friendly); a `[source] bytes a–b of N` - status line goes to **stderr**, noting if content was truncated above/below. -- `` comes from `orx runs ` (the run id, not the experiment id). - -```sh -# Grep a single run, or every run in an experiment: -orx search-logs "CUDA out of memory" --run -orx search-logs "Traceback" --experiment -orx search-logs "loss=nan" --experiment --max 5000 -``` - -- Search is **literal and case-sensitive**. One of `--run` / `--experiment` is required. -- Each hit prints as `:: ← :`. Feed those - byte offsets straight into `orx logs --range :` to pull the - surrounding context. Results are capped (raise with `--max`). -- **For training metrics, check W&B first.** If the run has a linked W&B run - (`orx wandb `), `orx chart wandb` is usually a better metrics read than - grepping the log (complete history, exact stats, visible trajectory). Logs - remain the right tool for debugging — tracebacks, OOMs, setup failures — and a - fine metrics fallback when W&B isn't linked or doesn't have the key you need. - -## Run artifacts — `orx artifacts` / `orx artifact` - -Beyond the terminal log, a run uploads **text artifacts** (eval outputs, reports, -generated files). - -**Where artifacts come from — you control this.** Every run has a -`.openresearch/artifacts/` directory **at the root of the repo** (the run command -executes with its working dir set to the repo root, so `.openresearch/artifacts/` -is a plain relative path from there — the same tree you cloned and edited in step -4; don't hardcode an absolute `$HOME`-based path). **Anything the run writes there -is synced to cloud storage (~every 10s, plus a final flush when the run ends) and -becomes readable later via `orx artifacts` / `orx artifact`.** Artifacts aren't -magic — they're whatever your experiment code chose to drop in that directory. -`EVAL.md` is just the conventional one (the run command writes it there); the same -mechanism is yours to use for anything you'll want to examine after the fact: -rollout transcripts, per-sample eval breakdowns, generated text, prompt/response -dumps, plots' underlying data, summary tables. When you implement a node's change -(step 4 of the loop), have the code save these to `.openresearch/artifacts/` — -that's how you turn a run into inspectable evidence instead of a one-shot log. - -- **Keep things you'll re-read as text.** The CLI read commands surface **text - artifacts** (JSON, JSONL, CSV, logs, markdown). Binary blobs — checkpoints, model - weights, `.npy`, images — still persist to storage, but you won't be able to read - them back through `orx artifact`, so dump a text-readable companion (e.g. a JSONL - of rollouts alongside the checkpoint) for anything you intend to analyze via CLI. - -```sh -orx artifacts # discover what a run uploaded (KEY + SIZE table) -orx artifact # read one artifact (tail by default) -orx artifact --head --bytes 200000 # from the start, raise the cap -``` - -- Start with `orx artifacts` to list keys, then `orx artifact ` to - read one. Reading an artifact also **caches it for `orx query`** so you can grep - artifact text via SQL. -- `orx artifact` content goes to **stdout**; byte-range and truncation metadata - go to **stderr**. -- For the run's **code diff**, use local git — see "Code diffs — local git" above. - -## Charting W&B metrics — `orx chart wandb` - -Renders a single W&B history metric across one or more linked runs as a PNG line -chart — the same renderer the OpenResearch assistant uses. The server fetches the -W&B history, draws the chart, and the CLI **downloads the PNG to a local file and -prints its path**. Because the output is an image, the intended flow is: run the -command, then **`Read` the printed PNG path to view the chart** with your vision. - -```sh -# One metric, two runs overlaid (label after a colon, optional): -orx chart wandb --metric "train/loss" \ - --run :experiment --run :baseline - -orx chart wandb --metric "train/reward" --run --smoothing 0.9 -orx chart wandb --metric "val/acc" --run --out ./charts -``` - -- `wandb` is a required first positional (the chart kind; only `wandb` is supported today). -- **`--metric`** is one W&B history key (e.g. `train/loss`). List available keys - first via `orx query "select distinct key from wandb_history_keys"`, - or find linked W&B runs with `orx wandb `. -- **`--run`** is repeatable — pass every run you want on the chart in one call - (up to 6). Append `:label` to set the legend label (defaults to the W&B run id). - The run id comes from `orx runs ` (the run id, not the experiment id). -- **`--smoothing`** is an EMA factor `0`–`0.99` (default `0.6`). If a chart looks - too noisy to read, re-run with a higher value (e.g. `0.9`) — don't switch metrics. -- **`--out`** sets the output directory (default `~/.cache/openresearch/charts/`). -- Per-run summary stats (`n`, `min`, `max`, `last`) print to **stdout** alongside - the file path, so you can cite exact numbers without opening the image. Runs - that produced no data are listed under `Skipped:`. -- Requires `WANDB_API_KEY` set in the project or org env vars; otherwise the - command reports that and exits non-zero. Run `orx env ` first to - confirm the key is present (it lists names only, never values). - -## Literature search & paper content — `orx lit` / `orx paper` - -These tap **alphaXiv's public corpus** (2.5M+ arXiv papers: CS, math, physics, -stats, q-bio/fin, EE — not PubMed/biomed). They need **no `orx login`** and hit -alphaXiv hosts, not the OpenResearch API. Use them to ground research in real -literature: find related work, pull a paper's structured report, and only drop to -its full text when you need a specific equation/table/section. - -```sh -orx lit "speculative decoding for LLMs" # ranked hits (id, title, date, votes, abstract) -orx lit "rotary position embeddings" --limit 10 # widen the result set (default 5) -orx lit "kv cache compression" --json # raw JSON for programmatic use -orx paper 2401.12345 # machine-readable report (the default) -orx paper https://arxiv.org/abs/2401.12345 # any arXiv/alphaXiv URL works too -orx paper 2401.12345v2 --full # full extracted text (fallback) -``` - -- **`orx lit`** prints, per hit: ` `, then `<date> · <votes> votes`, - then a truncated abstract. The **`paperId`** is what you feed to `orx paper`. - Results are relevance-ranked, capped at `--limit` (default 5). `--json` emits the - raw hit objects (incl. matched `snippets`) for piping. -- **`orx paper <id>`** writes the report markdown to **stdout** (pipe/redirect-friendly). - The id can be a bare id (`2401.12345`), a versioned id (`2401.12345v2`), or an - arXiv / alphaXiv URL — the CLI normalizes it. -- **The paper's code: `GitHub: <url>` line.** When alphaXiv has a GitHub repo linked - to the paper, `orx paper` prints it as the first line (with `--full` too). If the - report leaves you with questions about *how* something was actually implemented — - exact hyperparameters, training loop details, a trick the paper glosses over — - clone the repo into a temp dir and read the code: - - ```sh - dir=$(mktemp -d) && git clone --depth 1 <githubUrl> "$dir" - ``` - - Inspect it there (grep for the model/optimizer setup, read the configs), and rely - on it as the ground truth for reproducing the paper. No line means no repo is - linked. Note the linked repo is the most-starred one associated with the paper — - occasionally a big framework rather than the paper's own code; sanity-check the - repo name before leaning on it. -- **Report first, full text only when needed.** The default report is a compact - (~10 KB) structured analysis and is enough for most questions. Reach for `--full` - only when the report is missing a specific detail — it returns the entire paper. -- **404s are normal answers, not errors of yours.** A paper whose report hasn't been - generated yet exits non-zero with a hint to try `--full`; one with no extracted - text yet points you at the arXiv PDF. Try `--full`, then the PDF, before giving up. -- Override hosts with `ALPHAXIV_API_URL` (search) and `ALPHAXIV_WEB_URL` (paper docs) - if you ever need to point elsewhere. - -**Grounding a research loop in literature.** Before forming hypotheses for a project -(step 2 of the auto-research loop), search the literature for prior art on the knob -you're about to vary, pull the most relevant report, and let it inform the change you -write into a child's description: - -```sh -orx lit "learning rate warmup schedules transformers" --limit 5 -orx paper <bestPaperId> # read its report; cite the idea in the child's --description -``` - -## `orx query` — important - -The query runs against a **DuckDB "evidence" schema**, which is NOT the same -shape as the REST objects returned by `orx experiments` / `orx runs`. Don't -guess column names from what the other commands display — write queries against -the exact columns below. - -The two tables you'll hit first, with their **full** column lists: - -``` -experiments(id, project_id, parent_experiment_id, slug, title, description, - analysis, run_command, sandbox_id, updated_at) -runs(id, experiment_id, command, status, commit_sha, log_key, sandbox_id, - result_markdown, updated_at) -``` - -The guesses that look right but aren't: - -- **Experiments have no `status` — anywhere.** Status is a *run* property - (`runs.status`). To get "the experiment's status", join its runs: - ```sh - orx query <projectId> "select e.title, r.status, r.updated_at from experiments e left join runs r on r.experiment_id = e.id order by e.title, r.updated_at desc" - ``` -- The parent column is **`parent_experiment_id`**, not `parent_id`. -- There is **no `branch` column** — the git branch is derived from the slug - (`orx/<slug>`). - -There is also a unified **`entities` view** (projects, experiments, runs, and -sandboxes as one table) — handy for tree/graph questions: - -``` -entities(id, type, entity_id, entity_type, project_id, parent_id, parent_type, - parent_entity_id, parent_entity_type, title, name, slug, status, - description, analysis, run_command, updated_at) -``` - -Caveat: its `status` column is populated **only for run and sandbox rows** — -it's NULL for experiments (see above), so don't read it off experiment rows. -For an experiment row, `parent_id` is the parent experiment (or the project, -for the baseline). - -For any table not listed here, discover the real schema before writing queries: - -```sh -orx query <projectId> "select table_name, column_name from information_schema.columns order by 1, 2" -``` - -The full schema, table-by-table guidance, and worked examples live in the -canonical project-query skill — fetch it before doing anything non-trivial: - -```sh -orx skill project-query # the schema + workflow guide -orx skill project-query/references/runs-and-results # runs, metrics, results -orx skill project-query/references/run-diffs # code diffs per run -orx skill project-query/references/text-evidence # logs, artifacts, files -orx skill project-query/references/project-overview # high-level project shape -orx skill report # write a local markdown research report (with charts) -``` +- **orx-experiment-tree** — the experiment-tree model, the auto-research loop, and `orx exp desc`. +- **orx-create** — create a project, seed an empty baseline, add experiment nodes. +- **orx-compute** / **orx-compute-k8s** — launch runs on compute; the k8s manifest contract. +- **orx-git** — read, edit, and diff a node's code with plain git. +- **orx-evidence** — logs, search-logs, artifacts, W&B charts, and the `orx query` evidence DB. +- **orx-reports** — write and publish research reports. +- **orx-lit** — literature search and paper content. -These are the same skill docs the OpenResearch assistant reads. Run -`orx skill` with no path to re-print this overview; the list of fetchable skills -is also shown there when the API is reachable. +Deeper API-served references (the project-query schema and worked examples, the +report writing guide) are fetchable too — `orx skill` lists them at the end when +the API is reachable. ## Typical workflow @@ -1101,12 +150,11 @@ Orienting in a project (read-only discovery): ```sh orx projects # find the project id orx experiments <projectId> # see the tree, pick an experiment id -orx skill project-query # learn the evidence schema -orx query <projectId> "select title from experiments limit 10" +orx skill experiment-tree # the model + the auto-research loop orx runs <projectId> # find a run id orx logs <runId> # read its output ``` To actually **drive** a project toward a goal — branch children off the baseline, -edit each child's code on its git branch, and keep the GPU budget saturated — follow -the auto-research loop in "the experiment-tree model" above. +edit each child's code on its git branch, and keep the GPU budget saturated — +follow the auto-research loop in the `orx-experiment-tree` module. diff --git a/SYSTEM_PROMPT.md b/SYSTEM_PROMPT.md new file mode 100644 index 0000000..4d1cc09 --- /dev/null +++ b/SYSTEM_PROMPT.md @@ -0,0 +1,206 @@ +<!-- +This is the system prompt ("playbook") that `orx up` injects into every local +agent session, verbatim except for `{token}` substitution at render time +(project facts, the compute default, and the skills index — see +`playbook_md()` in src/local/opencode.rs). Each harness receives it through +its native channel: Claude Code via --append-system-prompt-file, Codex via +developerInstructions, OpenCode via the config `instructions` list. + +It carries only what must be in context every turn: identity, the cardinal +rules, session-collaboration rules, the command index, and the loop skeleton. +Everything topical lives in the native skills installed into the session +worktree from agent-skills/ — the prompt points at them instead of repeating +them. This leading comment is stripped at render time. +--> + +# OpenResearch local agent — {name} + +You are the OpenResearch research agent for the **local** project **{name}**, +running inside `orx up` on the user's own machine. Your working directory is +**your own git worktree** of the project's repo — private to this chat +session. Other chat sessions (other agents) work in sibling worktrees of the +same clone, sharing its branches and remotes. + +- Project id: `{id}` +- GitHub repo: `{repo}` +- Baseline branch: `{baseline}` +{paper_line}{compute_bullet} +- Files dir: `{files}` — every file in it shows up in the dashboard's + Files tab (reports, figures, CSVs), grouped by experiment + +## Start here + +Drive everything through the `orx` CLI. `orx` is the source of truth for the +experiment tree, runs, and logs — not the filesystem. This is **local mode**: +only the commands listed below exist; use this project id (`{id}`) for every +`orx` command that takes one. + +Orient with `orx projects` and `orx runs {id}`. + +**If the experiment tree is empty** (a fresh project), create the baseline +first: `orx create-experiment {id} --title "Baseline"` (no `--parent`). Give it +the run command, run it once for reference numbers, then branch children off it. + +## Skills + +Focused how-to guides are installed as **native skills for this session** — your +harness auto-loads them, and you can pull one up by name when a task calls for it: + +{skills_list} + +The cardinal rules, command index, and loop below are always in effect; the +skills carry the details (per-backend flags and sizing, the k8s manifest, tree +shaping, git recipes, log analysis, report layout). **Load the relevant skill +before acting in its area** — commands remembered from earlier in a long +session go stale; the skill is always current. If your harness hasn't surfaced +one, `orx skill <name>` prints it. + +## Working alongside other agents + +Several chat sessions may drive this project at once, each in its own worktree +of the same clone. Git state is shared between you: + +- **See their work before starting yours.** Local and remote branches are + shared across worktrees — `git branch -a` lists every experiment branch + (even unpushed ones), `orx runs {id}` shows what is running, and + `orx exp desc <expId>` holds each node's findings. Orient from these so you + extend the tree instead of duplicating a sibling's experiment. +- **Keep your notes current as you go.** Other agents orient from + `orx exp desc` — write findings there when you learn them, not only at the + end of a line of work. +- **One branch, one owner.** Git refuses to check out a branch that another + worktree already has checked out. If `git checkout <branch>` fails that + way, another agent owns that experiment — leave it alone and work on your + own node. +- Your worktree starts **detached on the baseline tip**; check out your + experiment's branch before editing. + +## Cardinal rules + +Breaking any of these silently invalidates results — they are not style +preferences. + +1. **Never edit a baseline (root experiment) once it exists.** A root is the + control its variants are measured against — on a fresh project you create + it (first `orx create-experiment`, no `--parent`), and from then on it is + frozen. To try an idea, **branch a child** + (`orx create-experiment … --parent <expId>`) and edit the child's branch. +2. **The run command and the environment are a fixed contract — identical on + every node.** Children inherit it verbatim. If the project has no run + command, set the default once with `orx project edit {id} --run-command + '<cmd>'` (or pass `--run-command` when creating the first experiment) — + children inherit it from then on. Never vary behavior through env vars or + env-prefixed commands. +3. **Vary code, not knobs-in-the-command.** Encode hyperparameters in committed + code/config and branch a child per variant. Every node runs the *same* + command over *different code*, so results stay comparable. +4. **Grow the tree downward, not sideways.** Fan a few siblings *within* a + round (the options of one decision), then **descend onto the winner** for + the next round. A root with a long flat row of children is the failure mode. +5. **Launch all compute via `orx exp run` — never `hf jobs`, `modal`, `kubectl`, raw `ssh`, or a training command in your own shell.** Direct jobs are unsupervised and invisible to the dashboard. + +## Command index (local mode) + +| Command | What it does | +|---|---| +| `orx projects` | List projects; local ones are tagged `(local)`. | +| `orx create-experiment {id} --title "<t>" [--description "<d>"] [--parent <expId> \| --baseline] [--run-command "<cmd>"]` | New node on its own `orx/<slug>` branch, pushed to GitHub — forked off the parent's tip, or off `{baseline}` for a root. Omit `--parent` to attach under the oldest root (or become the baseline on an empty project). | +| `orx project view {id}` / `orx project edit {id} --run-command "<cmd>"` | Inspect the project / set its default run command. | +| `orx exp status <expId>` | Node's branch, command, and latest run. | +| `orx exp desc <expId> [--set "<text>" \| --stdin]` | Read/overwrite the node's notes. Record findings here. | +| `orx exp run <expId> [--backend <hf\|modal\|k8s\|ssh\|slurm\|openresearch\|local>] [flags]` | Launch the node's run. Backend-specific flags, flavors, and sizing: **`orx-compute` skill** (k8s manifest: **`orx-compute-k8s`**). | +| `orx exp cancel <expId>` | Cancel the in-flight run. | +| `orx exp wait <expId> [--timeout <s>]` / `orx exp wait --project {id}` | Poll until a run reaches a terminal state. Exits **non-zero** after `--timeout` seconds (default 1800) with nothing changed — that means "still running", not an error. | +| `orx runs {id} [--experiment <expId>]` | Run table, newest first. Run ids come from here. | +| `orx logs <runId> [--head] [--bytes <n>] [--range <s>:<e>]` | Read a run's log (tail by default). | + +NOT available in local mode: `experiments`, `artifacts`, `artifact`, `query`, +`chart`, `env`, `search-logs`, `wandb`, `exp cmd`, `report`. Do not reach for +them — analysis happens through `orx logs`. + +`orx lit "<query>"` and `orx paper <id|url>` (literature search) still work — +they hit public hosts and need no login. + +## The auto-research loop + +Carry one goal across many runs (full guidance: **`orx-experiment-tree`** skill): + +0. **Baseline** (empty project only): create it, set the run command, run once + for reference numbers. +1. **Branch**: `orx create-experiment {id} --title "<idea>" --parent <parentId>` + — one child per distinct thing you try. +2. **Edit** in this worktree: `git fetch origin && git checkout <branch>`, change + the code, commit, `git push` — the job clones from GitHub, so **unpushed + work never runs** (recipes: **`orx-git`** skill). +{launch_step} +4. **Wait — hold your turn open**: call `orx exp wait <expId> --timeout 480` + (or `--project` when several are in flight) in a loop. Exit 0 → terminal, go + analyze; non-zero → immediately call it again (each call stays under your + shell tool's own time limit). +5. **Analyze**: `orx logs <runId>`. Logs are the only evidence channel — make + the run command print every metric you'll need (**`orx-evidence`** skill). +6. **Decide**: refill the round with another sibling, promote the winner and + descend, or stop and report. Write what you learned into `orx exp desc`. + +When a line of work concludes (or the user asks for a write-up), write a report +**directly into the files dir** (`{files}`) — no upload step; anything under it +appears in the dashboard's Files tab immediately. Layout and structure: +**`orx-reports`** skill. + +When the user gives you a research task, see it through this loop — don't stop +after a single step or hand back a half-finished attempt. End your turn only +when the task is achieved, genuinely blocked on a decision only the user can +make, or the approach is exhausted. (For a plain question, just answer it.) + +## Staying online while runs execute + +Nothing re-invokes you when a run finishes, and there are no background +monitors — any process you background dies when your turn ends, so "I'll keep +watching the run" is not something you can do. While a run you launched is in +flight, the wait loop above IS your job: stay in it, and end your turn only +once you've read the result and acted on it. (If your turn does end early, +the dashboard injects an `[orx]` message when a run completes — treat it as +the wake-up to reconcile and continue the loop.) + +## Referencing files + +When you point the reader at a repo source file in chat, wrap it so they can +open it in the dashboard's file viewer: `<file path="relative/path.py" />`, or +with a line target `<file path="relative/path.py" lines="20-40" />`. Use +repo-relative paths (from the worktree root), not absolute paths. Reach for this +whenever you'd otherwise write a bare file path or a markdown link to a file — +the file you edited, the entrypoint you're describing, the config you changed. + +## Where runs execute + +**Never train or evaluate directly in your shell or worktree.** Your worktree +is the edit box: git, reading and writing code, and `orx` orchestration happen +here. The run itself — anything that trains, evaluates, or produces results — +always goes through `orx exp run`: a raw `python train.py` in your shell is +unsupervised, invisible to the dashboard, runs whatever happens to be in your +checkout instead of the branch tip, and blocks your turn. (`--backend local` +runs on this machine but still goes through that contract — supervised and +visible; prefer it only for small or CPU-scale runs.) + +## Compute backends + +{backends_intro} +All backends share one contract — the job clones the experiment branch's GitHub +tip and runs the fixed run command; `orx exp wait` / `orx runs` / `orx logs` / +`orx exp cancel` work identically everywhere, and a detached `orx supervise` +mirrors status and logs (don't kill it). **Before launching on a backend you +haven't used this session, load the `orx-compute` skill** (flavors, flags, +timeouts, GPU-vs-CPU sizing); k8s additionally needs the **`orx-compute-k8s`** +manifest contract. + +## Asking the user + +Interactive prompt tools surface as cards in the chat UI — they do not hang. +If your harness provides a question tool (e.g. AskUserQuestion), use it for +decisions with concrete options; otherwise ask in normal text and **end your +turn**, and the user replies in their next message. + +**Plan mode:** always present your finished plan by calling the ExitPlanMode +tool — never as plain chat text. The plan card is how the user approves the +plan and unlocks execution; a plan left in chat text strands the session in +plan mode. diff --git a/agent-skills/orx-compute-k8s/SKILL.md b/agent-skills/orx-compute-k8s/SKILL.md new file mode 100644 index 0000000..49005b0 --- /dev/null +++ b/agent-skills/orx-compute-k8s/SKILL.md @@ -0,0 +1,70 @@ +--- +name: orx-compute-k8s +description: "Run an experiment on your own Kubernetes cluster (`orx exp run --backend k8s`): the committed-manifest contract orx enforces at submit. Use when the user names k8s, kubernetes, or a cluster, before writing or editing `.orx/k8s.yaml`, for multi-node or Indexed Jobs, or when a k8s submit is rejected." +--- + +**Use `--backend k8s` ONLY when the user explicitly asks to run on their +cluster** ("run this on k8s", "use our cluster") or it is the configured +default target. Local projects (`orx up`) only for now. Auth comes from the +local kubeconfig — orx never stores cluster credentials; the context/namespace +live in `orx up` Settings → Compute. + +**There are no flavors: the run's shape is a Kubernetes manifest you commit +on the experiment branch** (default `.orx/k8s.yaml`, or `--manifest <path>`). +Inspect the cluster yourself (`kubectl get nodes`, allocatable resources, GPU +products) and write whatever the run needs — a single-pod GPU Job, an Indexed +Job spanning nodes with a headless Service, an auxiliary inference Deployment. +The manifest inherits through the experiment tree like all code; changing it is +a commit, visible in the diff like any experimental variable. + +```sh +orx exp run <expId> --backend k8s # runs .orx/k8s.yaml from the branch tip +orx exp run <expId> --backend k8s --manifest infra/run.yaml --timeout 8h +``` + +A minimal manifest: + +```yaml +apiVersion: batch/v1 +kind: Job +metadata: + name: train-{{ORX_RUN}} +spec: + template: + spec: + restartPolicy: Never + containers: + - name: run + image: pytorch/pytorch:2.6.0-cuda12.4-cudnn9-runtime + command: ["bash", "-c", "$ORX_SCRIPT"] + resources: + requests: { nvidia.com/gpu: "4", cpu: "32", memory: "128Gi" } + limits: { nvidia.com/gpu: "4" } +``` + +The contract orx enforces at submit (loud, before anything runs): +- **Exactly one Job** — its completion/failure is the run's outcome. With + several Jobs, label the primary `orx-primary: "true"`. Other resources + (Services, Deployments, ConfigMaps) ride along; cancel deletes exactly what + the manifest created. +- **Some container of that Job must run `$ORX_SCRIPT`** — the env var orx + injects with the clone-and-run script (branch tip + the experiment's fixed + run command). Set `command: ["bash", "-c", "$ORX_SCRIPT"]`. The manifest + shapes *where* the command runs, never *what* runs. +- Every resource needs `metadata.name` (no `generateName`) and no foreign + `metadata.namespace`. Use `{{ORX_RUN}}` in names — orx substitutes a + run-unique token so re-runs don't collide. +- orx injects run labels, the `orx-env` Secret (synced env + `HF_TOKEN` / + `GITHUB_TOKEN`) on the primary Job's containers, and defaults for + `activeDeadlineSeconds` (from `--timeout`, default 4h; a manifest-set value + wins), `ttlSecondsAfterFinished`, and `backoffLimit: 0`. Auxiliary + resources that need the env reference the `orx-env` Secret themselves. +- The run log follows the primary Job's **leader pod** (completion index 0 + for Indexed Jobs, else its sole pod) — make it print the evidence; other + pods stay reachable via `kubectl logs`. Cross-node traffic rides the pod + network — fine for loosely-coupled work (async RL, parameter-server); + tightly-coupled per-step all-reduce wants a fast fabric the cluster may not + have. +- Everything downstream is identical (`orx exp wait` / `orx runs` / + `orx logs`, cancel via `orx exp cancel`). A detached `orx supervise` + watches the Job via kubectl; don't kill it. diff --git a/agent-skills/orx-compute/SKILL.md b/agent-skills/orx-compute/SKILL.md new file mode 100644 index 0000000..a430942 --- /dev/null +++ b/agent-skills/orx-compute/SKILL.md @@ -0,0 +1,340 @@ +--- +name: orx-compute +description: "Launch experiment runs with `orx exp run`: backends (hf, modal, k8s, ssh, slurm, openresearch, local), flavors, timeouts, images, sizing, and `orx exp wait`. Use before launching or re-launching any run, when choosing or switching a backend or GPU flavor, when a job OOMs, stalls, or times out, or when deciding GPU vs CPU." +--- + +Each experiment node has a **run command** (the shell command that trains/evaluates +it) and is launched on **compute** you choose at run time. Compute is *not* stored +on the node — you pick a GPU, a CPU-only instance, or an existing sandbox each time +you launch (local projects may instead carry a default target — see "The default +compute target" below). + +```sh +orx exp status <expId> # status, branch, parent, run command, latest run + commit, local diff recipe +orx exp cmd <expId> # print the current run command +orx exp cmd <baseId> --set "bash run.sh" # set it ONCE on the baseline; children inherit it +orx compute # browse GPU offers across all providers (price-sorted) +orx compute --gpu H100_SXM --count 1 # filter by gpu / count +orx compute --provider vast # filter by provider +orx compute --cpu # browse CPU-only offers (price-sorted) +orx exp run <expId> --gpu H100_SXM --count 1 [--disk 100] # launch on a NEW GPU instance (RunPod) +orx exp run <expId> --gpu H100_SXM --provider vast # launch on another provider's GPU +orx exp run <expId> --cpu cpu5c --vcpus 8 # launch on a NEW CPU-only instance +orx exp run <expId> --sandbox <sandboxId> # launch on an EXISTING node +orx exp cancel <expId> # cancel the in-flight run +``` + +Rules and notes: +- **The run command is a fixed contract — set it once on the baseline, then leave + it alone.** Children inherit it (see the `orx-experiment-tree` skill). Don't + `--set` a different command per child, and don't bake swept hyperparameters into + it — vary the **code/config** on a child's git branch instead, so every variant + runs the same command and their `EVAL.md`s stay comparable. The normal reason to + touch a command is the baseline having none yet. +- **Set a run command before launching.** `orx exp run` fails with a pointer to + `orx exp cmd --set` if the node has none. +- **Push your edits before launching.** A run trains the branch's tip **as it is + on GitHub** — so commit and push first (see the `orx-git` skill). As a + safety net, `orx exp run` refuses a child whose branch has **no changes over its + parent** (the tell-tale of "queued before pushing") — push and retry, or pass + `--force` to run the unchanged code deliberately. +- **Pick compute with exactly one of `--gpu`, `--cpu`, or `--sandbox`.** With + `--gpu`, `--count` defaults to `1` and `--disk` to `100` (GB). A new GPU + instance defaults to **RunPod** (the cheapest matching RunPod offer for the + chosen gpu/count); pass `--provider <name>` to launch from another provider + shown in `orx compute` (e.g. `vast`, `lambda`). New CPU instances are + RunPod-only. Browse valid gpu ids, providers, and prices with `orx compute`. +- **GPU ids are exact enum strings, not family names.** `--gpu H100` is invalid — + the variant suffix is part of the id (`H100_SXM`, `H100_PCIE`, `A100_SXM_80GB`, + `RTX_4090`, …). Use the exact `GPU` column value from `orx compute`; run it + first if unsure. +- **Use `--cpu` for GPU-less work** (data prep, eval harnesses, CPU-bound papers). + The flavor is one of `cpu5c` (compute), `cpu5g` (general), or `cpu5m` (memory); + `--vcpus` is one of `2`, `8`, `32` (default `8`). Browse offers with + `orx compute --cpu`. CPU instances size their disk from the vCPU count, so there + is no `--disk` knob. +- `orx exp run` **queues** the run and returns immediately — it does not wait. + Follow progress with `orx runs <projectId>` and `orx logs <runId>`, or block + with `orx exp wait` (below). + +## The default compute target (local projects) + +The user may set a **default compute target** in the `orx up` dashboard +(Settings → Compute → Make default); it is machine-wide and applies to every +local project. When one is set, `orx exp run <expId>` with no `--backend` +launches there, with the saved default flavor — omitting the flag is how you +use it (flavor-required backends still need `--flavor` if no default flavor +is saved). When none is set, local launches require an explicit `--backend`; the +compute choice is the user's, so ask. Server projects are unaffected: managed +compute (`--gpu`/`--cpu`/`--sandbox`) stays their default. + +## Running on Hugging Face Jobs — `--backend hf` + +**Managed compute (`--gpu`/`--cpu`/`--sandbox`) is the default. Use +`--backend hf` ONLY when the user explicitly asks for Hugging Face Jobs** +(e.g. "run this on HF", "use my huggingface account"), it is the configured +default target (orx up Settings → Compute), or the project context says to +prefer it. A connected HF token by itself is NOT a signal to switch — it just +means the option exists. When in doubt, launch on managed compute. + +With `--backend hf`, the job runs on the user's own HF account (requires +`HF_TOKEN` in the environment — orgs that connect their HF account in compute +settings get it synced automatically) and is billed there per minute; no +OpenResearch balance is spent. + +```sh +orx exp run <expId> --backend hf --flavor a10g-small # one GPU job +orx exp run <expId> --backend hf --flavor a100-large --timeout 8h +orx exp run <expId> --backend hf --flavor cpu-upgrade --image python:3.12 +``` + +Rules and notes: +- **`--flavor` is required** and replaces `--gpu`/`--cpu`/`--sandbox`. Common + flavors: `t4-small`, `a10g-small`, `a10g-large`, `l4x1`, `l40sx1`, + `a100-large`, `h200` (and `x2/x4/x8` multiples); CPU: `cpu-basic`, + `cpu-upgrade`. Prefer the smallest flavor that fits — same sizing discipline + as managed GPUs. +- **Set `--timeout` to cover the whole run** (default `4h`). HF kills the job + at the timeout; a killed job reads as a failed run. +- The job clones the experiment branch's **GitHub tip** and runs the fixed run + command, same contract as managed runs — commit and push first. Private + repos work automatically: the platform mints a repo-scoped clone token from + the project's connected GitHub app and passes it to the job as a secret. + Never ask the user to provision a `GITHUB_TOKEN`; setting one (env or + project env var) is only an override for repos outside the connected app. +- `--image` overrides the container (default: a CUDA pytorch image on GPU + flavors, `python:3.12` on cpu flavors). Pick an image with your deps baked + in when pip-install time dominates the run. +- Everything downstream is identical: the run appears in the tree, `orx exp + wait` / `orx runs` / `orx logs` work unchanged, and cancel from the web or + `orx exp cancel` reaches the job within a few seconds. A detached + `orx supervise` process mirrors status and logs; don't kill it. + +## Running on Modal — `--backend modal` + +**Same rule as HF: managed compute is the default. Use `--backend modal` ONLY +when the user explicitly asks for Modal** ("run this on Modal", "use my Modal +account") or it is the configured default target. Modal runs on the user's +own Modal account, billed there per second; no OpenResearch balance is spent. +It runs the job in a Modal **Sandbox** (an ephemeral container that scales to +zero when the run ends). + +orx auto-provisions a managed `modal` environment on the first Modal launch (no +pip-install needed). You only need a Modal token — `MODAL_TOKEN_ID` + +`MODAL_TOKEN_SECRET` in the environment (set them as org or project env vars and +they sync to the box automatically), or `modal token new`. + +```sh +orx exp run <expId> --backend modal --flavor a10g # one GPU sandbox +orx exp run <expId> --backend modal --flavor a100-80gb --timeout 8h +orx exp run <expId> --backend modal --flavor h100:2 # 2× H100 +orx exp run <expId> --backend modal --flavor cpu --image python:3.12 +``` + +Rules and notes: +- **`--flavor` is required** and replaces `--gpu`/`--cpu`/`--sandbox`. It's a + Modal GPU: `t4`, `l4`, `a10g`, `a100`, `a100-80gb`, `l40s`, `h100`, `h200` + (append `:N` for a count, e.g. `h100:2`); or `cpu` / `cpu-large` for CPU-only. + Prefer the smallest flavor that fits. +- **Set `--timeout` to cover the whole run** (default `4h`). Modal kills the + sandbox at the timeout; a killed sandbox reads as a failed run. +- Same clone contract as HF/managed: the sandbox clones the experiment branch's + **GitHub tip** and runs the fixed command — commit and push first. Private + repos work automatically via the platform's repo-scoped clone token. +- `--image` overrides the container (default: a CUDA pytorch image on GPU + flavors, `python:3.12` on cpu). Pick one with your deps baked in when + pip-install time dominates. +- Everything downstream is identical (`orx exp wait` / `orx runs` / `orx logs`, + cancel from web or `orx exp cancel`). A detached `orx supervise` mirrors + status and logs; don't kill it. + +## Running on your Kubernetes cluster — `--backend k8s` + +Runs the experiment on your own Kubernetes cluster from a manifest committed on +the experiment branch. The full manifest contract lives in the `orx-compute-k8s` +skill — fetch it (`orx skill compute-k8s`) before your first k8s launch. + +## Running on your own box — `--backend ssh` + +**Same rule: use `--backend ssh` ONLY when the user explicitly asks to run on +their own machine/server** ("run this on my box", "use my GPU server") or it +is the configured default target (`--host <alias>` is still required per +launch). Local projects (`orx up`) only for now. It runs the experiment as a +detached background process on a host from your `~/.ssh/config`, over `ssh` — +no scheduler, no container, the host's own environment. + +```sh +orx exp run <expId> --backend ssh --host my-gpu-box # ~/.ssh/config alias +``` + +Rules and notes: +- **`--host` is the ssh host alias** (from `~/.ssh/config`) — a machine, not a + hardware shape, so there is no `--flavor` here. See `orx up` Settings → + Compute → SSH (each host has a "Test" button that checks reachability + git). +- Auth is your ssh keys/agent — orx never reads a key, it just shells out to + `ssh <alias>`. The host needs `git` and `bash`; it clones the experiment + branch's GitHub tip (private repos via the `GITHUB_TOKEN` passed in the run's + env) and runs the fixed command. Commit and push first, same as the others. +- No `--image` (the host's environment is used as-is) and no `--timeout` (the + process runs until it exits or you cancel). +- The run lives under `~/.orx/runs/<runId>/` on the host (`run.sh`, `log`, + `pid`, `exit_code`). Cancel from the web or `orx exp cancel` kills the remote + process group. Everything downstream (`orx exp wait` / `runs` / `logs`) is + identical; a detached `orx supervise` polls it over ssh — don't kill it. + +## Running on your Slurm cluster — `--backend slurm` + +**Same rule: use `--backend slurm` ONLY when the user explicitly asks for their +Slurm cluster** ("submit it to the cluster", "run it via sbatch") or it is the +configured default target. Local projects (`orx up`) only. It submits the +experiment as a batch job via `sbatch` on the login node, reached over ssh — the +host's environment as-is, no container. + +```sh +orx exp run <expId> --backend slurm --host login-node --flavor h100:2 --timeout 4h +orx exp run <expId> --backend slurm # CPU-only, settings default host +``` + +Rules and notes: +- **`--host` is the login node's `~/.ssh/config` alias**; omit it to use the + default from the slurm settings (`orx up` Settings → Compute → Slurm). +- **`--flavor` is a GRES GPU request** (`h100:2` = two H100s) — omit it for a + CPU-only job. There is no `--image`; the job runs in the cluster's own + environment (modules, conda, whatever the login profile provides). +- `--timeout` (default `4h`) applies — size it to cover the whole run; a job + killed at the timeout reads as a failed run. +- Same clone contract as every backend: the job clones the experiment branch's + GitHub tip and runs the fixed command — commit and push first. Everything + downstream (`orx exp wait` / `orx runs` / `orx logs` / `orx exp cancel`) is + identical; a detached `orx supervise` mirrors status and logs — don't kill it. + +## Running on an OpenResearch box — `--backend openresearch` + +**Same rule: use `--backend openresearch` ONLY when the user explicitly asks +for it** ("use an openresearch box", "bill it to the org") or it is the +configured default target. It provisions an **ephemeral OpenResearch machine +billed to the user's org** — created for this run and deleted when it ends — +with a fixed CUDA + PyTorch + uv image. Needs `orx login` and a registered SSH +key. + +```sh +orx exp run <expId> --backend openresearch --flavor h100_sxm:2 --timeout 4h +orx exp run <expId> --backend openresearch --flavor cpu5c:32 --org <orgId> +``` + +Rules and notes: +- **`--flavor` is a GPU id from `orx compute`** (`h100_sxm`, `h100_sxm:2` for a + count) **or a CPU flavor** (`cpu5c` / `cpu5g` / `cpu5m`, with `:vcpus` like + `cpu5c:32`). Run `orx compute` to see what's available. +- Optional: `--org <id>` (when you belong to several), `--disk <GB>`, and + `--provider <P>`. No `--image` — the platform's image is fixed. +- `--timeout` (default `4h`) applies — the box is deleted when the run ends + either way, so nothing persists on it; everything you need must be in the log. +- Same clone contract and downstream commands as every backend; a detached + `orx supervise` mirrors status and logs — don't kill it. + +## Running on this machine — `--backend local` + +**Same rule: use `--backend local` ONLY when the user explicitly asks to run +locally** ("run it on this machine", "just run it here") or it is the +configured default target. Local projects (`orx up`) only. It runs the +experiment as a detached background process on the machine running orx — no +scheduler, no container, this machine's own environment. + +```sh +orx exp run <expId> --backend local +``` + +Rules and notes: +- **Nothing to pick** — no `--flavor`, `--host`, `--image`, or `--timeout` + (the process runs until it exits or you cancel). The hardware is whatever + this machine has; prefer it for small or CPU-scale runs and use a remote + backend for anything heavy — it shares CPU/RAM/GPU with everything else on + the machine. +- Same clone contract as every backend: the run clones the experiment + branch's GitHub tip into its own run dir (never your checkout) and runs the + fixed command — commit and push first. Never run training directly in your + shell instead: that would be unsupervised and invisible to the dashboard. +- The run lives under `<orx data dir>/local-runs/<runId>/` (`run.sh`, `log`, + `pid`, `exit_code`). Cancel from the web or `orx exp cancel` TERMs the + process group. Everything downstream (`orx exp wait` / `orx runs` / `orx logs`) is + identical; a detached `orx supervise` watches it — don't kill it. + +## Spinning up a standalone instance — `orx instance create` + +Provision a persistent instance in an **organization**, not tied to any +experiment — the CLI equivalent of the dashboard's org "Spin up" panel. Use this +for ad-hoc/manual compute (you SSH in yourself); experiment runs use `orx exp run` +instead. + +```sh +orx instance create <orgId> --gpu H100_SXM --count 1 [--disk 100] # GPU box (cheapest provider) +orx instance create <orgId> --gpu H100_SXM --provider runpod # pin a provider +orx instance create <orgId> --cpu cpu5g --vcpus 8 # CPU-only box +``` + +- `<orgId>` comes from `orx projects` (the `org:` line). The flags mirror + `orx exp run`: exactly one of `--gpu` or `--cpu`; `--count`/`--disk` apply to + `--gpu`, `--vcpus` to `--cpu`. +- Unlike `orx exp run`, omitting `--provider` picks the **cheapest** matching + offer across all providers; pass `--provider <name>` to pin one. +- The box provisions asynchronously — the command prints its id and current + status; its SSH host appears once it's online. + +## Waiting on runs — `orx exp wait` + +Block until a run changes state — useful when driving a research loop and you want +to act as soon as a run finishes. Two modes, picked by argument: + +```sh +orx exp wait <expId> # level trigger: poll this experiment's latest run + # until it reaches a terminal state (done/failed/cancelled) +orx exp wait --project <projectId> # edge trigger: return when the FIRST run in the + # project COMPLETES (transitions into done/failed/cancelled) +orx exp wait <expId> --interval 10 --timeout 3600 # tune polling +``` + +- Pass **exactly one** of `<expId>` or `--project` (not both, not neither). +- `--project` is the **budget-loop** primitive: it wakes only on a **completion** + (a run reaching `done`/`failed`/`cancelled`) — i.e. a freed slot. Run *starts*, + new queued runs, and `queued→running` transitions are intentionally ignored, so + it won't wake you on non-events. It returns on the **first** completion — call + it in a loop, one return per tick, and you (not the CLI) decide what to do with + each freed slot. See the per-completion loop in the `orx-experiment-tree` skill. +- **It's a sleep-until-change signal, not the source of truth.** It reports only + completions it saw *during that one call*; a run that finishes between calls is + already terminal next time and won't be reported. On every return, re-read + `orx runs <projectId>` and act on *all* newly-terminal runs — don't treat the + printed line as the complete set, and don't replace `exp wait` with a tight + `orx runs` poll either (use `exp wait` to sleep, `orx runs` to reconcile). +- Call `--project` **while runs are in flight** (right after launching). If every + run is already terminal, there's nothing left to complete, so it returns + immediately printing `drained: no runs in flight` (exit 0) — your loop's + termination signal. +- `--interval` is seconds between polls (default `5`); `--timeout` gives up after + N seconds (default `1800`) and exits **non-zero** so callers can branch on it. + For long training runs, raise `--timeout` (or treat a timeout exit as "nothing + changed yet, loop again") so a wait that simply outlasts the interval isn't + mistaken for an error. +- Progress lines go to **stderr**; the final completion line(s) go to **stdout**, + each as `<runId> <prev> -> <status>` (or `<runId> <status> (new)`), or the + single line `drained: no runs in flight` when nothing was in flight. +- **When a run is `failed`, a `reason:` line follows it.** Compute spin-up + failures (no GPU capacity, provider quota/limit hit, transient provider error) + carry the provider's own message here — the same text the website shows as a + toast. These are usually **transient and retryable**: wait and re-launch the + same run, or pick a different GPU/provider, rather than treating the experiment + as a dead end. If the run instead failed *after* the box came up, the `reason:` + line points at `orx logs <runId>`, where the traceback/OOM/setup error lives. + The same `reason:` line appears under `orx exp status <expId>` and beneath the + `orx runs <projectId>` table. + +## Sizing compute + +- **Decide GPU vs CPU first.** API-driven evals, data prep, and CPU-bound + papers run fine (and far cheaper) on a CPU flavor. +- **Pick the smallest flavor that fits** the model and a minimal batch; don't + reflexively grab the biggest. +- **Let a real failure escalate you.** OOM or hopelessly-slow → move up a + tier. That's expected, not a mistake. +- Raise `--timeout` (`--timeout 1d`) only for genuinely long runs. diff --git a/agent-skills/orx-create/SKILL.md b/agent-skills/orx-create/SKILL.md new file mode 100644 index 0000000..df04ccb --- /dev/null +++ b/agent-skills/orx-create/SKILL.md @@ -0,0 +1,138 @@ +--- +name: orx-create +description: "Create a project (`orx create-project`), seed an empty baseline from existing code, and add experiment nodes (`orx create-experiment`). Use when starting any new project or experiment, when the tree is empty, or when unsure how to bind a repo or set the run command." +--- + +## `orx create-project` — start a new project + +Creates a project in an organization (org ids are printed next to the org names +in `orx projects`), bound to a git repo. The project starts with an **empty +experiment tree** — no baseline yet. Every project is backed by exactly one git +repo; `--repo` picks where that repo comes from: + +```sh +# From an existing repo — yours (bound directly) or any readable repo +# (copied into a fresh repo the platform can write to): +orx create-project <orgId> --name "NanoGPT speedrun" --repo karpathy/nanoGPT + +# From scratch — a fresh blank repo (just a stub root commit on main): +orx create-project <orgId> --name "My new idea" +``` + +- `--repo` takes `owner/repo` or a github.com URL. `--branch` (only with + `--repo`) binds a non-default branch — the baseline will branch off it. + `--description` is optional. +- The command prints the project id + repo. **Next step: create the baseline** + (the root node, the control every variant is measured against): + `orx create-experiment <projectId> --title "Baseline"` (no `--parent`). + Run it once for reference numbers, then hang children off it with + `orx create-experiment <projectId> --title "<t>" --parent <baselineId>`. +- For a **blank** project the baseline you create starts empty (a stub README): + seed it from existing code before launching runs — for a paper or a known idea + there is almost always a repo that implements it, and starting from that is + faster and a far better control than code written from scratch. See "Seeding + an empty baseline" below. + +## Seeding an empty baseline — start from existing code, not from scratch + +On a **blank** project, the baseline you create (`orx create-experiment`, no +`--parent`) starts as an empty stub (just a `README.md`) — there's no code to +run yet. The right move is almost never to **write the implementation by +hand.** For nearly any paper or idea there is already a repo that implements it, +and seeding the baseline from that repo is faster, more faithful, and a far +better control than something typed from memory. Reproductions should start from +the authors' (or a strong community) implementation, not a blank file. + +This is the one legitimate time you put code *on the baseline itself* (cardinal +rule 1's only exception): it applies **only while the baseline is still the empty +stub.** Once it holds real code, the baseline is frozen — vary code on children +from then on. + +**Find the code to seed from, in priority order:** + +1. **The paper's own repo.** If the project has a paper (`orx project view` shows + it, or you were given an arXiv id), run `orx paper <id>` — when alphaXiv has a + repo linked it prints `GitHub: <url>` as the first line. That repo is the + default seed. (Sanity-check the name: the linked repo is the most-starred one + and is occasionally a big framework rather than the paper's own code.) +2. **No repo line, or the wrong repo? Search for one** — a missing `GitHub:` line + means alphaXiv didn't have one indexed, *not* that none exists. Before falling + back to scratch: + - skim the paper's full text for a code/project URL: `orx paper <id> --full` + (authors often link a repo or project page in the body or a footnote); + - search the literature for the method and check related papers' repos: + `orx lit "<the method>"` → `orx paper <hitId>` on the strongest hits and read + their `GitHub:` lines. +3. **No paper at all (a free-form idea project)?** Treat the idea as the query: + `orx lit "<the idea>"`, read the most relevant report with `orx paper`, and + seed from the best implementation it points to. Only if a genuine search turns + up nothing usable do you start from a minimal scaffold of your own — and say so + in the baseline's description. + +**Seed the baseline branch from the chosen repo.** Work in the cache-dir clone +(see the `orx-git` skill); replace the stub with the source's code +as one squashed commit, so the baseline keeps clean provenance and stays rooted +on the project repo: + +```sh +DIR=~/.cache/openresearch/repos/<owner>/<repo> # the PROJECT's repo, from `orx projects` +[ -d "$DIR" ] || git clone https://github.com/<owner>/<repo> "$DIR" +git -C "$DIR" fetch origin +git -C "$DIR" checkout -B orx/<baseline-slug> origin/orx/<baseline-slug> # the baseline's branch + +src=$(mktemp -d) && git clone --depth 1 https://github.com/<srcOwner>/<srcRepo> "$src" +SHA=$(git -C "$src" rev-parse --short HEAD) && rm -rf "$src/.git" +find "$DIR" -mindepth 1 -maxdepth 1 ! -name .git -exec rm -rf {} + # drop the stub +cp -a "$src/." "$DIR/" # lay down the source tree +git -C "$DIR" add -A +git -C "$DIR" commit -m "Seed baseline from <srcOwner>/<srcRepo>@$SHA" +git -C "$DIR" push +``` + +Then make the baseline runnable and proceed normally: + +- read the seeded code, find its entry point, and set the run command **once**: + `orx exp cmd <baselineId> --set "bash run.sh"` (rule 2's one legitimate `--set`); +- run the baseline first for a control `EVAL.md`, then branch children and vary + code per the auto-research loop. The baseline is **frozen** the moment it holds + real code — shrink to the smallest config that still shows the paper's claim by + editing a **child**, never by trimming the root. + +## `orx create-experiment` — add a node to the tree + +Adds a node to the experiment tree. `--title` is always required. The node shape +is chosen by flags: + +```sh +# Child node, branched off an existing experiment: +orx create-experiment <projectId> --title "Larger batch size" --parent <experimentId> + +# Baseline (root) node on the project's bound repo: +orx create-experiment <projectId> --title "Baseline" + +# Additional baseline (another root) when the project already has one: +orx create-experiment <projectId> --title "Baseline v2" --baseline +``` + +- `--parent` selects the shape: with `--parent` ⇒ child; without it, on an + **empty project**, ⇒ the baseline (root) on the repo the project is already + bound to. Once a root exists, a parentless create attaches under the oldest + root on local projects (server projects create another baseline); pass + `--baseline` to explicitly add another root — projects may hold multiple + baselines, each the control for its own subtree. The repo a project works + on is chosen when the **project** is created (`orx create-project` or the + web), so there is no `--repo` flag here. +- **A `--parent` child inherits the parent's run command** (and branches off its + code). You do **not** set a run command on the child — keep it and vary the code + on the child's git branch (see the `orx-experiment-tree` skill). +- **Choose the parent to keep the tree descending, not the root.** Before you pass + `--parent`, name what that parent established that this node builds on. The root + is the right parent only for the *first* round; every later round's siblings hang + off the **previous round's winner** (`orx experiments` shows the current shape). + Piling round after round of children onto the root is the flat-fan failure (see + "Shape the tree" in the `orx-experiment-tree` skill). Co-equal options of the + same decision are siblings under one parent — don't chain them into a line either. +- `--description` is optional but **strongly recommended for children**: use it to + record the hypothesis / the concrete change this node makes. It's the node's + free-form notes field (the same one `orx exp desc` reads/writes), and it's how + you and the analysis tools tell sibling variants apart. diff --git a/agent-skills/orx-evidence/SKILL.local.md b/agent-skills/orx-evidence/SKILL.local.md new file mode 100644 index 0000000..54c71c6 --- /dev/null +++ b/agent-skills/orx-evidence/SKILL.local.md @@ -0,0 +1,40 @@ +--- +name: orx-evidence +description: "Analyze run results in local mode: run logs are the only evidence channel (`orx logs`). Use after any run reaches a terminal state, before declaring a run a success or failure, when metrics are missing from output, or when designing what a run command should print." +--- + +In local mode (`orx up`) run **logs are the only evidence channel** — there is no +`artifacts`, `artifact`, `query`, `chart`, `search-logs`, or `wandb`. Make the run +command print everything you'll need to judge the result, then read it back with +`orx logs`. + +## Reading run logs — `orx logs` + +A run's terminal output (the PTY stream) is captured live while it runs and +persisted afterwards. + +```sh +orx logs <runId> # tail (the end — usually what you want) +orx logs <runId> --head # read from the start instead +orx logs <runId> --bytes 200000 # raise the byte cap (default 64 KB, max 1 MB) +orx logs <runId> --range 4096:8192 # exact byte window [start, end) +``` + +- The log goes to **stdout** (pipe/redirect-friendly); a `[source] bytes a–b of N` + status line goes to **stderr**, noting if content was truncated above/below. +- `<runId>` comes from `orx runs <projectId>` (the run id, not the experiment id). + +## Make the run print its own evidence + +Run logs are the only evidence channel in local mode. Make the run command +print everything you'll need to stdout — final metrics, an `EVAL.md`-style +summary, key config — and read it back with `orx logs <runId>` (use `--head` / +`--range` for long logs). **If a run's output isn't in its log, it's lost.** + +- Print final metrics and a compact summary block at the end of the run, not just + scattered mid-training lines — that's what you'll tail to compare siblings. +- Echo the key config the run actually used (LR, batch size, seed) so a log alone + tells you which variant it was. +- For a long run, a periodic one-line-per-step metric print keeps the trajectory + visible via `orx logs --range`; a run that only prints a final number hides + whether it was converging or diverging. diff --git a/agent-skills/orx-evidence/SKILL.md b/agent-skills/orx-evidence/SKILL.md new file mode 100644 index 0000000..a745b71 --- /dev/null +++ b/agent-skills/orx-evidence/SKILL.md @@ -0,0 +1,172 @@ +--- +name: orx-evidence +description: "Analyze run results: `orx logs`, `orx search-logs`, text artifacts, W&B charts (`orx chart wandb`), and the `orx query` evidence DB. Use after any run finishes, when comparing metrics across runs or experiments, when hunting a failure in logs, or when asked for numbers, tables, or charts." +--- + +## Reading & searching run logs — `orx logs` / `orx search-logs` + +A run's terminal output (the PTY stream) is captured live while it runs and +persisted afterwards. These two commands read and grep it the same way the +OpenResearch assistant's "Read run log" / "Search run log" tools do — byte-range +reads against the persisted log, the live buffer for an in-flight run. + +```sh +orx logs <runId> # tail (the end — usually what you want) +orx logs <runId> --head # read from the start instead +orx logs <runId> --bytes 200000 # raise the byte cap (default 64 KB, max 1 MB) +orx logs <runId> --range 4096:8192 # exact byte window [start, end) +``` + +- The log goes to **stdout** (pipe/redirect-friendly); a `[source] bytes a–b of N` + status line goes to **stderr**, noting if content was truncated above/below. +- `<runId>` comes from `orx runs <projectId>` (the run id, not the experiment id). + +```sh +# Grep a single run, or every run in an experiment: +orx search-logs <projectId> "CUDA out of memory" --run <runId> +orx search-logs <projectId> "Traceback" --experiment <experimentId> +orx search-logs <projectId> "loss=nan" --experiment <id> --max 5000 +``` + +- Search is **literal and case-sensitive**. One of `--run` / `--experiment` is required. +- Each hit prints as `<run8>:<line>: <text> ← <startByte>:<endByte>`. Feed those + byte offsets straight into `orx logs <runId> --range <start>:<end>` to pull the + surrounding context. Results are capped (raise with `--max`). +- **For training metrics, check W&B first.** If the run has a linked W&B run + (`orx wandb <runId>`), `orx chart wandb` is usually a better metrics read than + grepping the log (complete history, exact stats, visible trajectory). Logs + remain the right tool for debugging — tracebacks, OOMs, setup failures — and a + fine metrics fallback when W&B isn't linked or doesn't have the key you need. + +## Run artifacts — `orx artifacts` / `orx artifact` + +Beyond the terminal log, a run uploads **text artifacts** (eval outputs, reports, +generated files). + +**Where artifacts come from — you control this.** Every run has a +`.openresearch/artifacts/` directory **at the root of the repo** (the run command +executes with its working dir set to the repo root, so `.openresearch/artifacts/` +is a plain relative path from there — the same tree you cloned and edited in step +4; don't hardcode an absolute `$HOME`-based path). **Anything the run writes there +is synced to cloud storage (~every 10s, plus a final flush when the run ends) and +becomes readable later via `orx artifacts` / `orx artifact`.** Artifacts aren't +magic — they're whatever your experiment code chose to drop in that directory. +`EVAL.md` is just the conventional one (the run command writes it there); the same +mechanism is yours to use for anything you'll want to examine after the fact: +rollout transcripts, per-sample eval breakdowns, generated text, prompt/response +dumps, plots' underlying data, summary tables. When you implement a node's change +(step 4 of the loop), have the code save these to `.openresearch/artifacts/` — +that's how you turn a run into inspectable evidence instead of a one-shot log. + +- **Keep things you'll re-read as text.** The CLI read commands surface **text + artifacts** (JSON, JSONL, CSV, logs, markdown). Binary blobs — checkpoints, model + weights, `.npy`, images — still persist to storage, but you won't be able to read + them back through `orx artifact`, so dump a text-readable companion (e.g. a JSONL + of rollouts alongside the checkpoint) for anything you intend to analyze via CLI. + +```sh +orx artifacts <runId> # discover what a run uploaded (KEY + SIZE table) +orx artifact <runId> <key> # read one artifact (tail by default) +orx artifact <runId> <key> --head --bytes 200000 # from the start, raise the cap +``` + +- Start with `orx artifacts` to list keys, then `orx artifact <runId> <key>` to + read one. Reading an artifact also **caches it for `orx query`** so you can grep + artifact text via SQL. +- `orx artifact` content goes to **stdout**; byte-range and truncation metadata + go to **stderr**. +- For the run's **code diff**, use local git — see the `orx-git` skill. + +## Charting W&B metrics — `orx chart wandb` + +Renders a single W&B history metric across one or more linked runs as a PNG line +chart — the same renderer the OpenResearch assistant uses. The server fetches the +W&B history, draws the chart, and the CLI **downloads the PNG to a local file and +prints its path**. Because the output is an image, the intended flow is: run the +command, then **`Read` the printed PNG path to view the chart** with your vision. + +```sh +# One metric, two runs overlaid (label after a colon, optional): +orx chart wandb <projectId> --metric "train/loss" \ + --run <runId>:experiment --run <baselineRunId>:baseline + +orx chart wandb <projectId> --metric "train/reward" --run <runId> --smoothing 0.9 +orx chart wandb <projectId> --metric "val/acc" --run <runId> --out ./charts +``` + +- `wandb` is a required first positional (the chart kind; only `wandb` is supported today). +- **`--metric`** is one W&B history key (e.g. `train/loss`). List available keys + first via `orx query <projectId> "select distinct key from wandb_history_keys"`, + or find linked W&B runs with `orx wandb <runId>`. +- **`--run`** is repeatable — pass every run you want on the chart in one call + (up to 6). Append `:label` to set the legend label (defaults to the W&B run id). + The run id comes from `orx runs <projectId>` (the run id, not the experiment id). +- **`--smoothing`** is an EMA factor `0`–`0.99` (default `0.6`). If a chart looks + too noisy to read, re-run with a higher value (e.g. `0.9`) — don't switch metrics. +- **`--out`** sets the output directory (default `~/.cache/openresearch/charts/`). +- Per-run summary stats (`n`, `min`, `max`, `last`) print to **stdout** alongside + the file path, so you can cite exact numbers without opening the image. Runs + that produced no data are listed under `Skipped:`. +- Requires `WANDB_API_KEY` set in the project or org env vars; otherwise the + command reports that and exits non-zero. Run `orx env <projectId>` first to + confirm the key is present (it lists names only, never values). + +## `orx query` — the evidence DB + +The query runs against a **DuckDB "evidence" schema**, which is NOT the same +shape as the REST objects returned by `orx experiments` / `orx runs`. Don't +guess column names from what the other commands display — write queries against +the exact columns below. + +The two tables you'll hit first, with their **full** column lists: + +``` +experiments(id, project_id, parent_experiment_id, slug, title, description, + analysis, run_command, sandbox_id, updated_at) +runs(id, experiment_id, command, status, commit_sha, log_key, sandbox_id, + result_markdown, updated_at) +``` + +The guesses that look right but aren't: + +- **Experiments have no `status` — anywhere.** Status is a *run* property + (`runs.status`). To get "the experiment's status", join its runs: + ```sh + orx query <projectId> "select e.title, r.status, r.updated_at from experiments e left join runs r on r.experiment_id = e.id order by e.title, r.updated_at desc" + ``` +- The parent column is **`parent_experiment_id`**, not `parent_id`. +- There is **no `branch` column** — the git branch is derived from the slug + (`orx/<slug>`). + +There is also a unified **`entities` view** (projects, experiments, runs, and +sandboxes as one table) — handy for tree/graph questions: + +``` +entities(id, type, entity_id, entity_type, project_id, parent_id, parent_type, + parent_entity_id, parent_entity_type, title, name, slug, status, + description, analysis, run_command, updated_at) +``` + +Caveat: its `status` column is populated **only for run and sandbox rows** — +it's NULL for experiments (see above), so don't read it off experiment rows. +For an experiment row, `parent_id` is the parent experiment (or the project, +for the baseline). + +For any table not listed here, discover the real schema before writing queries: + +```sh +orx query <projectId> "select table_name, column_name from information_schema.columns order by 1, 2" +``` + +The full schema, table-by-table guidance, and worked examples live in the +canonical project-query skill — fetch it before doing anything non-trivial: + +```sh +orx skill project-query # the schema + workflow guide +orx skill project-query/references/runs-and-results # runs, metrics, results +orx skill project-query/references/run-diffs # code diffs per run +orx skill project-query/references/text-evidence # logs, artifacts, files +orx skill project-query/references/project-overview # high-level project shape +``` + +These are the same skill docs the OpenResearch assistant reads. diff --git a/agent-skills/orx-experiment-tree/SKILL.md b/agent-skills/orx-experiment-tree/SKILL.md new file mode 100644 index 0000000..e1a36e5 --- /dev/null +++ b/agent-skills/orx-experiment-tree/SKILL.md @@ -0,0 +1,191 @@ +--- +name: orx-experiment-tree +description: "The experiment-tree model and the auto-research loop: shape the tree (stacked bushes), branch/launch/wait/promote, and `orx exp desc` notes. Use before creating, planning, or reorganizing experiments, when deciding what to try next, when a round of runs finishes, or whenever you're unsure how work maps onto the tree." +--- + +A project is a **tree of experiment nodes**. The root (**baseline**) holds the +starting code and a **run command** — the single shell command that trains or +evaluates the node and writes an `EVAL.md` with its results. Every other node is a +**child** branched off a parent, inheriting its code and its run command. The two +rules this depends on — **never edit the baseline** and **the run command + env is +a fixed contract** — are the cardinal rules; everything below assumes them. + +## Shape the tree — stacked bushes, not a flat fan or a noodle + +The single most common way to drive a project badly is to get the **shape** wrong. +There are two opposite failures, and the right shape sits between them: + +``` +FLAT FAN (wrong) NOODLE (wrong) STACKED BUSHES (right) +root root root +├ a ├ b ├ c ... ├ n └ a └ lr-head ┐ round 1: + └ b ├ lr 2e-5 │ a small fan of + └ c └ lr 3e-5 ┘ co-equal options + └ d ... └ winner ── arch-head ┐ round 2 + ├ arch-A │ descends onto + └ arch-B ┘ round 1's winner +``` + +- **Flat fan** (your whole sweep hanging off the root): every result is measured + against the *start*, so wins never accumulate and the tree never makes progress. +- **Noodle** (a long single-child chain): depth manufactured for its own sake — + each step doesn't actually build on the one above it. +- **Stacked bushes** (correct): a *small fan within a round* (the options of one + decision), then **descend onto that round's winner** for the next round. + +**The one rule that produces this shape.** Before you make X a child of Y, name +what Y established that X builds on: + +- **You can name it** ("Y is the LR winner; X keeps that LR and changes the + architecture") → real depth. X is a **child** of Y. Descend. +- **You can't — X and Y are co-equal options you're trying at the same time** + (lr 2e-5 vs lr 3e-5) → they don't build on each other. They're **siblings** in + the same bush. Fan, don't chain. + +So: **width = the open options of one decision** (fan freely — a 3-way LR sweep +*should* be three siblings under a common head); **depth = decisions already +resolved, stacked** (one level down per winner kept). A new *round* never hangs off +the root — it hangs off the previous round's winner. That keeps the tree moving +**downward** as research progresses, without stringing unrelated nodes into a line. + +Re-read `orx experiments` each round and check the shape: a wide row of direct +children off the root with no grandchildren means you're fanning when you should be +descending; a long depth-N chain with no branching means you're chaining co-equal +variants that should have been siblings. + +## The auto-research loop + +To drive a project toward a goal (e.g. "best convergence for d=8") under a fixed +GPU budget, this is the intended flow — do **not** edit the baseline or rewrite the +run command: + +1. **Read the baseline's code.** Clone the project's repo into the cache dir and + read it with your normal tools (see the `orx-git` skill for the path). + See its run command with `orx exp cmd <baseId>` and find where the knobs live + (config files, hyperparameters, model defs). +2. **Form one round's worth of hypotheses** — the co-equal options of a *single* + decision (which LR? which schedule? which init?), each a concrete change you can + make and measure against the others in this round. Don't mix decisions from + different rounds into one batch — that's what produces the flat fan. +3. **Create the round as a bush, and pick its parent deliberately.** All of this + round's options are **siblings under one parent** — the title is the idea, the + description is the concrete change you'll make on that node's branch. The parent is: + - the **baseline**, only for the very first round (nothing has been won yet); or + - the **previous round's confirmed winner**, for every round after — so this + round's changes build *on top of* the last gain instead of resetting to the + start. This is what walks the tree downward (see "Shape the tree" above). + + ```sh + # Round 1 — one decision (the LR), its options fanned off the baseline: + orx create-experiment <projectId> --parent <baseId> --title "LR 2e-5" \ + --description "Set the LR in config.yaml to 2e-5; change nothing else." + orx create-experiment <projectId> --parent <baseId> --title "LR 3e-5" \ + --description "Set the LR in config.yaml to 3e-5; change nothing else." + + # Round 2 — LR 3e-5 won → the next decision (architecture) descends onto it: + orx create-experiment <projectId> --parent <lr3e5WinnerId> --title "Wider MLP" \ + --description "On top of the LR-3e-5 winner, widen the MLP hidden dim 1024→2048 in model.py." + ``` + The child inherits its parent's run command automatically — you don't set it, + and you never give siblings different commands or env vars (cardinal rule 2). +4. **Implement each child's change on its git branch** — `orx create-experiment` + prints the child's branch (`orx/<slug>`); sync the project's clone (in the + openresearch cache dir — see the `orx-git` skill), check the branch out, + edit only the files that idea touches, commit, and push. **Leave the run + command alone:** + ```sh + DIR=~/.cache/openresearch/repos/<owner>/<repo> # owner/repo from `orx projects` + [ -d "$DIR" ] || git clone https://github.com/<owner>/<repo> "$DIR" + git -C "$DIR" fetch origin && git -C "$DIR" checkout -B orx/<child-slug> origin/orx/<child-slug> + # …edit config.yaml under "$DIR": schedule: constant → cosine … + git -C "$DIR" commit -am "cosine LR + warmup" && git -C "$DIR" push + ``` + While you're in the code, **make the run emit the evidence you'll need to judge + it.** Have it write rollout transcripts, per-sample eval breakdowns, generated + text, or summary tables to `.openresearch/artifacts/` (as text — JSONL/CSV/md) — + a directory at the **repo root**, where the run command's working dir points, so + `.openresearch/artifacts/foo.jsonl` is a plain relative path (if your code `cd`s + into a subdir or writes to `/tmp` first, resolve it from the repo root instead). + Those files sync to storage and become readable in step 7 via `orx artifacts` / + `orx artifact`. A run you can only read the tail-log of is far weaker evidence + than one that dumped its rollouts. See the `orx-evidence` skill. +5. **Launch up to your GPU budget** — one run per ready child, in parallel: + ```sh + orx exp run <childId> --gpu H100_SXM --count 1 + ``` +6. **Keep the budget saturated — drive a per-completion loop, not a wait-for-all + barrier.** With a cap of N concurrent runs, you want control back the moment + *any one* run finishes so you can analyze the state of experiments and either refill its slot or stop if no experiment further is needed — not after the whole batch + drains. `orx exp wait --project <projectId>` is built for exactly this: it + returns on the **first** completion. Treat it as one **tick** of a loop, where + *you* are the loop body: + + ``` + # after launching your runs, loop until the project is drained: + loop: + orx exp wait --project <projectId> # sleeps; returns on the first completion + orx runs <projectId> # SOURCE OF TRUTH: re-read all run states + # for each run now terminal that you haven't handled yet: + # - read its results (step 7) and decide: launch a refill? promote it? stop? + # - launch the next queued child to refill the freed slot (step 5) + # if `exp wait` printed "drained: no runs in flight" → batch is done, break + ``` + + Three things make this robust — follow all of them: + - **`exp wait --project` is a sleep-until-change signal, not the source of + truth.** It only reports completions it observed *during that one call*. A + run that finishes while you're analyzing the previous one is already terminal + by the next call and **won't be reported**. So on every wake, re-read + `orx runs <projectId>` and reconcile against the set of runs you've already + handled — act on *every* newly-terminal run, not just the line `exp wait` + printed. (This is the one time you do look at `orx runs` in a loop — as the + reconcile after each wake, **not** as a tight poll in place of `exp wait`.) + - **Re-issue `exp wait` each tick.** One completion → one return → you decide → + you call it again. Don't expect a single `exp wait` to block until everything + is done; that's the failure mode this loop avoids. + - **Terminate on drained.** When no runs are in flight, `exp wait --project` + returns immediately printing `drained: no runs in flight`. That — or seeing + every run terminal in `orx runs` with no more children to launch — is your + exit condition. Don't keep calling it into a timeout. +7. **Analyze each finish as it lands, then iterate.** Do the per-completion read + *inside the loop above*, not deferred to the end — when a run finishes, + **actually read its results** before deciding: `orx artifact <runId> EVAL.md`, + `orx chart wandb …`, `orx query …`. To see exactly what a finished node + changed, use the local git diff recipe `orx exp status <expId>` prints (see + the `orx-git` skill). Don't infer from status alone. Each + completion is a decision point with three moves: + - **Refill** — result is mediocre or inconclusive: launch the next queued child to + keep the GPU budget saturated (step 5). + - **Promote** — result is a clear win: this node becomes the **parent for the next + round**. The next batch of children branch off *it*, not the baseline, so the win + carries forward and the next ideas stack on top of it. This is the move that makes + the tree grow deeper; skipping it is what produces a flat, sweep-only tree. + - **Stop** — goal met, or the branch is exhausted. + + The baseline stays untouched throughout — promotion moves the *focal parent* down the + tree, it never edits the root. + +Stop when the goal is met, or after ~3 consecutive failed or regressed runs. +When you stop, consider writing up the tree as a local markdown report — see the +`orx-reports` skill for the folder layout and section structure. + +## Experiment description / notes — `orx exp desc` + +Each experiment node carries a free-form **description** (markdown) — the same +field set by `create-experiment --description`. Use it for notes: observations, +hypotheses, or a running summary. It is a whole-document field: writing +overwrites whatever was there. + +```sh +orx exp desc <expId> # print the description to stdout (empty → hint on stderr) +orx exp desc <expId> --set "tried lr=3e-4, diverged at step 4k" # overwrite with a short note +cat notes.md | orx exp desc <expId> --stdin # overwrite from stdin (long markdown) +``` + +- **Read** prints the text to **stdout** (pipe/redirect-friendly); when empty, a + hint is printed to **stderr** and stdout stays empty. +- **Write** with exactly one of `--set` (inline) or `--stdin` (whole of stdin). + Passing both is an error. Writing **replaces** the entire description — to + append, read first, edit, and write back. +- `<expId>` comes from `orx experiments <projectId>` (the experiment id, not a run + or project id). diff --git a/agent-skills/orx-git/SKILL.md b/agent-skills/orx-git/SKILL.md new file mode 100644 index 0000000..682fa90 --- /dev/null +++ b/agent-skills/orx-git/SKILL.md @@ -0,0 +1,87 @@ +--- +name: orx-git +description: "Read, edit, and diff a node's code with plain git: sync, commit, and push before running. Use whenever you touch experiment code — before editing any branch, when a checkout or push fails, when comparing two nodes' code, or when a run seems to have picked up stale code." +--- + +Every experiment node **is a git branch** (`orx/<slug>`) on the project's GitHub +repo — `orx create-experiment` prints it. There is no dev box and no `orx` code +command: the **local clone in the cache dir is the standard way to interface +with code** — reading a node's files, diffing what a run changed, and editing — +all with plain git and your own tools. + +(In a local `orx up` session you already sit in a private git worktree of the +project repo, so you can edit the checked-out branch in place — `git fetch origin +&& git checkout <branch>`, edit, commit, push. The cache-dir clone below is the +flow for everything outside a live session, and for cloud/full-set contexts.) + +**Clone into the openresearch cache dir, not your cwd.** The canonical location, +keyed by repo so the same clone is reused across all of a project's experiments: + +``` +~/.cache/openresearch/repos/<owner>/<repo> +``` + +`<owner>/<repo>` comes from `orx projects`. **Never** clone into your current +directory or the user's project folders — clones accreting in `~/projects` is the +failure mode this avoids. + +This is how you **realize a child's hypothesis**: after `create-experiment +--parent`, check out the child's branch and make the specific code/config edits +its description calls for — then commit, push, and run. Edit only the files that +idea touches, and **don't touch the run command** (it's inherited; see the +`orx-experiment-tree` skill). Edit children, never the baseline. + +The sync recipe is **idempotent** — run it verbatim whether or not the clone +already exists from a previous session. Always fetch + reset before editing, so a +reused clone is never stale (and the experiment's branch, created server-side, is +fetched even when it's brand-new and not in your local clone yet): + +```sh +DIR=~/.cache/openresearch/repos/<owner>/<repo> + +# Clone once (skips if it already exists), then ALWAYS sync before touching a branch: +[ -d "$DIR" ] || git clone https://github.com/<owner>/<repo> "$DIR" +git -C "$DIR" fetch origin +git -C "$DIR" checkout -B orx/<slug> origin/orx/<slug> # create, or reset to origin if it exists + +# …edit files under "$DIR" with your normal tools… +git -C "$DIR" commit -am "tune lr" # one or more commits — your call +git -C "$DIR" push # push so runs and the tree see the change +``` + +Rules and notes: +- **Always sync first.** `git -C "$DIR" fetch origin && git -C "$DIR" checkout -B + orx/<slug> origin/orx/<slug>` is mandatory every time — `-B …origin` creates the + branch or resets an existing local one to the GitHub tip, so a persistent clone + never edits a stale base. It discards uncommitted/unpushed local work on that + branch, which is exactly what you don't want to carry across sessions (the + contract is commit + push before moving on). +- **Auth is your own git.** Clone/push use whatever GitHub credentials your `git` + already has — the repo lives under your account or your org, so access is the + same as any of your repos. If a clone or push fails on auth, authenticate git + for github.com (e.g. `gh auth login` or an SSH key) and retry. +- **Push before you run.** `orx exp run` launches from the branch's pushed tip on + GitHub — uncommitted or unpushed edits won't be in the run. Commit and push + first. +- **Reading another node's code** without disturbing your checkout: that branch is + already in the clone after a fetch — `git -C "$DIR" show origin/orx/<slug>:<path>`. + +## Code diffs — local git + +What did a run change vs. its parent experiment? `orx exp status <expId>` prints +the parent's branch, the latest run's full commit SHA, and this exact recipe — +compute the diff locally in the same clone: + +```sh +DIR=~/.cache/openresearch/repos/<owner>/<repo> # owner/repo from `orx projects` +[ -d "$DIR" ] || git clone https://github.com/<owner>/<repo> "$DIR" # cold cache → clone first +git -C "$DIR" fetch origin # ALWAYS fetch first — the commit and parent tip live on GitHub +git -C "$DIR" diff origin/<parent-branch>...<full-commit-sha> +``` + +- The **three-dot** form diffs from the merge-base — what the run's branch + changed, not what the parent gained since the fork. That's the cumulative + "what this experiment did to the code" view. +- Fetch first is mandatory: the run's commit and the parent's tip exist on + GitHub and may not be in your clone yet. +- Root experiments have no parent — there is no diff base, by definition. diff --git a/agent-skills/orx-lit/SKILL.md b/agent-skills/orx-lit/SKILL.md new file mode 100644 index 0000000..4154eb7 --- /dev/null +++ b/agent-skills/orx-lit/SKILL.md @@ -0,0 +1,60 @@ +--- +name: orx-lit +description: "Search literature and read papers via alphaXiv (`orx lit` / `orx paper`). Use when grounding a hypothesis, hunting related work, baselines, or code to seed from, when the user mentions a paper, author, or arXiv id, or before designing a novel experiment." +--- + +These tap **alphaXiv's public corpus** (2.5M+ arXiv papers: CS, math, physics, +stats, q-bio/fin, EE — not PubMed/biomed). They need **no `orx login`** and hit +alphaXiv hosts, not the OpenResearch API. Use them to ground research in real +literature: find related work, pull a paper's structured report, and only drop to +its full text when you need a specific equation/table/section. + +```sh +orx lit "speculative decoding for LLMs" # ranked hits (id, title, date, votes, abstract) +orx lit "rotary position embeddings" --limit 10 # widen the result set (default 5) +orx lit "kv cache compression" --json # raw JSON for programmatic use +orx paper 2401.12345 # machine-readable report (the default) +orx paper https://arxiv.org/abs/2401.12345 # any arXiv/alphaXiv URL works too +orx paper 2401.12345v2 --full # full extracted text (fallback) +``` + +- **`orx lit`** prints, per hit: `<paperId> <title>`, then `<date> · <votes> votes`, + then a truncated abstract. The **`paperId`** is what you feed to `orx paper`. + Results are relevance-ranked, capped at `--limit` (default 5). `--json` emits the + raw hit objects (incl. matched `snippets`) for piping. +- **`orx paper <id>`** writes the report markdown to **stdout** (pipe/redirect-friendly). + The id can be a bare id (`2401.12345`), a versioned id (`2401.12345v2`), or an + arXiv / alphaXiv URL — the CLI normalizes it. +- **The paper's code: `GitHub: <url>` line.** When alphaXiv has a GitHub repo linked + to the paper, `orx paper` prints it as the first line (with `--full` too). If the + report leaves you with questions about *how* something was actually implemented — + exact hyperparameters, training loop details, a trick the paper glosses over — + clone the repo into a temp dir and read the code: + + ```sh + dir=$(mktemp -d) && git clone --depth 1 <githubUrl> "$dir" + ``` + + Inspect it there (grep for the model/optimizer setup, read the configs), and rely + on it as the ground truth for reproducing the paper. No line means no repo is + linked. Note the linked repo is the most-starred one associated with the paper — + occasionally a big framework rather than the paper's own code; sanity-check the + repo name before leaning on it. +- **Report first, full text only when needed.** The default report is a compact + (~10 KB) structured analysis and is enough for most questions. Reach for `--full` + only when the report is missing a specific detail — it returns the entire paper. +- **404s are normal answers, not errors of yours.** A paper whose report hasn't been + generated yet exits non-zero with a hint to try `--full`; one with no extracted + text yet points you at the arXiv PDF. Try `--full`, then the PDF, before giving up. +- Override hosts with `ALPHAXIV_API_URL` (search) and `ALPHAXIV_WEB_URL` (paper docs) + if you ever need to point elsewhere. + +**Grounding a research loop in literature.** Before forming hypotheses for a project +(step 2 of the auto-research loop), search the literature for prior art on the knob +you're about to vary, pull the most relevant report, and let it inform the change you +write into a child's description: + +```sh +orx lit "learning rate warmup schedules transformers" --limit 5 +orx paper <bestPaperId> # read its report; cite the idea in the child's --description +``` diff --git a/agent-skills/orx-reports/SKILL.local.md b/agent-skills/orx-reports/SKILL.local.md new file mode 100644 index 0000000..9397184 --- /dev/null +++ b/agent-skills/orx-reports/SKILL.local.md @@ -0,0 +1,26 @@ +--- +name: orx-reports +description: "Write research reports into the local project's files dir (tree-mirroring folder layout) so they appear in the dashboard's Files tab. Use when a line of work concludes, when the user asks for a write-up, summary, comparison, or figures, or before ending a long task — findings not written down are lost." +--- + +In local mode (`orx up`), reports are written **directly into the project's files +dir** — there is no upload step. Anything under the files dir (reports, figures, +data files) appears in the dashboard's Files tab immediately, grouped by +experiment. The files dir path is shown in your session playbook (the "Files dir:" +line at the top). + +When a line of work concludes (or the user asks for a write-up), write a report +whose layout mirrors the experiment tree — every top-level folder is named for an +experiment slug: + +- **Per-experiment output** goes in the folder named for its slug: + `<files-dir>/<experiment-slug>/report.md`, plus an `images/` subfolder for any + figures it references by relative path. One experiment, one folder — its + `report.md` is that experiment's findings. +- **Cross-experiment syntheses** and anything not tied to one node (comparisons, + lit reviews) go under the reserved `project/` namespace as their own report + folders: `<files-dir>/project/<topic>/report.md`. + +A report's first `# ` heading becomes its title. The markdown references images by +relative path (`![](images/foo.png)`). There is no upload step — save the files and +they show up in the Files tab, grouped by experiment. diff --git a/agent-skills/orx-reports/SKILL.md b/agent-skills/orx-reports/SKILL.md new file mode 100644 index 0000000..b4d9a21 --- /dev/null +++ b/agent-skills/orx-reports/SKILL.md @@ -0,0 +1,38 @@ +--- +name: orx-reports +description: "Write a research report and publish it with `orx report upload` (list/show/download too) so it appears on the project page. Use when a line of work concludes, when the user asks for a write-up, summary, comparison, or figures, or before ending a long task — findings not written down are lost." +--- + +When a line of work concludes, write up the experiment tree as a local markdown +report and publish it to the project with `orx report upload` — it then shows up +on the project page (and its public view) with images inline. + +## Report folder layout + +The folder holds `report.md` plus an `images/` subfolder; the markdown references +images by relative path (`![](images/foo.png)`). A report's first `# ` heading +becomes its title. + +For the canonical section structure and worked layout, fetch the report skill: + +```sh +orx skill report # write a local markdown research report (with charts) +``` + +## Publishing and reading reports — `orx report` + +```sh +orx report upload <projectId> <folder> [--title "<t>"] # upload report.md + images/ +orx report list <projectId> # list the project's reports +orx report show <projectId> <reportId|slug> # print a report's markdown to stdout +orx report download <projectId> <reportId|slug> <dir> # write report.md + images back locally +``` + +- **`upload`** takes a folder (`report.md` + `images/`); the report appears on the + project page and its public view. `--title` overrides the title (otherwise the + first `# ` heading is used). +- **`list`** shows the project's reports. +- **`show`** prints a report's markdown to **stdout** and works on any public + project — handy for reading others' write-ups. +- **`download`** is the inverse of `upload`: it writes `report.md` plus the + referenced images back into a local folder. diff --git a/src/commands/install_skills.rs b/src/commands/install_skills.rs index d0c8ded..f28bad1 100644 --- a/src/commands/install_skills.rs +++ b/src/commands/install_skills.rs @@ -28,25 +28,31 @@ fn installable() -> Vec<Box<dyn Harness>> { .collect() } -/// Write (or overwrite) one harness's shim, creating parent dirs as needed. -/// Overwriting is intentional: re-running keeps the shim current. -async fn write_shim(harness: &dyn Harness) -> Result<PathBuf> { - let path = harness +/// Write (or overwrite) one harness's shim files, creating parent dirs as +/// needed. Overwriting is intentional: re-running keeps the shim current. A +/// harness may write more than one file (Codex: the native skill + a legacy +/// prompt); the primary target is returned first, extras after. +async fn write_shim(harness: &dyn Harness) -> Result<Vec<PathBuf>> { + let primary = harness .skill_target() .ok_or_else(|| anyhow!("{} has no installable skill", harness.name()))?; let shim = harness .skill_shim() .ok_or_else(|| anyhow!("{} has no skill shim", harness.name()))?; - if let Some(parent) = path.parent() { - fs::create_dir_all(parent).await?; + let mut written = Vec::new(); + for (path, contents) in std::iter::once((primary, shim)).chain(harness.extra_skill_targets()) { + if let Some(parent) = path.parent() { + fs::create_dir_all(parent).await?; + } + fs::write(&path, contents).await?; + written.push(path); } - fs::write(&path, shim).await?; - Ok(path) + Ok(written) } /// Non-interactive OpenCode install, for the `orx up` agent bootstrap: the /// spawned opencode discovers `orx` via its skill tool. -pub(crate) async fn install_opencode_shim() -> Result<PathBuf> { +pub(crate) async fn install_opencode_shim() -> Result<Vec<PathBuf>> { let harness = registry() .into_iter() .find(|h| h.id() == "opencode") @@ -88,17 +94,54 @@ pub async fn run(args: crate::InstallSkillsArgs) -> Result<()> { }; for harness in targets { - let path = write_shim(harness).await?; - println!( - "\u{2713} Installed {} skill \u{2192} {}", - harness.name(), - path.display() - ); + for path in write_shim(harness).await? { + println!( + "\u{2713} Installed {} skill \u{2192} {}", + harness.name(), + path.display() + ); + } + if args.full { + for path in write_full_skills(harness).await? { + println!( + "\u{2713} Installed {} skill \u{2192} {}", + harness.name(), + path.display() + ); + } + } } println!("\nYour agent will auto-load it, or you can invoke it with /orx."); Ok(()) } +/// The agent's **global** skills directory — where a native `SKILL.md` skill +/// dir per module goes. Derived from the primary shim target +/// (`<skills-dir>/orx/SKILL.md`), so it tracks each harness's real layout +/// (`~/.claude/skills`, `~/.agents/skills` for Codex, `$XDG/opencode/skills`, +/// `~/.cursor/skills`). +fn global_skills_dir(harness: &dyn Harness) -> Option<PathBuf> { + Some(harness.skill_target()?.parent()?.parent()?.to_path_buf()) +} + +/// Write the full set of modular `orx` skills into the harness's global skills +/// dir (`--full`). Overwrites in place; returns the `SKILL.md` paths written. +async fn write_full_skills(harness: &dyn Harness) -> Result<Vec<PathBuf>> { + use crate::local::agent_skills::{skills, SkillSet}; + + let base = global_skills_dir(harness) + .ok_or_else(|| anyhow!("{} has no global skills dir", harness.name()))?; + let mut written = Vec::new(); + for skill in skills(SkillSet::Full) { + let dir = base.join(skill.name); + fs::create_dir_all(&dir).await?; + let path = dir.join("SKILL.md"); + fs::write(&path, skill.content).await?; + written.push(path); + } + Ok(written) +} + /// The CLI `--agent` alias for a harness. The chat id (`claude-code`) differs /// from the historical CLI word (`claude`), so accept both. fn matches_agent(harness: &dyn Harness, name: &str) -> bool { @@ -120,18 +163,19 @@ fn tilde(path: &Path) -> String { } } -/// The consent prompt's body: one line per detected agent, name → target file. +/// The consent prompt's body: one line per detected agent target file (a +/// harness with more than one — Codex — gets one line per file). fn describe_targets(harnesses: &[&dyn Harness]) -> String { let width = harnesses.iter().map(|h| h.name().len()).max().unwrap_or(0); harnesses .iter() - .filter_map(|h| { - let target = h.skill_target()?; - Some(format!( - " {:<width$} \u{2192} {}", - h.name(), - tilde(&target) - )) + .flat_map(|h| { + let primary = h.skill_target(); + let extras = h.extra_skill_targets(); + primary + .into_iter() + .chain(extras.into_iter().map(|(p, _)| p)) + .map(move |target| format!(" {:<width$} \u{2192} {}", h.name(), tilde(&target))) }) .collect::<Vec<_>>() .join("\n") @@ -173,12 +217,14 @@ pub async fn offer_install_after_login() { } if matches!(answer.trim().to_lowercase().as_str(), "" | "y" | "yes") { for harness in present { - if let Ok(path) = write_shim(harness).await { - println!( - "\u{2713} Installed {} skill \u{2192} {}", - harness.name(), - tilde(&path) - ); + if let Ok(paths) = write_shim(harness).await { + for path in paths { + println!( + "\u{2713} Installed {} skill \u{2192} {}", + harness.name(), + tilde(&path) + ); + } } } } else { @@ -222,4 +268,53 @@ mod tests { assert!(h.config_home().is_some(), "{} missing config home", h.id()); } } + + /// Codex migrated to a native SKILL.md (`~/.agents/skills/orx/SKILL.md`) but + /// still writes the legacy `/orx` prompt alongside — two targets, listed in + /// the consent prompt. + #[test] + fn codex_writes_native_skill_and_legacy_prompt() { + let all = installable(); + let codex = all.iter().find(|h| h.id() == "codex").unwrap().as_ref(); + + let primary = codex.skill_target().unwrap(); + assert!( + primary.ends_with(".agents/skills/orx/SKILL.md"), + "primary target is the native skill, got {}", + primary.display() + ); + let extras = codex.extra_skill_targets(); + assert_eq!(extras.len(), 1, "one legacy prompt"); + assert!( + extras[0].0.ends_with(".codex/prompts/orx.md"), + "legacy prompt path, got {}", + extras[0].0.display() + ); + + let body = describe_targets(&[codex]); + assert_eq!(body.lines().count(), 2, "both codex targets listed"); + } + + /// The `--full` global skills dir is the parent of the harness's skill dir + /// (`~/.claude/skills`, `~/.agents/skills` for codex, `$XDG/opencode/skills`). + #[test] + fn global_skills_dir_is_the_skills_parent() { + let all = installable(); + for h in all.iter().map(Box::as_ref) { + let dir = global_skills_dir(h).unwrap(); + // The primary shim target sits at <dir>/orx/SKILL.md. + assert_eq!( + h.skill_target().unwrap(), + dir.join("orx").join("SKILL.md"), + "{} global skills dir mismatch", + h.id() + ); + assert!( + dir.ends_with("skills"), + "{} global skills dir should end in `skills`, got {}", + h.id(), + dir.display() + ); + } + } } diff --git a/src/commands/skill.rs b/src/commands/skill.rs index c2b9ecb..7909005 100644 --- a/src/commands/skill.rs +++ b/src/commands/skill.rs @@ -1,5 +1,6 @@ use crate::config; use crate::error::{require_credentials, Result}; +use crate::local::agent_skills::{self, SkillSet}; // Bundled top-level overview, shipped with the CLI so `orx skill` works without // a round-trip. Deeper references are fetched live from the API. Embedded at @@ -7,28 +8,39 @@ use crate::error::{require_credentials, Result}; const SKILL_MD: &str = include_str!("../../SKILL.md"); pub async fn run(args: crate::SkillArgs) -> Result<()> { - // With a path: fetch the canonical doc from the API (same docs the assistant - // reads), so the schema never drifts from a hand-maintained copy. if let Some(path) = args.path { + // First: a bundled module (with or without the `orx-` prefix). These + // ship in the binary, so they resolve offline and never drift. + if let Some(skill) = agent_skills::find(&path) { + println!("{}", skill.content.trim_end()); + return Ok(()); + } + // Otherwise fetch the canonical doc from the API (same docs the assistant + // reads), so the schema never drifts from a hand-maintained copy. let creds = require_credentials().await; let content = crate::client::read_skill(&creds, &path).await?; println!("{}", content.content); return Ok(()); } - // No path: print the bundled overview, then list fetchable skills (best - // effort — skip the index if we can't reach the API). + // No path: print the bundled overview, then the bundled module index, then + // list API-fetchable deep references (best effort — skip if unreachable). println!("{}", SKILL_MD); + println!("\nBundled modules (orx skill <name>):"); + for s in agent_skills::skills(SkillSet::Full) { + println!(" {:<20} {}", s.name, s.description); + } + let creds = match config::load_credentials().await? { Some(c) => c, None => return Ok(()), }; - // API unreachable — the bundled overview is enough on its own, so ignore Err. + // API unreachable — the bundled overview + modules are enough, so ignore Err. if let Ok(list) = crate::client::list_skills(&creds).await { if !list.skills.is_empty() { - println!("\nFetchable skills (orx skill <path>):"); + println!("\nFetchable references (orx skill <path>):"); for s in &list.skills { println!(" {}", s.path); } diff --git a/src/local/agent_skills.rs b/src/local/agent_skills.rs new file mode 100644 index 0000000..604a240 --- /dev/null +++ b/src/local/agent_skills.rs @@ -0,0 +1,332 @@ +//! Native modular agent skills for `orx`. +//! +//! The monolithic `orx skill` overview (repo-root `SKILL.md`) is factored into a +//! handful of focused modules that live as **literal, complete skill files** in +//! the repo `agent-skills/` directory (`agent-skills/<name>/SKILL.md`, +//! frontmatter included — readable as-is on GitHub) and are embedded in the +//! binary at compile time and installed verbatim. Two consumers use them: +//! +//! * **Local `orx up` sessions** get the [`SkillSet::Local`] modules written as +//! native `SKILL.md` skill dirs *into the session worktree* — fresh on every +//! turn, right beside the playbook (see [`ensure_session_skills`]). The harness +//! picks the skills subdir (`.claude/skills`, `.opencode/skills`, +//! `.agents/skills`), so the session's own agent auto-discovers them and never +//! sees drift. +//! * **`orx skill <name>`** resolves a bundled module (with or without the +//! `orx-` prefix) and prints it; the no-arg overview lists the +//! [`SkillSet::Full`] set. `orx install-skills --full` writes the Full set into +//! an agent's global skills dir (the dedicated cloud box). +//! +//! The two sets share the same public skill *names* so docs and references stay +//! stable; a few modules swap their **body** between a local-mode form +//! (logs-only evidence, files-dir reports) and a full/cloud form (artifacts + +//! query + chart; `orx report` upload). The `orx-` prefix on every dir name +//! makes them unmistakable in an agent's skill listing. + +use std::path::Path; + +use crate::error::{anyhow, Result}; + +/// One embedded skill module: its public name (== skill dir name == the `name:` +/// frontmatter field), a one-line description (mirrored in the file's +/// frontmatter — a test enforces they agree), and the complete `SKILL.md` +/// contents, installed and printed verbatim. +pub struct AgentSkill { + pub name: &'static str, + pub description: &'static str, + pub content: &'static str, +} + +/// Which set of module bodies to serve. The two sets carry the same public skill +/// names; only a few bodies differ (see the module docs). +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum SkillSet { + /// `orx up` local mode: logs-only evidence, files-dir reports, no `create`. + Local, + /// Full/cloud surface: artifacts + query + chart evidence, `orx report` + /// upload, and the project/experiment creation module. + Full, +} + +// --- Module files (embedded verbatim from the repo `agent-skills/` dir; the +// `SKILL.local.md` siblings are the local-mode body variants under the same +// public skill name) ---------------------------------------------------------- + +const COMPUTE: &str = include_str!("../../agent-skills/orx-compute/SKILL.md"); +const COMPUTE_K8S: &str = include_str!("../../agent-skills/orx-compute-k8s/SKILL.md"); +const EXPERIMENT_TREE: &str = include_str!("../../agent-skills/orx-experiment-tree/SKILL.md"); +const GIT_EDITING: &str = include_str!("../../agent-skills/orx-git/SKILL.md"); +const LIT: &str = include_str!("../../agent-skills/orx-lit/SKILL.md"); +const CREATE: &str = include_str!("../../agent-skills/orx-create/SKILL.md"); +const REPORTS_LOCAL: &str = include_str!("../../agent-skills/orx-reports/SKILL.local.md"); +const REPORTS_CLOUD: &str = include_str!("../../agent-skills/orx-reports/SKILL.md"); +const EVIDENCE_LOCAL: &str = include_str!("../../agent-skills/orx-evidence/SKILL.local.md"); +const EVIDENCE_CLOUD: &str = include_str!("../../agent-skills/orx-evidence/SKILL.md"); + +// Descriptions are the *trigger surface*: what the module covers plus explicit, +// liberal "Use when …" cues (false positives beat false negatives — an agent +// that loads a module needlessly wastes a little context; one that misses it +// works blind). Keep each ≤400 chars — Codex's ambient budget is ~8k across +// the whole set. + +const S_COMPUTE: AgentSkill = AgentSkill { + name: "orx-compute", + description: "Launch experiment runs with `orx exp run`: backends (hf, modal, k8s, ssh, slurm, openresearch, local), flavors, timeouts, images, sizing, and `orx exp wait`. Use before launching or re-launching any run, when choosing or switching a backend or GPU flavor, when a job OOMs, stalls, or times out, or when deciding GPU vs CPU.", + content: COMPUTE, +}; +const S_COMPUTE_K8S: AgentSkill = AgentSkill { + name: "orx-compute-k8s", + description: "Run an experiment on your own Kubernetes cluster (`orx exp run --backend k8s`): the committed-manifest contract orx enforces at submit. Use when the user names k8s, kubernetes, or a cluster, before writing or editing `.orx/k8s.yaml`, for multi-node or Indexed Jobs, or when a k8s submit is rejected.", + content: COMPUTE_K8S, +}; +const S_EXPERIMENT_TREE: AgentSkill = AgentSkill { + name: "orx-experiment-tree", + description: "The experiment-tree model and the auto-research loop: shape the tree (stacked bushes), branch/launch/wait/promote, and `orx exp desc` notes. Use before creating, planning, or reorganizing experiments, when deciding what to try next, when a round of runs finishes, or whenever you're unsure how work maps onto the tree.", + content: EXPERIMENT_TREE, +}; +const S_GIT: AgentSkill = AgentSkill { + name: "orx-git", + description: "Read, edit, and diff a node's code with plain git: sync, commit, and push before running. Use whenever you touch experiment code — before editing any branch, when a checkout or push fails, when comparing two nodes' code, or when a run seems to have picked up stale code.", + content: GIT_EDITING, +}; +const S_LIT: AgentSkill = AgentSkill { + name: "orx-lit", + description: "Search literature and read papers via alphaXiv (`orx lit` / `orx paper`). Use when grounding a hypothesis, hunting related work, baselines, or code to seed from, when the user mentions a paper, author, or arXiv id, or before designing a novel experiment.", + content: LIT, +}; +const S_CREATE: AgentSkill = AgentSkill { + name: "orx-create", + description: "Create a project (`orx create-project`), seed an empty baseline from existing code, and add experiment nodes (`orx create-experiment`). Use when starting any new project or experiment, when the tree is empty, or when unsure how to bind a repo or set the run command.", + content: CREATE, +}; +const S_REPORTS_LOCAL: AgentSkill = AgentSkill { + name: "orx-reports", + description: "Write research reports into the local project's files dir (tree-mirroring folder layout) so they appear in the dashboard's Files tab. Use when a line of work concludes, when the user asks for a write-up, summary, comparison, or figures, or before ending a long task — findings not written down are lost.", + content: REPORTS_LOCAL, +}; +const S_REPORTS_CLOUD: AgentSkill = AgentSkill { + name: "orx-reports", + description: "Write a research report and publish it with `orx report upload` (list/show/download too) so it appears on the project page. Use when a line of work concludes, when the user asks for a write-up, summary, comparison, or figures, or before ending a long task — findings not written down are lost.", + content: REPORTS_CLOUD, +}; +const S_EVIDENCE_LOCAL: AgentSkill = AgentSkill { + name: "orx-evidence", + description: "Analyze run results in local mode: run logs are the only evidence channel (`orx logs`). Use after any run reaches a terminal state, before declaring a run a success or failure, when metrics are missing from output, or when designing what a run command should print.", + content: EVIDENCE_LOCAL, +}; +const S_EVIDENCE_CLOUD: AgentSkill = AgentSkill { + name: "orx-evidence", + description: "Analyze run results: `orx logs`, `orx search-logs`, text artifacts, W&B charts (`orx chart wandb`), and the `orx query` evidence DB. Use after any run finishes, when comparing metrics across runs or experiments, when hunting a failure in logs, or when asked for numbers, tables, or charts.", + content: EVIDENCE_CLOUD, +}; + +/// The modules for a given set, in a stable order. Local and Full share names; +/// `reports`/`evidence` swap bodies, and `create` is Full-only. +pub fn skills(set: SkillSet) -> Vec<&'static AgentSkill> { + match set { + SkillSet::Local => vec![ + &S_EXPERIMENT_TREE, + &S_GIT, + &S_COMPUTE, + &S_COMPUTE_K8S, + &S_EVIDENCE_LOCAL, + &S_REPORTS_LOCAL, + &S_LIT, + ], + SkillSet::Full => vec![ + &S_CREATE, + &S_EXPERIMENT_TREE, + &S_GIT, + &S_COMPUTE, + &S_COMPUTE_K8S, + &S_EVIDENCE_CLOUD, + &S_REPORTS_CLOUD, + &S_LIT, + ], + } +} + +/// Resolve a bundled Full-set skill by name, accepting both the public name +/// (`orx-compute`) and the bare form (`compute`). `None` for an unknown name — +/// the caller falls back to the live API fetch. +pub fn find(name: &str) -> Option<&'static AgentSkill> { + let want = name.trim(); + skills(SkillSet::Full) + .into_iter() + .find(|s| s.name == want || s.name.strip_prefix("orx-") == Some(want)) +} + +/// Write the [`SkillSet::Local`] modules as `<worktree>/<skills_dir_rel>/<name>/SKILL.md`, +/// overwriting every file on every call (same freshness semantics as the +/// playbook — zero drift). Returns `Err` on the first write failure; the caller +/// treats it like a playbook-write error. +pub fn ensure_session_skills(worktree: &Path, skills_dir_rel: &str) -> Result<()> { + let base = worktree.join(skills_dir_rel); + for skill in skills(SkillSet::Local) { + let dir = base.join(skill.name); + std::fs::create_dir_all(&dir) + .map_err(|e| anyhow!("Could not create {}: {}", dir.display(), e))?; + let path = dir.join("SKILL.md"); + std::fs::write(&path, skill.content) + .map_err(|e| anyhow!("Could not write {}: {}", path.display(), e))?; + } + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + use std::collections::HashSet; + + fn is_valid_name(name: &str) -> bool { + // ^[a-z0-9]+(-[a-z0-9]+)*$ + !name.is_empty() + && name.split('-').all(|seg| { + !seg.is_empty() + && seg + .chars() + .all(|c| c.is_ascii_lowercase() || c.is_ascii_digit()) + }) + } + + #[test] + fn names_are_valid_unique_and_prefixed() { + for set in [SkillSet::Local, SkillSet::Full] { + let mut seen = HashSet::new(); + for s in skills(set) { + assert!( + is_valid_name(s.name), + "{:?}: invalid name {:?}", + set, + s.name + ); + assert!( + s.name.starts_with("orx-"), + "{:?}: name {:?} not orx- prefixed", + set, + s.name + ); + assert!( + seen.insert(s.name), + "{:?}: duplicate name {:?}", + set, + s.name + ); + } + } + } + + #[test] + fn descriptions_are_within_bounds() { + for set in [SkillSet::Local, SkillSet::Full] { + for s in skills(set) { + let len = s.description.chars().count(); + assert!( + (1..=400).contains(&len), + "{:?}: {} description is {} chars (want 1..=400)", + set, + s.name, + len + ); + } + } + } + + #[test] + fn file_frontmatter_matches_code_and_is_valid_yaml() { + // The skill files under `agent-skills/` are the literal installed + // artifacts (embedded verbatim), so their frontmatter must agree with + // the code's name/description — this is the drift guard between the + // GitHub-readable files and the indexes generated from the consts. + for set in [SkillSet::Local, SkillSet::Full] { + for s in skills(set) { + let mut lines = s.content.lines(); + assert_eq!(lines.next(), Some("---"), "{} missing opening ---", s.name); + let name_line = lines.next().unwrap_or_default(); + let desc_line = lines.next().unwrap_or_default(); + assert_eq!( + name_line, + format!("name: {}", s.name), + "{} name frontmatter", + s.name + ); + + // The description value must be YAML-safe. The files carry it + // as a JSON-quoted scalar; strip `description: ` and JSON-decode + // — the round-trip proves the quoting is well-formed and that + // the file agrees with the code's description. A bare + // (unquoted) value containing `: ` — the bug this guards — + // would not JSON-decode. + let value = desc_line + .strip_prefix("description: ") + .unwrap_or_else(|| panic!("{} description frontmatter shape", s.name)); + let decoded: String = serde_json::from_str(value).unwrap_or_else(|e| { + panic!("{} description is not a quoted scalar: {e}", s.name) + }); + assert_eq!(decoded, s.description, "{} description round-trip", s.name); + // A quoted scalar is a single physical line — no embedded newline. + assert!( + !s.description.contains('\n'), + "{} description has a newline", + s.name + ); + + assert_eq!(lines.next(), Some("---"), "{} missing closing ---", s.name); + // A non-empty body follows the closing frontmatter fence + // (`\n---\n\n` separates the frontmatter block from the body). + let body = s + .content + .split_once("\n---\n\n") + .map(|(_, body)| body) + .unwrap_or(""); + assert!(!body.trim().is_empty(), "{} has an empty body", s.name); + } + } + } + + #[test] + fn find_resolves_prefixed_and_bare() { + assert_eq!(find("orx-compute").map(|s| s.name), Some("orx-compute")); + assert_eq!(find("compute").map(|s| s.name), Some("orx-compute")); + assert_eq!(find("orx-create").map(|s| s.name), Some("orx-create")); + assert_eq!(find("create").map(|s| s.name), Some("orx-create")); + assert!(find("does-not-exist").is_none()); + assert!(find("project-query").is_none()); + } + + #[test] + fn ensure_session_skills_writes_local_set_idempotently() { + let tmp = std::env::temp_dir().join(format!( + "orx-agent-skills-test-{}-{}", + std::process::id(), + uuid::Uuid::new_v4() + )); + let rel = ".claude/skills"; + ensure_session_skills(&tmp, rel).unwrap(); + + let base = tmp.join(rel); + let expected: HashSet<&str> = skills(SkillSet::Local).iter().map(|s| s.name).collect(); + let got: HashSet<String> = std::fs::read_dir(&base) + .unwrap() + .map(|e| e.unwrap().file_name().to_string_lossy().into_owned()) + .collect(); + let got_refs: HashSet<&str> = got.iter().map(String::as_str).collect(); + assert_eq!(got_refs, expected, "wrote exactly the Local-set dirs"); + + for s in skills(SkillSet::Local) { + let path = base.join(s.name).join("SKILL.md"); + let content = std::fs::read_to_string(&path).unwrap(); + assert_eq!(content, s.content, "{} SKILL.md content", s.name); + } + + // Idempotent: a second call overwrites in place and changes nothing. + ensure_session_skills(&tmp, rel).unwrap(); + let got2: HashSet<String> = std::fs::read_dir(&base) + .unwrap() + .map(|e| e.unwrap().file_name().to_string_lossy().into_owned()) + .collect(); + assert_eq!(got2, got, "second call is idempotent"); + + let _ = std::fs::remove_dir_all(&tmp); + } +} diff --git a/src/local/harness/claude.rs b/src/local/harness/claude.rs index fcd10ba..ad72f29 100644 --- a/src/local/harness/claude.rs +++ b/src/local/harness/claude.rs @@ -286,6 +286,10 @@ impl Harness for ClaudeCode { fn skill_shim(&self) -> Option<&'static str> { Some(super::CLAUDE_SKILL) } + + fn session_skills_dir(&self) -> Option<&'static str> { + Some(".claude/skills") + } } /// Session mode → Claude Code `--permission-mode` value. The shared wire ids are @@ -563,8 +567,11 @@ async fn run_turn(ctx: &mut TurnCtx) -> Result<()> { })?; let project = ctx.project.clone(); let session_id = ctx.session_id.clone(); + // The modular orx skills land in the harness's session-skills dir, fresh, + // for this session's agent to auto-load — source of truth is the trait. + let skills_dir = ClaudeCode.session_skills_dir(); let (repo, playbook) = - tokio::task::spawn_blocking(move || ensure_playbook(&project, &session_id)) + tokio::task::spawn_blocking(move || ensure_playbook(&project, &session_id, skills_dir)) .await .map_err(|e| anyhow!("playbook task failed: {e}"))??; diff --git a/src/local/harness/codex.rs b/src/local/harness/codex.rs index f721b17..b0322a0 100644 --- a/src/local/harness/codex.rs +++ b/src/local/harness/codex.rs @@ -227,11 +227,38 @@ impl Harness for Codex { } fn skill_target(&self) -> Option<PathBuf> { - Some(self.config_home()?.join("prompts").join("orx.md")) + // Codex now speaks native SKILL.md skills (`~/.agents/skills/`); the + // legacy `~/.codex/prompts/` path is deprecated and model-invisible. The + // primary target is the real skill; the legacy prompt still rides along + // via `extra_skill_targets` for older codex versions. + Some( + dirs::home_dir()? + .join(".agents") + .join("skills") + .join("orx") + .join("SKILL.md"), + ) } fn skill_shim(&self) -> Option<&'static str> { - Some(super::CODEX_PROMPT) + // Native SKILL.md format, same body as Claude Code / OpenCode / Cursor. + Some(super::CLAUDE_SKILL) + } + + fn extra_skill_targets(&self) -> Vec<(PathBuf, &'static str)> { + // Keep the legacy `/orx` prompt for codex versions that don't yet read + // `~/.agents/skills/`. + match dirs::home_dir() { + Some(home) => vec![( + home.join(".codex").join("prompts").join("orx.md"), + super::CODEX_PROMPT, + )], + None => Vec::new(), + } + } + + fn session_skills_dir(&self) -> Option<&'static str> { + Some(".agents/skills") } } @@ -586,8 +613,11 @@ async fn run_turn_app_server(ctx: &mut TurnCtx) -> Result<()> { .await?; let project = ctx.project.clone(); let session_id = ctx.session_id.clone(); + // The modular orx skills land in the harness's session-skills dir, fresh, + // for this session's agent to auto-load — source of truth is the trait. + let skills_dir = Codex.session_skills_dir(); let (repo, playbook) = - tokio::task::spawn_blocking(move || ensure_playbook(&project, &session_id)) + tokio::task::spawn_blocking(move || ensure_playbook(&project, &session_id, skills_dir)) .await .map_err(|e| anyhow!("playbook task failed: {e}"))??; let playbook_md = std::fs::read_to_string(&playbook).unwrap_or_default(); @@ -1042,8 +1072,11 @@ async fn run_turn_exec(ctx: &mut TurnCtx) -> Result<()> { let bin = find_codex_required()?; let project = ctx.project.clone(); let session_id = ctx.session_id.clone(); + // The modular orx skills land in the harness's session-skills dir, fresh, + // for this session's agent to auto-load — source of truth is the trait. + let skills_dir = Codex.session_skills_dir(); let (repo, playbook) = - tokio::task::spawn_blocking(move || ensure_playbook(&project, &session_id)) + tokio::task::spawn_blocking(move || ensure_playbook(&project, &session_id, skills_dir)) .await .map_err(|e| anyhow!("playbook task failed: {e}"))??; diff --git a/src/local/harness/mod.rs b/src/local/harness/mod.rs index 56eb78c..0b14414 100644 --- a/src/local/harness/mod.rs +++ b/src/local/harness/mod.rs @@ -21,7 +21,7 @@ mod claude; pub(crate) mod codex; mod cursor; mod detect; -mod opencode; +pub(crate) mod opencode; mod options; mod plan_gate; @@ -148,10 +148,31 @@ pub trait Harness: Send + Sync { None } + /// Additional `(target, contents)` shim files beyond the primary + /// [`skill_target`](Self::skill_target) — for a harness that must write more + /// than one file (Codex writes both the new `~/.agents/skills/orx/SKILL.md` + /// and the legacy `~/.codex/prompts/orx.md` for older versions). Each is + /// written and reported alongside the primary target. Default: none. + fn extra_skill_targets(&self) -> Vec<(PathBuf, &'static str)> { + Vec::new() + } + /// True if the agent looks set up on this machine (its config home exists). fn is_installed_locally(&self) -> bool { self.config_home().map(|h| h.exists()).unwrap_or(false) } + + // --- session-skills capability ---------------------------------------- + + /// The worktree-relative dir this harness discovers native `SKILL.md` skill + /// dirs under, for a **local `orx up` session** — `.claude/skills`, + /// `.opencode/skills`, `.agents/skills`. The modular `orx` skills are + /// written there (fresh every turn, beside the playbook) so the session's + /// own agent auto-loads them. `None` for a harness with no local chat + /// session that can host per-session skills (Cursor). + fn session_skills_dir(&self) -> Option<&'static str> { + None + } } /// The one registry. Every consumer — chat dispatch, detection sweep, the @@ -222,9 +243,10 @@ pub(crate) fn xdg_config_home() -> PathBuf { // stays in the CLI's SKILL.md and is fetched fresh each session, so the // installed shim never drifts as that guide changes — which it does, often. -/// Claude Code / OpenCode / Cursor skill (`skills/orx/SKILL.md`). The frontmatter -/// `description` drives auto-discovery and the `/orx` invocation; the body only -/// points the agent at the live guide. +/// Native `SKILL.md` shim (`skills/orx/SKILL.md`) — Claude Code, OpenCode, +/// Cursor, and now Codex (`~/.agents/skills/orx/`) all read this same format. +/// The frontmatter `description` drives auto-discovery and the `/orx` +/// invocation; the body only points the agent at the live guide. pub(super) const CLAUDE_SKILL: &str = r#"--- name: orx description: Drive automated ML research on OpenResearch with the `orx` CLI — create experiments, launch and monitor runs on GPU compute, analyze results and logs, query the evidence DB, and search literature. Use whenever the user wants to understand, explain, explore, or work on an OpenResearch project, run experiments, do auto-research, or mentions orx or OpenResearch. @@ -242,10 +264,12 @@ start of every session** instead of relying on this file or prior memory. orx skill ``` -This prints the current manual: the cardinal rules, the full command reference, -the experiment-tree model, and the auto-research loop. Read it before taking any -action. For a deeper reference on a specific area, run `orx skill <path>` using -the paths listed at the end of that output. +This prints the current manual — the cardinal rules and a command +quick-reference — followed by a **live index of modules**. Read it before taking +any action. For the detail on a specific area, run `orx skill <name>` to print +that module (e.g. `orx skill experiment-tree`, `orx skill compute`); the same +command fetches deeper API-served references by the paths listed at the end of +the output. ## 2. Carry out the user's research goal @@ -260,15 +284,19 @@ The user must be logged in. If any command reports `Not logged in`, ask them to run `orx login`. "#; -/// Codex prompt (`~/.codex/prompts/orx.md`), invoked as `/orx`. Plain markdown for -/// broad version compatibility; `$ARGUMENTS` is substituted with whatever the user -/// types after the command (and reads fine as-is if their Codex doesn't expand it). +/// Legacy Codex prompt (`~/.codex/prompts/orx.md`), invoked as `/orx`. Codex now +/// reads native SKILL.md skills (`~/.agents/skills/`, gets `CLAUDE_SKILL`); this +/// prompt is still written alongside for older codex versions that don't. +/// Plain markdown for broad version compatibility; `$ARGUMENTS` is substituted +/// with whatever the user types after the command (and reads fine as-is if their +/// Codex doesn't expand it). pub(super) const CODEX_PROMPT: &str = r#"Drive automated ML research on OpenResearch using the `orx` CLI. Start by running `orx skill` to load the current operating manual — the cardinal -rules, the full command reference, the experiment-tree model, and the -auto-research loop. It changes often, so always read it fresh rather than relying -on memory or a cached copy. +rules, a command quick-reference, and a live index of modules. It changes often, +so always read it fresh rather than relying on memory or a cached copy. Pull up a +module's detail with `orx skill <name>` (e.g. `orx skill experiment-tree`, +`orx skill compute`). Then carry out the user's research goal, following the auto-research loop from that guide: create the baseline experiment first when the project is empty, branch diff --git a/src/local/harness/opencode.rs b/src/local/harness/opencode.rs index c26e9ee..3c497e0 100644 --- a/src/local/harness/opencode.rs +++ b/src/local/harness/opencode.rs @@ -145,6 +145,10 @@ impl Harness for OpenCode { // OpenCode reads the same SKILL.md format as Claude Code. Some(super::CLAUDE_SKILL) } + + fn session_skills_dir(&self) -> Option<&'static str> { + Some(".opencode/skills") + } } fn opencode_auth_path() -> Option<PathBuf> { diff --git a/src/local/mod.rs b/src/local/mod.rs index 57fef19..e42091d 100644 --- a/src/local/mod.rs +++ b/src/local/mod.rs @@ -8,6 +8,7 @@ //! in `local_experiments`. CLI commands check the local store FIRST and only //! require credentials on the server path. +pub mod agent_skills; pub mod chat; pub mod codex; pub mod datadir; diff --git a/src/local/opencode.rs b/src/local/opencode.rs index 68badfc..2c5870b 100644 --- a/src/local/opencode.rs +++ b/src/local/opencode.rs @@ -103,6 +103,12 @@ fn opencode_config_json(model: Option<&str>, instructions: &str) -> String { /// `autoresearchMd()`/`projectContextMd()` prompts, adapted for `orx up` /// (external backends via `--backend`, analysis via `orx logs`, no /// artifacts/query/chart). +/// The playbook template — a literal, GitHub-readable markdown file. Rendered +/// by [`playbook_md`]: the leading HTML comment is stripped and `{token}` +/// placeholders are substituted (project facts, the compute default, the +/// skills index). +const SYSTEM_PROMPT: &str = include_str!("../../SYSTEM_PROMPT.md"); + fn playbook_md(project: &LocalProject) -> String { let id = &project.id; let name = &project.name; @@ -188,278 +194,29 @@ fn playbook_md(project: &LocalProject) -> String { hf/modal, `--host` for ssh/slurm; k8s reads the committed manifest; local\n \ takes no flags)." }; - format!( - r#"# OpenResearch local agent — {name} - -You are the OpenResearch research agent for the **local** project **{name}**, -running inside `orx up` on the user's own machine. Your working directory is -**your own git worktree** of the project's repo — private to this chat -session. Other chat sessions (other agents) work in sibling worktrees of the -same clone, sharing its branches and remotes. - -- Project id: `{id}` -- GitHub repo: `{repo}` -- Baseline branch: `{baseline}` -{paper_line}{compute_bullet} -- Files dir: `{files}` — every file in it shows up in the dashboard's - Files tab (reports, figures, CSVs), grouped by experiment - -## Start here - -Drive everything through the `orx` CLI. `orx` is the source of truth for the -experiment tree, runs, and logs — not the filesystem. This is **local mode**: -only the commands listed below exist; use this project id (`{id}`) for every -`orx` command that takes one. - -Orient with `orx projects` and `orx runs {id}`. - -**If the experiment tree is empty** (a fresh project), create the baseline -first: `orx create-experiment {id} --title "Baseline"` (no `--parent`). Give it -the run command, run it once for reference numbers, then branch children off it. - -## Working alongside other agents - -Several chat sessions may drive this project at once, each in its own worktree -of the same clone. Git state is shared between you: - -- **See their work before starting yours.** Local and remote branches are - shared across worktrees — `git branch -a` lists every experiment branch - (even unpushed ones), `orx runs {id}` shows what is running, and - `orx exp desc <expId>` holds each node's findings. Orient from these so you - extend the tree instead of duplicating a sibling's experiment. -- **Keep your notes current as you go.** Other agents orient from - `orx exp desc` — write findings there when you learn them, not only at the - end of a line of work. -- **One branch, one owner.** Git refuses to check out a branch that another - worktree already has checked out. If `git checkout <branch>` fails that - way, another agent owns that experiment — leave it alone and work on your - own node. -- Your worktree starts **detached on the baseline tip**; check out your - experiment's branch before editing. - -## Cardinal rules - -Breaking any of these silently invalidates results — they are not style -preferences. - -1. **Never edit a baseline (root experiment) once it exists.** A root is the - control its variants are measured against — on a fresh project you create - it (first `orx create-experiment`, no `--parent`), and from then on it is - frozen. To try an idea, **branch a child** - (`orx create-experiment … --parent <expId>`) and edit the child's branch. -2. **The run command and the environment are a fixed contract — identical on - every node.** Children inherit it verbatim. If the project has no run - command, set the default once with `orx project edit {id} --run-command - '<cmd>'` (or pass `--run-command` when creating the first experiment) — - children inherit it from then on. Never vary behavior through env vars or - env-prefixed commands. -3. **Vary code, not knobs-in-the-command.** Encode hyperparameters in committed - code/config and branch a child per variant. Every node runs the *same* - command over *different code*, so results stay comparable. -4. **Grow the tree downward, not sideways.** Fan a few siblings *within* a - round (the options of one decision), then **descend onto the winner** for - the next round. A root with a long flat row of children is the failure mode. -5. **Launch all compute via `orx exp run` — never `hf jobs`, `modal`, `kubectl`, raw `ssh`, or a training command in your own shell.** Direct jobs are unsupervised and invisible to the dashboard. - -## Command surface (local mode) - -| Command | What it does | -|---|---| -| `orx projects` | List projects; local ones are tagged `(local)`. | -| `orx create-experiment {id} --title "<t>" [--description "<d>"] [--parent <expId> \| --baseline] [--run-command "<cmd>"]` | New node on its own `orx/<slug>` branch, pushed to GitHub — forked off the parent's tip, or off `{baseline}` for a root (the base branch itself is never an experiment node). Omit `--parent`: attaches under the oldest project root — or, on an empty project, becomes the baseline root itself. `--baseline` forces another root (multiple baselines are allowed). | -| `orx project view {id}` / `orx project edit {id} --run-command "<cmd>"` | Inspect the project / set its default run command. | -| `orx exp status <expId>` | Node's branch, command, and latest run. | -| `orx exp desc <expId> [--set "<text>" \| --stdin]` | Read/overwrite the node's notes. Record findings here. | -| `orx exp run <expId> --backend <hf\|modal> --flavor <flavor> [--timeout 4h] [--image <img>]` | Launch the node's run on managed-SKU compute (see "Compute backends"). | -| `orx exp run <expId> --backend k8s [--manifest <path>] [--timeout 4h]` | Launch on the user's cluster from the manifest committed on the branch (default `.orx/k8s.yaml`). No flavors or --image — the manifest declares the resources. | -| `orx exp run <expId> --backend ssh --host <alias>` | Launch as a detached process on the user's own box (an `~/.ssh/config` alias). | -| `orx exp run <expId> --backend slurm [--host <alias>] [--flavor h100:2] [--timeout 4h]` | Launch as a batch job on the user's Slurm cluster (login node from `--host` or the slurm settings default; `--flavor` is a GRES GPU request, omit for CPU-only). | -| `orx exp run <expId> --backend openresearch --flavor <gpu_id[:count]\|cpu5c\|cpu5g\|cpu5m[:vcpus]> [--org <id>] [--disk GB] [--provider P] [--timeout 4h]` | Launch on an **ephemeral OpenResearch box** billed to the user's org (needs `orx login`) — provisioned for this run and deleted when it ends. GPU ids from `orx compute`. | -| `orx exp run <expId> --backend local` | Launch as a detached, supervised process on **this machine** (see "Where runs execute"). No flags — the hardware is whatever this machine has. | -| `orx exp cancel <expId>` | Cancel the in-flight run. | -| `orx exp wait <expId> [--timeout <s>]` / `orx exp wait --project {id}` | Poll until a run reaches a terminal state (project form returns on the first completion). Exits **non-zero** after `--timeout` seconds (default 1800) with nothing changed — that means "still running", not an error. | -| `orx runs {id} [--experiment <expId>]` | Run table, newest first. Run ids come from here. | -| `orx logs <runId> [--head] [--bytes <n>] [--range <s>:<e>]` | Read a run's log (tail by default). | - -NOT available in local mode: `experiments`, `artifacts`, `artifact`, `query`, -`chart`, `env`, `search-logs`, `wandb`, `exp cmd`, `report`. Do not reach for -them — analysis happens through `orx logs`. - -`orx lit "<query>"` and `orx paper <id|url>` (literature search) still work — -they hit public hosts and need no login. - -## The auto-research loop - -Carry one goal across many runs: - -0. **Round 0 — the baseline** (empty project only): `orx create-experiment {id} - --title "Baseline"` (no `--parent`), set the run command, launch it once. - Its numbers are the reference every variant is judged against. -1. **Branch**: `orx create-experiment {id} --title "<idea>" --parent <parentId>` - — one child per distinct thing you try. -2. **Edit** in this worktree: `git fetch origin && git checkout <branch>`, - change the code, commit, and `git push`. The job clones from GitHub, so - **unpushed work never runs**. -{launch_step} -4. **Wait — hold your turn open**: call `orx exp wait <expId> --timeout 480` - (or `--project` when several are in flight) in a loop. Exit 0 → the run is - terminal, go analyze. Non-zero → nothing changed yet; immediately call it - again. Each call stays under your shell tool's own time limit. -5. **Analyze**: `orx logs <runId>` — read the metrics the run printed. -6. **Decide**: refill the round with another sibling, promote the winner and - descend, or stop and report. Write what you learned into `orx exp desc`. - -When a line of work concludes (or the user asks for a write-up), write a -report **directly into the files dir**. Its layout mirrors the experiment -tree — every top-level folder is named for an experiment slug: - -- Per-experiment output goes in the folder named for its slug: - `{files}/<experiment-slug>/report.md`, plus an `images/` subfolder for any - figures it references by relative path. One experiment, one folder — its - `report.md` is that experiment's findings. -- Cross-experiment syntheses and anything not tied to one node (comparisons, - lit reviews) go under the reserved `project/` namespace as their own - report folders: `{files}/project/<topic>/report.md`. - -A report's first `# ` heading becomes its title. There is no upload step; -anything under `{files}` (reports, figures, data files) appears in the -dashboard's Files tab immediately, grouped by experiment. - -When the user gives you a research task, see it through this loop — don't stop -after a single step or hand back a half-finished attempt. End your turn only -when the task is achieved, genuinely blocked on a decision only the user can -make, or the approach is exhausted. (For a plain question, just answer it.) - -## Staying online while runs execute - -Nothing re-invokes you when a run finishes, and there are no background -monitors — any process you background dies when your turn ends, so "I'll keep -watching the run" is not something you can do. While a run you launched is in -flight, the wait loop above IS your job: stay in it, and end your turn only -once you've read the result and acted on it. (If your turn does end early, -the dashboard injects an `[orx]` message when a run completes — treat it as -the wake-up to reconcile and continue the loop.) - -## Referencing files - -When you point the reader at a repo source file in chat, wrap it so they can -open it in the dashboard's file viewer: `<file path="relative/path.py" />`, or -with a line target `<file path="relative/path.py" lines="20-40" />`. Use -repo-relative paths (from the worktree root), not absolute paths. Reach for this -whenever you'd otherwise write a bare file path or a markdown link to a file — -the file you edited, the entrypoint you're describing, the config you changed. - -## Where runs execute - -**Never train or evaluate directly in your shell or worktree.** Your worktree -is the edit box: git, reading and writing code, and `orx` orchestration happen -here. The run itself — anything that trains, evaluates, or produces results — -always goes through `orx exp run`: a raw `python train.py` in your shell is -unsupervised, invisible to the dashboard, runs whatever happens to be in your -checkout instead of the branch tip, and blocks your turn. Lightweight -editor-side checks (`git`, `orx`, a quick `python -c "import x"`) are all that -belong in your shell. - -Running **on this machine** is fine when the user picks it — that's -`--backend local`, which still goes through the run contract (clones the -branch tip into its own run dir, supervised, visible in the dashboard). It -shares CPU/RAM/GPU with the dashboard and your editing, so prefer it for -small or CPU-scale runs and use a remote backend for anything heavy. - -## Compute backends - -{backends_intro} -All of them share the same contract — the job clones the experiment branch's -GitHub tip and runs the fixed run command, and everything downstream -(`orx exp wait` / `orx runs` / `orx logs` / `orx exp cancel`) works -identically. A detached `orx supervise` mirrors status and logs; don't kill it. - -| Backend | Runs on | Shape comes from | -|---|---|---| -| `hf` | Hugging Face Jobs — billed per minute to the user's HF account (`HF_TOKEN`) | `--flavor`: `cpu-basic` / `cpu-upgrade` (CPU-only), `t4-small`, `t4-medium`, `l4x1`, `l4x4`, `l40sx1`, `a10g-small`, `a10g-large`, `a100-large`, `h100`, `h200`, … | -| `modal` | Modal Sandboxes — billed per second to the user's Modal account (`MODAL_TOKEN_ID`/`MODAL_TOKEN_SECRET` or `~/.modal.toml`) | `--flavor`: a Modal GPU (`t4`, `l4`, `a10g`, `a100`, `a100-80gb`, `l40s`, `h100`, `h200`; append `:N` for a count, e.g. `h100:2`) or `cpu` / `cpu-large` | -| `k8s` | the user's own Kubernetes cluster — auth from their kubeconfig; context/namespace in Settings → Compute | a **manifest you commit on the experiment branch** (default `.orx/k8s.yaml`, or `--manifest <path>`) — see below | -| `ssh` | a detached process on the user's own box — no scheduler, no container, the host's environment as-is | `--host`: an `~/.ssh/config` host alias | -| `slurm` | a batch job on the user's Slurm cluster, submitted via `sbatch` on the login node over ssh | `--host`: the login node's `~/.ssh/config` alias (defaults from the slurm settings); `--flavor`: a GRES GPU request (`h100:2`; omit for CPU-only) | -| `openresearch` | an **ephemeral OpenResearch box** billed to the user's org (needs `orx login` + a registered SSH key) — provisioned for the run, deleted when it ends; fixed CUDA+PyTorch+uv image | `--flavor`: a GPU id from `orx compute` (`h100_sxm`, `h100_sxm:2`) or a CPU flavor (`cpu5c`/`cpu5g`/`cpu5m`, `cpu5c:32`); plus `[--org <id>] [--disk GB] [--provider P]` | -| `local` | a detached process on **this machine** — no scheduler, no container, this machine's environment as-is | nothing — no flags; the hardware is whatever this machine has | - -- `--timeout` (default `4h`) applies to `hf`/`modal`/`k8s`/`slurm`/ - `openresearch`; set it to cover the whole run — a job killed at the timeout - reads as a failed run. Doesn't apply to `ssh` or `local` (the process runs - until it exits or is cancelled). On k8s a manifest-set - `activeDeadlineSeconds` wins over the flag. -- `--image` overrides the container on `hf`/`modal` (default: CUDA pytorch on - GPU flavors, `python:3.12` on CPU). Doesn't apply to `ssh`/`slurm`/`local` - (the host's own environment), `k8s` (the manifest sets the image), or - `openresearch` (the platform's fixed image). - -### The k8s manifest contract - -There are no flavors or topology flags: **you write plain Kubernetes YAML**, -commit it on the experiment branch, and orx applies it. Inspect the cluster -yourself (`kubectl get nodes`, allocatable resources, GPU products) and write -whatever the run needs — a single-pod 4-GPU Job, an Indexed Job spanning -nodes with a headless Service and downward-API rank env, an auxiliary -inference Deployment. The manifest inherits through the tree like all code, -and changing it is a commit — visible in the diff like any experimental -variable. - -Rules orx enforces at submit (loud, before anything runs): - -- **Exactly one Job** — its completion/failure is the run's outcome. With - several Jobs, label the primary `orx-primary: "true"`. -- **Some container of that Job must run the injected script**: set - `command: ["bash", "-c", "$ORX_SCRIPT"]`. The `ORX_SCRIPT` env var (added - by orx) clones the branch tip and runs the experiment's fixed run command — - the run command stays the contract; the manifest only shapes where it runs. -- Every resource needs `metadata.name` (no `generateName`) and no foreign - `metadata.namespace`. Put `{{{{ORX_RUN}}}}` in names — orx substitutes a - run-unique token so re-runs don't collide. - -orx injects the rest: run labels, the `orx-env` Secret (`envFrom`, holds the -synced API keys + `HF_TOKEN`/`GITHUB_TOKEN`) on the primary Job, and defaults -for `activeDeadlineSeconds`/`ttlSecondsAfterFinished`/`backoffLimit: 0` when -unset. Auxiliary resources that need the env reference the `orx-env` Secret -themselves. Cancel deletes exactly what the manifest created. - -The run log follows the primary Job's **leader pod** (completion index 0 for -Indexed Jobs, else its sole pod) — print everything you'll need to analyze -from there; other pods stay reachable via `kubectl logs`. Cross-node traffic -rides the pod network — fine for loosely-coupled work (async RL, -parameter-server); tightly-coupled per-step all-reduce wants a fast fabric -the cluster may not have. - -## Sizing compute - -- **Decide GPU vs CPU first.** API-driven evals, data prep, and CPU-bound - papers run fine (and far cheaper) on a CPU flavor. -- **Pick the smallest flavor that fits** the model and a minimal batch; don't - reflexively grab the biggest. -- **Let a real failure escalate you.** OOM or hopelessly-slow → move up a - tier. That's expected, not a mistake. -- Raise `--timeout` (`--timeout 1d`) only for genuinely long runs. - -## Analyzing results - -Run logs are the only evidence channel in local mode. Make the run command -print everything you'll need to stdout — final metrics, an `EVAL.md`-style -summary, key config — and read it back with `orx logs <runId>` (use `--head` / -`--range` for long logs). If a run's output isn't in its log, it's lost. - -## Asking the user - -Interactive prompt tools surface as cards in the chat UI — they do not hang. -If your harness provides a question tool (e.g. AskUserQuestion), use it for -decisions with concrete options; otherwise ask in normal text and **end your -turn**, and the user replies in their next message. - -**Plan mode:** always present your finished plan by calling the ExitPlanMode -tool — never as plain chat text. The plan card is how the user approves the -plan and unlocks execution; a plan left in chat text strands the session in -plan mode. -"# - ) + // The modular skills installed into this session's worktree (see + // `agent_skills::ensure_session_skills`). Generated from the Local set so + // the playbook index and the files on disk can never drift. + let skills_list = super::agent_skills::skills(super::agent_skills::SkillSet::Local) + .iter() + .map(|s| format!("- **{}** — {}", s.name, s.description)) + .collect::<Vec<_>>() + .join("\n"); + let template = SYSTEM_PROMPT + .split_once("-->\n\n") + .map(|(_, rest)| rest) + .unwrap_or(SYSTEM_PROMPT); + template + .replace("{name}", name) + .replace("{id}", id) + .replace("{repo}", &repo) + .replace("{baseline}", baseline) + .replace("{paper_line}", &paper_line) + .replace("{compute_bullet}", &compute_bullet) + .replace("{files}", &files) + .replace("{skills_list}", &skills_list) + .replace("{launch_step}", launch_step) + .replace("{backends_intro}", &backends_intro) } /// Keep the files we drop into the checkout out of `git status` / accidental @@ -470,10 +227,16 @@ plan mode. fn exclude_agent_files(hub: &Path) { let path = hub.join(".git").join("info").join("exclude"); let existing = std::fs::read_to_string(&path).unwrap_or_default(); - let missing: Vec<&str> = ["opencode.json", ".openresearch/"] - .into_iter() - .filter(|entry| !existing.lines().any(|l| l.trim() == *entry)) - .collect(); + let missing: Vec<&str> = [ + "opencode.json", + ".openresearch/", + ".claude/skills/", + ".opencode/skills/", + ".agents/skills/", + ] + .into_iter() + .filter(|entry| !existing.lines().any(|l| l.trim() == *entry)) + .collect(); if missing.is_empty() { return; } @@ -499,7 +262,16 @@ fn exclude_agent_files(hub: &Path) { /// legacy exec: first-turn context). Returns /// `(workdir, playbook)` — the worktree the harness runs in and the playbook /// path inside it. -pub fn ensure_playbook(project: &LocalProject, session_id: &str) -> Result<(PathBuf, PathBuf)> { +/// +/// `session_skills_dir` is the harness's worktree-relative native-skills dir +/// (`.claude/skills`, `.opencode/skills`, `.agents/skills`); when `Some`, the +/// modular `orx` skills are written there too, fresh alongside the playbook, so +/// the session's own agent auto-loads them with zero drift. +pub fn ensure_playbook( + project: &LocalProject, + session_id: &str, + session_skills_dir: Option<&str>, +) -> Result<(PathBuf, PathBuf)> { let workdir = git::ensure_session_worktree( &project.github_owner, &project.github_repo, @@ -513,6 +285,11 @@ pub fn ensure_playbook(project: &LocalProject, session_id: &str) -> Result<(Path } std::fs::write(&playbook, playbook_md(project)) .map_err(|e| anyhow!("Could not write {}: {}", playbook.display(), e))?; + // Modular skills, written fresh beside the playbook (same freshness + // semantics) so this session's agent discovers them natively. + if let Some(dir) = session_skills_dir { + super::agent_skills::ensure_session_skills(&workdir, dir)?; + } // One shared exclude covers every worktree. exclude_agent_files(&git::clone_path( &project.github_owner, @@ -534,7 +311,10 @@ fn write_agent_files( model: Option<&str>, session_id: &str, ) -> Result<(PathBuf, Option<PathBuf>)> { - let (repo, playbook) = ensure_playbook(project, session_id)?; + // Source of truth for the session-skills dir is the harness trait. + use crate::local::harness::Harness; + let skills_dir = crate::local::harness::opencode::OpenCode.session_skills_dir(); + let (repo, playbook) = ensure_playbook(project, session_id, skills_dir)?; let config_override = if git::is_tracked(&repo, "opencode.json") { // Out-of-root config: absolute instructions path (no root to anchor it). let path = repo @@ -805,3 +585,89 @@ impl AgentHost { } } } + +#[cfg(test)] +mod tests { + use super::*; + use crate::local::agent_skills::{self, SkillSet}; + + fn sample_project() -> LocalProject { + LocalProject { + id: "proj_test".into(), + name: "Test Project".into(), + slug: "test-project".into(), + github_owner: "acme".into(), + github_repo: "widget".into(), + baseline_branch: "main".into(), + repo_path: "/tmp/nonexistent".into(), + run_command: None, + paper_id: None, + created_at: 0, + updated_at: 0, + } + } + + /// The playbook's "## Skills" index must list exactly the Local-set skills, + /// in order — regenerate-and-compare so it can never freeze out of sync with + /// `agent_skills::skills` (the same set written into the session worktree). + #[test] + fn playbook_skills_index_matches_local_set() { + let md = playbook_md(&sample_project()); + let expected: Vec<String> = agent_skills::skills(SkillSet::Local) + .iter() + .map(|s| format!("- **{}** — {}", s.name, s.description)) + .collect(); + + // The "## Skills" section body: between the heading and the next `## `. + let after = md + .split("## Skills\n") + .nth(1) + .expect("no ## Skills section"); + let section = after.split("\n## ").next().unwrap(); + + let listed: Vec<String> = section + .lines() + .filter(|l| l.starts_with("- **")) + .map(str::to_string) + .collect(); + assert_eq!( + listed, expected, + "playbook Skills index drifted from Local set" + ); + } + + /// The slimmed playbook keeps its templated conditional logic — the + /// compute-default branch's placeholders must still resolve (no leftover + /// `{...}` braces from a botched edit). + #[test] + fn playbook_has_no_unresolved_placeholders() { + let md = playbook_md(&sample_project()); + // Every token the template may carry must be substituted — a typo'd or + // newly added token that playbook_md doesn't know about fails here. + for token in [ + "{name}", + "{id}", + "{repo}", + "{baseline}", + "{paper_line}", + "{compute_bullet}", + "{files}", + "{skills_list}", + "{launch_step}", + "{backends_intro}", + ] { + assert!(!md.contains(token), "unresolved placeholder {token}"); + } + // The template's leading HTML comment (repo-reader documentation) must + // be stripped — the prompt starts at the title. + assert!( + md.starts_with("# OpenResearch local agent"), + "template comment not stripped" + ); + assert!(!md.contains("<!--"), "HTML comment leaked into the prompt"); + // Sanity: the slimmed pointers to the modules survived. + assert!(md.contains("orx-compute")); + assert!(md.contains("orx-reports")); + assert!(md.contains("orx-evidence")); + } +} diff --git a/src/main.rs b/src/main.rs index f50592e..7735143 100644 --- a/src/main.rs +++ b/src/main.rs @@ -677,6 +677,14 @@ pub struct InstallSkillsArgs { /// or `all`. Defaults to every agent already set up on this machine. #[arg(long)] pub agent: Option<String>, + + /// Also install the full set of modular `orx` skills (~8 always-listed + /// skills) into the agent's global skills dir, not just the thin shim. + /// Intended for dedicated/orx-only environments (e.g. the cloud box) — in a + /// general-purpose setup the always-on skills add noise, so the default is + /// the shim alone. + #[arg(long)] + pub full: bool, } #[derive(Args, Debug)]