diff --git a/CHANGELOG.md b/CHANGELOG.md index 027088e..ed1ee3d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,19 @@ All notable changes to RushTI are documented in this file. +## Unreleased — docs: `--mode` for cube reads (#160) + +- **Docs fix:** corrected the long-standing claim that `--mode` is + deprecated/ignored. It is only auto-detected (and ignored) for **file + sources** (`--tasks`). A **cube read** (`--tm1-instance`) cannot infer + the mode — every workflow occupies the same cube measures — so it + defaults to `norm` and silently drops the `predecessors` measure unless + `--mode opt` is passed. This caused predecessors to disappear from + cube-read execution plans (e.g. `Sample_Optimal_Mode`). Updated the CLI + reference, settings reference, migration guide, TM1 integration guide + (new "Choosing the mode for cube reads" section), getting-started + task-files page, and the `rushti run --help` text. + ## Unreleased — `feat/issue-154-v12-load-results` - Fix: `rushti build` now installs a TM1-version-aware `}rushti.load.results` diff --git a/docs/advanced/cli-reference.md b/docs/advanced/cli-reference.md index 6a14ddb..6d235d8 100644 --- a/docs/advanced/cli-reference.md +++ b/docs/advanced/cli-reference.md @@ -55,7 +55,7 @@ rushti --tasks FILE [options] # 'run' is the default command | `--retries` | `-r` | INT | `0` | Retry count for failed TI executions. Uses exponential backoff. | | `--result` | `-o` | PATH | *(empty)* | Output CSV path for execution summary. Omit to skip CSV creation. | | `--settings` | `-s` | PATH | auto | Path to `settings.ini`. Auto-discovered if omitted. | -| `--mode` | `-m` | CHOICE | auto | **Deprecated.** Mode is auto-detected from file content. Ignored. | +| `--mode` | `-m` | CHOICE | `norm` | Execution mode: `norm` or `opt`. **Ignored for file sources** (`--tasks`), where mode is auto-detected from file content. **Required for cube reads** (`--tm1-instance`) when the workflow uses explicit `predecessors` — pass `--mode opt`, otherwise the cube is read in `norm` mode and predecessors are dropped. See [TM1 integration → Choosing the mode for cube reads](../features/tm1-integration.md#choosing-the-mode-for-cube-reads). | | `--exclusive` | `-x` | FLAG | `false` | Enable exclusive mode. Waits for other RushTI sessions to finish. | | `--force` | `-f` | FLAG | `false` | Bypass exclusive mode checks and run immediately. | | `--optimize` | | CHOICE | *(none)* | Enable task optimization with a scheduling algorithm: `longest_first` or `shortest_first`. | @@ -80,9 +80,12 @@ rushti run --tasks critical.json --exclusive --result results/critical.csv # Override workflow name for a file-based run rushti run --tasks daily-etl.json --workflow DailyETL --max-workers 8 -# Read task file from TM1 cube +# Read task file from TM1 cube (norm mode — wait-based sequencing) rushti run --tm1-instance tm1srv01 --workflow DailyETL --max-workers 8 +# Read an optimized workflow from the cube — --mode opt is required to honour predecessors +rushti run --tm1-instance tm1srv01 --workflow DailyETL --mode opt --max-workers 8 + # Optimize with shortest-first scheduling (good for shared-resource TM1 workloads) rushti run --tasks tasks.json --max-workers 20 --optimize shortest_first diff --git a/docs/advanced/migration-guide.md b/docs/advanced/migration-guide.md index b276e02..9c8b328 100644 --- a/docs/advanced/migration-guide.md +++ b/docs/advanced/migration-guide.md @@ -62,16 +62,19 @@ Logging is configured separately via `config/logging_config.ini` (Python's stand All settings have sensible defaults. You do not need to create a `settings.ini` to use v2.0 -- it works with built-in defaults. -### --mode Flag: Deprecated +### --mode Flag: Auto-Detected for Files, Still Required for Cube Reads In v1.x, the `--mode` flag (`norm` or `opt`) controlled whether RushTI used level-based (wait) or dependency-based execution. -In v2.0, mode is **auto-detected** from file content: +In v2.0, mode is **auto-detected from file content** for file sources (`--tasks`): - JSON files always use DAG execution (optimized mode). - TXT files use `norm` mode if they contain `wait` keywords, or `opt` mode if tasks have `id` and `predecessors` fields. -The `--mode` flag is accepted for backward compatibility but ignored. +For file sources the `--mode` flag is therefore accepted for backward compatibility but ignored. + +!!! warning "Cube reads still need `--mode`" + Auto-detection does **not** apply when reading from a TM1 cube (`--tm1-instance`). The cube stores every workflow with the same measures, so RushTI cannot tell a wait-based workflow from a predecessor-based one. Cube reads default to `norm` (wait-based) and **ignore the `predecessors` measure** unless you pass `--mode opt`. See [TM1 integration → Choosing the mode for cube reads](../features/tm1-integration.md#choosing-the-mode-for-cube-reads). ### Task File: id and predecessors in TXT Files @@ -413,9 +416,9 @@ rushti run --tasks tasks.json --log-level DEBUG Check that `max_workers` is set in the correct place with the correct spelling. -### Issue: Mode Deprecation Warning +### Issue: Predecessors Missing When Running from a Cube -If you see a deprecation warning about `--mode`, it is safe to ignore. Remove `--mode` from your scripts when convenient -- mode detection is automatic. +If you run an optimized workflow from the cube (`--tm1-instance`) and the resulting plan ignores your `predecessors`, you are running in the default `norm` mode. Add `--mode opt` to the command (or set `mode = opt` in `settings.ini`). Mode auto-detection applies to file sources only — see [TM1 integration → Choosing the mode for cube reads](../features/tm1-integration.md#choosing-the-mode-for-cube-reads). For `--tasks` file sources you can safely omit `--mode` entirely. ### Issue: TXT File Not Auto-Detecting Mode diff --git a/docs/advanced/settings-reference.md b/docs/advanced/settings-reference.md index c43ea87..5047560 100644 --- a/docs/advanced/settings-reference.md +++ b/docs/advanced/settings-reference.md @@ -46,9 +46,9 @@ Common execution settings that control basic RushTI behavior. | `max_workers` | int | `4` | Maximum number of parallel workers. Valid range: 1--100. Recommended: start at 4, increase based on TM1 server capacity. | | `retries` | int | `0` | Number of retry attempts for failed TI process executions. Valid range: 0--10. Retries use exponential backoff (1s, 2s, 4s, ...). | | `result_file` | str | `""` (empty) | CSV output path for execution summary. Empty string means no CSV is created. | -| `mode` | str | `norm` | **Deprecated.** Execution mode is now auto-detected from file content. JSON files always use DAG execution; TXT files use the mode indicated by their content structure. Kept for backward compatibility only. | +| `mode` | str | `norm` | Default execution mode (`norm` or `opt`) for **cube reads** (`--tm1-instance`). A cube read cannot auto-detect its mode, so this value (or the `--mode` CLI override) decides whether the `predecessors` measure is honoured (`opt`) or the `wait` measure drives sequencing (`norm`). **Ignored for file sources**, where the mode is auto-detected from file content. See [Choosing the mode for cube reads](../features/tm1-integration.md#choosing-the-mode-for-cube-reads). | -**Overridable via CLI:** `--max-workers` / `-w`, `--retries` / `-r`, `--result` / `-o`, `--mode` / `-m` (deprecated) +**Overridable via CLI:** `--max-workers` / `-w`, `--retries` / `-r`, `--result` / `-o`, `--mode` / `-m` **Overridable via JSON task file:** `max_workers`, `retries`, `result_file` @@ -243,7 +243,10 @@ Copy this template to `config/settings.ini` and uncomment the settings you want # Default: (empty - no CSV created) # result_file = rushti.csv -# Execution mode (deprecated - auto-detected from file content) +# Default execution mode for cube reads (--tm1-instance): norm or opt. +# Cube reads cannot auto-detect the mode; set opt here if most of your +# cube-stored workflows use explicit predecessors. Ignored for file +# sources (--tasks), where the mode is auto-detected from file content. # Default: norm # mode = norm diff --git a/docs/features/tm1-integration.md b/docs/features/tm1-integration.md index 3d9caca..b2683a5 100644 --- a/docs/features/tm1-integration.md +++ b/docs/features/tm1-integration.md @@ -69,6 +69,17 @@ rushti run --tm1-instance tm1srv01 --workflow Sample_Stage_Mode --max-workers 4 RushTI reads the task definitions from the cube, builds the execution plan, and runs them — exactly like running from a file. +!!! warning "Optimized workflows need `--mode opt`" + When you run from the cube, RushTI **cannot auto-detect** whether the workflow is wait-based (`norm`) or predecessor-based (`opt`) — the cube row layout is identical either way. It therefore defaults to `norm`, which uses the `wait` measure for sequencing and **ignores the `predecessors` measure entirely**. + + To run an optimized workflow (one that defines explicit `predecessors`), you must pass `--mode opt`: + + ```bash + rushti run --tm1-instance tm1srv01 --workflow Sample_Optimal_Mode --mode opt --max-workers 4 + ``` + + Without `--mode opt`, the predecessors silently disappear from the execution plan (RushTI logs a `predecessors ignored in norm mode` warning, but the run still proceeds). See [Choosing the mode for cube reads](#choosing-the-mode-for-cube-reads) below. + ### View Results in the Cube Once the run completes, RushTI writes the results back to the same cube under a timestamped `rushti_run_id` element. Open the `rushti_results` subset to see execution data: @@ -180,14 +191,41 @@ rushti run --tm1-instance tm1srv01 --workflow Sample_Normal_Mode --max-workers 4 # Run the stage-based sample rushti run --tm1-instance tm1srv01 --workflow Sample_Stage_Mode --max-workers 4 -# Run the optimized sample (with explicit dependencies) -rushti run --tm1-instance tm1srv01 --workflow Sample_Optimal_Mode --max-workers 4 +# Run the optimized sample (with explicit dependencies) — note --mode opt +rushti run --tm1-instance tm1srv01 --workflow Sample_Optimal_Mode --mode opt --max-workers 4 ``` +`Sample_Optimal_Mode` defines its task ordering through the `predecessors` measure, so it **requires `--mode opt`**. Run it without that flag and RushTI reads the cube in `norm` mode, drops the predecessors, and falls back to wait-based sequencing — the run "works" but does not respect the dependency graph you defined. + Check the `rushti` cube afterward to confirm that results were written. --- +## Choosing the Mode for Cube Reads + +Unlike file sources — where the mode is auto-detected from the file's structure — a cube read **cannot infer the mode**. Every workflow occupies the same set of measures in the `rushti` cube (`wait`, `predecessors`, `instance`, `process`, …), so there is no structural signal that distinguishes a wait-based workflow from a predecessor-based one. You tell RushTI which one to use with `--mode`: + +| `--mode` | Sequencing driver | `wait` measure | `predecessors` measure | +|----------|-------------------|----------------|------------------------| +| `norm` (default) | `wait` markers create sequential groups | **Used** | **Ignored** (logs a warning if populated) | +| `opt` | Explicit `predecessors` form a DAG | Ignored | **Used** | + +```bash +# Wait-based (norm) workflow — the default, no flag needed +rushti run --tm1-instance tm1srv01 --workflow Sample_Normal_Mode --max-workers 4 + +# Predecessor-based (opt) workflow — --mode opt is required +rushti run --tm1-instance tm1srv01 --workflow Sample_Optimal_Mode --mode opt --max-workers 4 +``` + +!!! tip "Make `opt` the default if most of your cube workflows use predecessors" + If the majority of your cube-stored workflows are predecessor-based, set `mode = opt` in the `[defaults]` section of `settings.ini` so you don't have to pass `--mode opt` on every run. The CLI `--mode` flag still overrides it per run. + +!!! note "`--mode` and file sources" + For `--tasks` (JSON/TXT files) the mode is auto-detected from content and `--mode` is ignored. The flag only changes behaviour for cube reads (`--tm1-instance`). + +--- + ## Configuration ### Settings Reference diff --git a/docs/getting-started/task-files.md b/docs/getting-started/task-files.md index 404f217..51e0810 100644 --- a/docs/getting-started/task-files.md +++ b/docs/getting-started/task-files.md @@ -72,11 +72,14 @@ Each task is defined by writing values to these measures: ### Run Tasks from the Cube ```bash -rushti run --tm1-instance tm1srv01 --workflow daily-refresh --max-workers 4 +rushti run --tm1-instance tm1srv01 --workflow daily-refresh --mode opt --max-workers 4 ``` RushTI reads the task definitions from the cube, builds the DAG, and executes — exactly like running from a file. +!!! warning "`--mode opt` is required for predecessor-based workflows" + Because the workflow above uses the `predecessors` measure, you must pass `--mode opt`. A cube read cannot auto-detect its mode (unlike a file source) and defaults to `norm`, which ignores `predecessors` and sequences tasks by the `wait` measure instead. See [TM1 integration → Choosing the mode for cube reads](../features/tm1-integration.md#choosing-the-mode-for-cube-reads). + ### View Results When result pushing is enabled, RushTI writes execution results back to the same cube. Each run gets its own `rushti_run_id` element (e.g., `20260209_143022`), so you can see the full execution history: diff --git a/src/rushti/cli.py b/src/rushti/cli.py index fbf80c1..baa998b 100644 --- a/src/rushti/cli.py +++ b/src/rushti/cli.py @@ -249,11 +249,14 @@ def create_argument_parser() -> argparse.ArgumentParser: %(prog)s -t tasks.txt -w 4 -r 2 -o results.csv %(prog)s --tasks tasks.json --max-workers 8 --retries 3 %(prog)s --tm1-instance tm1srv01 --workflow Sample --max-workers 4 + %(prog)s --tm1-instance tm1srv01 --workflow Sample_Optimal_Mode --mode opt Configuration: Default values can be set in settings.ini (see settings.ini.template). Settings precedence: CLI args > JSON task file > settings.ini > defaults - File format is auto-detected (--mode is deprecated for file sources). + File sources auto-detect the mode (--mode is ignored). Cube reads + (--tm1-instance) cannot, and default to norm; pass --mode opt to honour + the 'predecessors' measure. """, ) @@ -286,7 +289,7 @@ def create_argument_parser() -> argparse.ArgumentParser: dest="execution_mode", choices=["norm", "opt"], default=None, - help="[Deprecated] Execution mode is now auto-detected from file content. This option is kept for backwards compatibility but ignored.", + help="Execution mode: norm or opt. Auto-detected (and ignored) for file sources. For cube reads (--tm1-instance) it defaults to norm; pass 'opt' to honour the 'predecessors' measure.", ) parser.add_argument(