Skip to content

Commit c1c3e80

Browse files
docs: fix volume export docs build and add parameter reference
Repair the malformed reStructuredText block in runtime architecture docs that caused the GitHub Docs workflow to fail with an unexpected-indentation error. Add a dedicated workflow reference page for volume_export, wire it into the workflows index, expand the runtime CLI/execution guide with the normalized parameter contract, and mirror the parameter reference in the visualization subsystem README.
1 parent b886607 commit c1c3e80

5 files changed

Lines changed: 266 additions & 4 deletions

File tree

docs/source/runtime/architecture-overview.rst

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -69,10 +69,9 @@ These contracts are stable and expected by multiple modules:
6969
``results/<analysis>/latest``.
7070
- Internal image-producing analysis arrays live under
7171
``clearex/runtime_cache/results/<analysis>/latest``.
72-
- ``volume_export`` is hybrid:
73-
- OME-Zarr runs publish ``results/volume_export/latest``
74-
- OME-TIFF runs keep artifacts under
75-
``clearex/results/volume_export/latest/files``
72+
- ``volume_export`` is hybrid: OME-Zarr runs publish
73+
``results/volume_export/latest``; OME-TIFF runs keep artifacts under
74+
``clearex/results/volume_export/latest/files``.
7675
- ClearEx-owned metadata, provenance, GUI state, and non-image artifacts live
7776
under ``clearex/...``.
7877
- Store-level placement metadata is persisted in

docs/source/runtime/cli-and-execution.rst

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -217,8 +217,17 @@ Volume Export
217217
``volume_export`` is a visualization-family export workflow for one selected
218218
image-producing source component.
219219

220+
- ``input_source`` resolves the upstream image-producing component to export.
221+
It accepts the usual runtime aliases such as ``data``, ``flatfield``,
222+
``deconvolution``, ``shear_transform``, ``fusion``, and explicit internal
223+
component paths.
224+
- ``force_rerun`` bypasses provenance-based reuse and rewrites the latest
225+
export payload even when matching latest metadata already exists.
220226
- ``export_scope=current_selection`` exports one explicit ``(t, p, c)`` volume.
221227
- ``export_scope=all_indices`` exports every available ``(t, p, c)`` volume.
228+
- ``t_index``, ``p_index``, and ``c_index`` select the exported volume when
229+
``export_scope=current_selection``. They are ignored for
230+
``export_scope=all_indices``.
222231
- ``resolution_level`` can reuse discovered source-adjacent pyramid levels or
223232
generate missing deeper levels during export.
224233
- ``export_format=ome-zarr`` publishes ``results/volume_export/latest`` from
@@ -229,10 +238,23 @@ image-producing source component.
229238
OME-TIFF or one all-position BigTIFF with one ``TCZYX`` series per position.
230239
- ``tiff_file_layout=per_volume_files`` writes one ``ZYX`` OME-TIFF per
231240
exported ``(t, p, c)`` volume.
241+
- ``memory_overhead_factor`` is preserved in the normalized workflow payload as
242+
a scheduler hint, even though the current export implementation does not use
243+
it directly.
244+
- ``chunk_basis``, ``detect_2d_per_slice``, ``use_map_overlap``, and
245+
``overlap_zyx`` are normalized compatibility fields inherited from other
246+
volume analyses and are pinned to 3D/no-overlap behavior for
247+
``volume_export``.
232248

233249
When a requested source component does not exist, runtime raises an input
234250
dependency error instead of silently falling back.
235251

252+
The CLI currently exposes ``--volume-export`` as the operation flag. Detailed
253+
parameter editing is currently done through the GUI or a programmatic
254+
``WorkflowConfig.analysis_parameters["volume_export"]`` payload. The full
255+
parameter-by-parameter reference lives in
256+
:doc:`../workflows/analysis-volume-export`.
257+
236258
Progress and Run Lifecycle
237259
--------------------------
238260

Lines changed: 217 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,217 @@
1+
.. _analysis-workflow-volume-export:
2+
3+
Analysis Workflow: Volume Export
4+
================================
5+
6+
Implemented behavior
7+
--------------------
8+
9+
This analysis path exports one selected image-producing ClearEx source component
10+
from a canonical OME-Zarr store into either an in-store public OME-Zarr result
11+
or in-store OME-TIFF artifacts.
12+
13+
``volume_export`` is part of the visualization workflow family, but it exports
14+
resolved runtime data rather than viewer metadata. The source can be any
15+
previously materialized image-producing component, including the canonical
16+
source data plus flatfield, deconvolution, shear transform, fusion, or other
17+
analysis outputs that resolve to canonical internal 6D arrays in
18+
``(t, p, c, z, y, x)`` order.
19+
20+
Default analysis parameter dictionary
21+
-------------------------------------
22+
23+
``WorkflowConfig`` carries ``analysis_parameters``, keyed by analysis method,
24+
with defaults in ``DEFAULT_ANALYSIS_OPERATION_PARAMETERS``.
25+
26+
All operation dictionaries include:
27+
28+
- ``execution_order`` (int): defines run order when multiple routines are
29+
selected. ``volume_export`` defaults to ``12`` so it runs after
30+
``compile_movie`` and before ``mip_export``.
31+
- ``input_source`` (str): logical source alias or explicit internal component
32+
path. The default is ``data``.
33+
- ``force_rerun`` (bool): bypasses provenance-based skip logic and reruns the
34+
export even when a matching latest result already exists. The default is
35+
``False``.
36+
37+
The ``volume_export`` defaults include:
38+
39+
- ``chunk_basis`` (str, default ``"3d"``): normalized execution-compatibility
40+
field inherited from other volume analyses. ``volume_export`` pins this to
41+
3D chunks and does not currently expose alternate chunk bases.
42+
- ``detect_2d_per_slice`` (bool, default ``False``): compatibility field
43+
inherited from chunked detection workflows. It is normalized for consistency
44+
but not used by ``volume_export``.
45+
- ``use_map_overlap`` (bool, default ``False``): compatibility field inherited
46+
from chunk-overlap planning. ``volume_export`` keeps overlap masking off.
47+
- ``overlap_zyx`` (list[int], default ``[0, 0, 0]``): compatibility field for
48+
overlap-aware chunk planning. ``volume_export`` does not use overlap padding.
49+
- ``memory_overhead_factor`` (float, default ``1.0``): normalized scheduler
50+
hint carried through the workflow payload. The current export implementation
51+
runs eagerly and does not consume this field directly, but it remains part of
52+
the persisted parameter set.
53+
- ``export_scope`` (str, default ``"current_selection"``): chooses between
54+
exporting one selected ``(t, p, c)`` volume or exporting all available
55+
indices. Accepted values are ``current_selection`` and ``all_indices``.
56+
- ``t_index`` (int, default ``0``): time index used only when
57+
``export_scope=current_selection``. Negative values are normalized to ``0``.
58+
- ``p_index`` (int, default ``0``): position index used only when
59+
``export_scope=current_selection``. Negative values are normalized to ``0``.
60+
- ``c_index`` (int, default ``0``): channel index used only when
61+
``export_scope=current_selection``. Negative values are normalized to ``0``.
62+
- ``resolution_level`` (int, default ``0``): source-adjacent pyramid level to
63+
export. ``0`` uses the base source component. Higher levels reuse existing
64+
display/source-adjacent levels when present and generate missing deeper levels
65+
on demand during export.
66+
- ``export_format`` (str, default ``"ome-zarr"``): chooses between public
67+
in-store OME-Zarr publication and in-store OME-TIFF artifacts. Accepted
68+
normalized values are ``ome-zarr`` and ``ome-tiff``; aliases such as
69+
``zarr`` and ``tiff`` normalize to those canonical names.
70+
- ``tiff_file_layout`` (str, default ``"single_file"``): TIFF-only policy that
71+
chooses one BigTIFF/OME-TIFF file versus one file per exported volume.
72+
Accepted values are ``single_file`` and ``per_volume_files``.
73+
74+
Execution sequence and upstream input behavior
75+
----------------------------------------------
76+
77+
- Runtime resolves ``input_source`` through the shared analysis-source alias
78+
resolver.
79+
- Common aliases include:
80+
81+
- ``data`` -> ``clearex/runtime_cache/source/data``
82+
- ``flatfield`` -> ``clearex/runtime_cache/results/flatfield/latest/data``
83+
- ``deconvolution`` -> ``clearex/runtime_cache/results/deconvolution/latest/data``
84+
- ``shear_transform`` -> ``clearex/runtime_cache/results/shear_transform/latest/data``
85+
- ``fusion`` -> ``clearex/runtime_cache/results/fusion/latest/data``
86+
- ``usegment3d`` -> ``clearex/runtime_cache/results/usegment3d/latest/data``
87+
88+
- Explicit internal component paths are also supported when they resolve to a
89+
canonical 6D image array.
90+
- If the requested source component is missing or is not a canonical 6D array,
91+
``volume_export`` raises an input dependency error instead of silently
92+
falling back.
93+
94+
Export scope and resolution behavior
95+
------------------------------------
96+
97+
- ``current_selection`` slices the resolved source to exactly one
98+
``(t, p, c, z, y, x)`` payload with singleton leading axes.
99+
- ``all_indices`` exports the full resolved ``(t, p, c, z, y, x)`` array for
100+
the selected source.
101+
- ``resolution_level=0`` uses the base source component directly.
102+
- Higher ``resolution_level`` values first reuse existing source-adjacent
103+
levels discovered for the selected source.
104+
- If the requested level is not already materialized, ``volume_export`` builds
105+
the missing deeper level(s) before writing the export payload.
106+
- Resolved voxel calibration is scaled to the selected level and persisted in
107+
metadata plus OME output headers.
108+
109+
Volume-export execution workflow
110+
--------------------------------
111+
112+
1. Normalize the ``volume_export`` parameter block.
113+
2. Resolve ``input_source`` to the canonical internal source component.
114+
3. Resolve or generate the requested ``resolution_level`` component.
115+
4. Validate the selected ``t_index``, ``p_index``, and ``c_index`` when
116+
``export_scope=current_selection``.
117+
5. Rewrite the export runtime-cache payload at
118+
``clearex/runtime_cache/results/volume_export/latest/data``.
119+
6. Materialize either:
120+
121+
- a public OME image collection under ``results/volume_export/latest`` when
122+
``export_format=ome-zarr``, or
123+
- OME-TIFF artifacts under
124+
``clearex/results/volume_export/latest/files`` when
125+
``export_format=ome-tiff``.
126+
127+
7. Rewrite latest export metadata under ``clearex/results/volume_export/latest``.
128+
8. Register the latest-output reference in provenance.
129+
130+
Storage and output contract
131+
---------------------------
132+
133+
The export always rewrites the latest runtime payload under:
134+
135+
- ``clearex/runtime_cache/results/volume_export/latest/data``
136+
137+
Latest metadata for the run is stored under:
138+
139+
- ``clearex/results/volume_export/latest``
140+
141+
The metadata group records:
142+
143+
- requested ``input_source``,
144+
- resolved source and resolution components,
145+
- ``export_scope``,
146+
- ``resolution_level``,
147+
- whether deeper levels were generated on demand,
148+
- ``export_format``,
149+
- ``tiff_file_layout``,
150+
- selected shape metadata,
151+
- resolved voxel size,
152+
- emitted artifact paths.
153+
154+
OME-Zarr output contract
155+
------------------------
156+
157+
- ``export_format=ome-zarr`` publishes a public OME image collection under
158+
``results/volume_export/latest``.
159+
- ``current_selection`` publishes one exported field at the selected
160+
resolution.
161+
- ``all_indices`` publishes every available exported field from the resolved
162+
source.
163+
- The public result reflects the selected export resolution; it does not
164+
publish a new multiscale pyramid for the export itself.
165+
166+
OME-TIFF layout policy
167+
----------------------
168+
169+
- ``current_selection`` always writes one ``ZYX`` OME-TIFF/BigTIFF file named
170+
from the selected ``t/p/c`` indices.
171+
- ``all_indices + single_file`` writes one OME-TIFF/BigTIFF file with one
172+
``TCZYX`` series per exported position.
173+
- ``all_indices + per_volume_files`` writes one ``ZYX`` OME-TIFF file per
174+
exported ``(t, p, c)`` volume.
175+
- TIFF artifacts are stored inside the analysis store under
176+
``clearex/results/volume_export/latest/files`` so they remain namespaced with
177+
the latest export metadata.
178+
- TIFF metadata preserves resolved ``PhysicalSizeZ/Y/X`` calibration.
179+
180+
CLI and GUI surface
181+
-------------------
182+
183+
- The CLI currently exposes the operation flag ``--volume-export``.
184+
- Detailed parameter editing is currently done through the GUI or a programmatic
185+
``WorkflowConfig.analysis_parameters["volume_export"]`` payload.
186+
- The visualization GUI exposes controls for:
187+
188+
- ``input_source``
189+
- ``export_scope``
190+
- ``t_index`` / ``p_index`` / ``c_index``
191+
- ``resolution_level``
192+
- ``export_format``
193+
- ``tiff_file_layout``
194+
195+
Provenance integration
196+
----------------------
197+
198+
- ``volume_export`` registers its latest output reference under
199+
``clearex/provenance/latest_outputs/volume_export``.
200+
- Workflow provenance stores the normalized parameter block plus resolved export
201+
metadata such as resolved source component, selected scope, resolution, file
202+
layout, and artifact paths.
203+
- OME-Zarr exports publish a public latest analysis collection; OME-TIFF
204+
exports clear any stale public ``results/volume_export/latest`` collection so
205+
the public store contract matches the selected export format.
206+
207+
Verification
208+
------------
209+
210+
When this workflow changes, validation should cover:
211+
212+
- parameter normalization for all accepted scope/format/layout values,
213+
- current-selection versus all-indices export behavior,
214+
- on-demand generation of missing resolution levels,
215+
- voxel-calibration preservation for OME-Zarr and OME-TIFF outputs,
216+
- cleanup of stale TIFF artifacts or stale public OME collections,
217+
- provenance latest-output registration.

docs/source/workflows/index.rst

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,3 +6,4 @@ Workflows
66

77
zarr-materialization
88
analysis-particle-detection
9+
analysis-volume-export

src/clearex/visualization/README.md

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -380,12 +380,35 @@ image-producing source component.
380380

381381
- Input source is one canonical 6D component in `(t, p, c, z, y, x)` order.
382382
- The operation uses `analysis_parameters["volume_export"]["input_source"]`.
383+
- `force_rerun` preserves the standard latest-only rerun semantics for the
384+
export result.
383385
- `export_scope=current_selection` exports one explicit `(t, p, c)` volume.
384386
- `export_scope=all_indices` exports every available `(t, p, c)` volume for
385387
the selected source.
386388
- `resolution_level` reuses discovered source-adjacent pyramid levels when
387389
available and can generate deeper missing levels during export.
388390

391+
### Parameter reference
392+
393+
- `execution_order=12` places `volume_export` after `compile_movie` and before
394+
`mip_export`.
395+
- `input_source` selects the image-producing upstream component or explicit
396+
internal component path to export.
397+
- `force_rerun` bypasses provenance-based reuse of the latest export result.
398+
- `memory_overhead_factor` is carried through normalized workflow payloads as a
399+
scheduler hint; the current export path does not consume it directly.
400+
- `export_scope` chooses `current_selection` or `all_indices`.
401+
- `t_index`, `p_index`, and `c_index` are used only for
402+
`export_scope=current_selection`.
403+
- `resolution_level` selects the source-adjacent pyramid level to export,
404+
generating missing deeper levels when needed.
405+
- `export_format` chooses between `ome-zarr` and `ome-tiff`.
406+
- `tiff_file_layout` chooses `single_file` or `per_volume_files` for TIFF
407+
output.
408+
- `chunk_basis`, `detect_2d_per_slice`, `use_map_overlap`, and `overlap_zyx`
409+
are normalized compatibility fields inherited from other volume analyses and
410+
remain pinned to 3D/no-overlap behavior for `volume_export`.
411+
389412
### Storage contract
390413

391414
- Latest task metadata is stored at:

0 commit comments

Comments
 (0)