|
| 1 | +.. _analysis-workflow-volume-export: |
| 2 | + |
| 3 | +Analysis Workflow: Volume Export |
| 4 | +================================ |
| 5 | + |
| 6 | +Implemented behavior |
| 7 | +-------------------- |
| 8 | + |
| 9 | +This analysis path exports one selected image-producing ClearEx source component |
| 10 | +from a canonical OME-Zarr store into either an in-store public OME-Zarr result |
| 11 | +or in-store OME-TIFF artifacts. |
| 12 | + |
| 13 | +``volume_export`` is part of the visualization workflow family, but it exports |
| 14 | +resolved runtime data rather than viewer metadata. The source can be any |
| 15 | +previously materialized image-producing component, including the canonical |
| 16 | +source data plus flatfield, deconvolution, shear transform, fusion, or other |
| 17 | +analysis outputs that resolve to canonical internal 6D arrays in |
| 18 | +``(t, p, c, z, y, x)`` order. |
| 19 | + |
| 20 | +Default analysis parameter dictionary |
| 21 | +------------------------------------- |
| 22 | + |
| 23 | +``WorkflowConfig`` carries ``analysis_parameters``, keyed by analysis method, |
| 24 | +with defaults in ``DEFAULT_ANALYSIS_OPERATION_PARAMETERS``. |
| 25 | + |
| 26 | +All operation dictionaries include: |
| 27 | + |
| 28 | +- ``execution_order`` (int): defines run order when multiple routines are |
| 29 | + selected. ``volume_export`` defaults to ``12`` so it runs after |
| 30 | + ``compile_movie`` and before ``mip_export``. |
| 31 | +- ``input_source`` (str): logical source alias or explicit internal component |
| 32 | + path. The default is ``data``. |
| 33 | +- ``force_rerun`` (bool): bypasses provenance-based skip logic and reruns the |
| 34 | + export even when a matching latest result already exists. The default is |
| 35 | + ``False``. |
| 36 | + |
| 37 | +The ``volume_export`` defaults include: |
| 38 | + |
| 39 | +- ``chunk_basis`` (str, default ``"3d"``): normalized execution-compatibility |
| 40 | + field inherited from other volume analyses. ``volume_export`` pins this to |
| 41 | + 3D chunks and does not currently expose alternate chunk bases. |
| 42 | +- ``detect_2d_per_slice`` (bool, default ``False``): compatibility field |
| 43 | + inherited from chunked detection workflows. It is normalized for consistency |
| 44 | + but not used by ``volume_export``. |
| 45 | +- ``use_map_overlap`` (bool, default ``False``): compatibility field inherited |
| 46 | + from chunk-overlap planning. ``volume_export`` keeps overlap masking off. |
| 47 | +- ``overlap_zyx`` (list[int], default ``[0, 0, 0]``): compatibility field for |
| 48 | + overlap-aware chunk planning. ``volume_export`` does not use overlap padding. |
| 49 | +- ``memory_overhead_factor`` (float, default ``1.0``): normalized scheduler |
| 50 | + hint carried through the workflow payload. The current export implementation |
| 51 | + runs eagerly and does not consume this field directly, but it remains part of |
| 52 | + the persisted parameter set. |
| 53 | +- ``export_scope`` (str, default ``"current_selection"``): chooses between |
| 54 | + exporting one selected ``(t, p, c)`` volume or exporting all available |
| 55 | + indices. Accepted values are ``current_selection`` and ``all_indices``. |
| 56 | +- ``t_index`` (int, default ``0``): time index used only when |
| 57 | + ``export_scope=current_selection``. Negative values are normalized to ``0``. |
| 58 | +- ``p_index`` (int, default ``0``): position index used only when |
| 59 | + ``export_scope=current_selection``. Negative values are normalized to ``0``. |
| 60 | +- ``c_index`` (int, default ``0``): channel index used only when |
| 61 | + ``export_scope=current_selection``. Negative values are normalized to ``0``. |
| 62 | +- ``resolution_level`` (int, default ``0``): source-adjacent pyramid level to |
| 63 | + export. ``0`` uses the base source component. Higher levels reuse existing |
| 64 | + display/source-adjacent levels when present and generate missing deeper levels |
| 65 | + on demand during export. |
| 66 | +- ``export_format`` (str, default ``"ome-zarr"``): chooses between public |
| 67 | + in-store OME-Zarr publication and in-store OME-TIFF artifacts. Accepted |
| 68 | + normalized values are ``ome-zarr`` and ``ome-tiff``; aliases such as |
| 69 | + ``zarr`` and ``tiff`` normalize to those canonical names. |
| 70 | +- ``tiff_file_layout`` (str, default ``"single_file"``): TIFF-only policy that |
| 71 | + chooses one BigTIFF/OME-TIFF file versus one file per exported volume. |
| 72 | + Accepted values are ``single_file`` and ``per_volume_files``. |
| 73 | + |
| 74 | +Execution sequence and upstream input behavior |
| 75 | +---------------------------------------------- |
| 76 | + |
| 77 | +- Runtime resolves ``input_source`` through the shared analysis-source alias |
| 78 | + resolver. |
| 79 | +- Common aliases include: |
| 80 | + |
| 81 | + - ``data`` -> ``clearex/runtime_cache/source/data`` |
| 82 | + - ``flatfield`` -> ``clearex/runtime_cache/results/flatfield/latest/data`` |
| 83 | + - ``deconvolution`` -> ``clearex/runtime_cache/results/deconvolution/latest/data`` |
| 84 | + - ``shear_transform`` -> ``clearex/runtime_cache/results/shear_transform/latest/data`` |
| 85 | + - ``fusion`` -> ``clearex/runtime_cache/results/fusion/latest/data`` |
| 86 | + - ``usegment3d`` -> ``clearex/runtime_cache/results/usegment3d/latest/data`` |
| 87 | + |
| 88 | +- Explicit internal component paths are also supported when they resolve to a |
| 89 | + canonical 6D image array. |
| 90 | +- If the requested source component is missing or is not a canonical 6D array, |
| 91 | + ``volume_export`` raises an input dependency error instead of silently |
| 92 | + falling back. |
| 93 | + |
| 94 | +Export scope and resolution behavior |
| 95 | +------------------------------------ |
| 96 | + |
| 97 | +- ``current_selection`` slices the resolved source to exactly one |
| 98 | + ``(t, p, c, z, y, x)`` payload with singleton leading axes. |
| 99 | +- ``all_indices`` exports the full resolved ``(t, p, c, z, y, x)`` array for |
| 100 | + the selected source. |
| 101 | +- ``resolution_level=0`` uses the base source component directly. |
| 102 | +- Higher ``resolution_level`` values first reuse existing source-adjacent |
| 103 | + levels discovered for the selected source. |
| 104 | +- If the requested level is not already materialized, ``volume_export`` builds |
| 105 | + the missing deeper level(s) before writing the export payload. |
| 106 | +- Resolved voxel calibration is scaled to the selected level and persisted in |
| 107 | + metadata plus OME output headers. |
| 108 | + |
| 109 | +Volume-export execution workflow |
| 110 | +-------------------------------- |
| 111 | + |
| 112 | +1. Normalize the ``volume_export`` parameter block. |
| 113 | +2. Resolve ``input_source`` to the canonical internal source component. |
| 114 | +3. Resolve or generate the requested ``resolution_level`` component. |
| 115 | +4. Validate the selected ``t_index``, ``p_index``, and ``c_index`` when |
| 116 | + ``export_scope=current_selection``. |
| 117 | +5. Rewrite the export runtime-cache payload at |
| 118 | + ``clearex/runtime_cache/results/volume_export/latest/data``. |
| 119 | +6. Materialize either: |
| 120 | + |
| 121 | + - a public OME image collection under ``results/volume_export/latest`` when |
| 122 | + ``export_format=ome-zarr``, or |
| 123 | + - OME-TIFF artifacts under |
| 124 | + ``clearex/results/volume_export/latest/files`` when |
| 125 | + ``export_format=ome-tiff``. |
| 126 | + |
| 127 | +7. Rewrite latest export metadata under ``clearex/results/volume_export/latest``. |
| 128 | +8. Register the latest-output reference in provenance. |
| 129 | + |
| 130 | +Storage and output contract |
| 131 | +--------------------------- |
| 132 | + |
| 133 | +The export always rewrites the latest runtime payload under: |
| 134 | + |
| 135 | +- ``clearex/runtime_cache/results/volume_export/latest/data`` |
| 136 | + |
| 137 | +Latest metadata for the run is stored under: |
| 138 | + |
| 139 | +- ``clearex/results/volume_export/latest`` |
| 140 | + |
| 141 | +The metadata group records: |
| 142 | + |
| 143 | +- requested ``input_source``, |
| 144 | +- resolved source and resolution components, |
| 145 | +- ``export_scope``, |
| 146 | +- ``resolution_level``, |
| 147 | +- whether deeper levels were generated on demand, |
| 148 | +- ``export_format``, |
| 149 | +- ``tiff_file_layout``, |
| 150 | +- selected shape metadata, |
| 151 | +- resolved voxel size, |
| 152 | +- emitted artifact paths. |
| 153 | + |
| 154 | +OME-Zarr output contract |
| 155 | +------------------------ |
| 156 | + |
| 157 | +- ``export_format=ome-zarr`` publishes a public OME image collection under |
| 158 | + ``results/volume_export/latest``. |
| 159 | +- ``current_selection`` publishes one exported field at the selected |
| 160 | + resolution. |
| 161 | +- ``all_indices`` publishes every available exported field from the resolved |
| 162 | + source. |
| 163 | +- The public result reflects the selected export resolution; it does not |
| 164 | + publish a new multiscale pyramid for the export itself. |
| 165 | + |
| 166 | +OME-TIFF layout policy |
| 167 | +---------------------- |
| 168 | + |
| 169 | +- ``current_selection`` always writes one ``ZYX`` OME-TIFF/BigTIFF file named |
| 170 | + from the selected ``t/p/c`` indices. |
| 171 | +- ``all_indices + single_file`` writes one OME-TIFF/BigTIFF file with one |
| 172 | + ``TCZYX`` series per exported position. |
| 173 | +- ``all_indices + per_volume_files`` writes one ``ZYX`` OME-TIFF file per |
| 174 | + exported ``(t, p, c)`` volume. |
| 175 | +- TIFF artifacts are stored inside the analysis store under |
| 176 | + ``clearex/results/volume_export/latest/files`` so they remain namespaced with |
| 177 | + the latest export metadata. |
| 178 | +- TIFF metadata preserves resolved ``PhysicalSizeZ/Y/X`` calibration. |
| 179 | + |
| 180 | +CLI and GUI surface |
| 181 | +------------------- |
| 182 | + |
| 183 | +- The CLI currently exposes the operation flag ``--volume-export``. |
| 184 | +- Detailed parameter editing is currently done through the GUI or a programmatic |
| 185 | + ``WorkflowConfig.analysis_parameters["volume_export"]`` payload. |
| 186 | +- The visualization GUI exposes controls for: |
| 187 | + |
| 188 | + - ``input_source`` |
| 189 | + - ``export_scope`` |
| 190 | + - ``t_index`` / ``p_index`` / ``c_index`` |
| 191 | + - ``resolution_level`` |
| 192 | + - ``export_format`` |
| 193 | + - ``tiff_file_layout`` |
| 194 | + |
| 195 | +Provenance integration |
| 196 | +---------------------- |
| 197 | + |
| 198 | +- ``volume_export`` registers its latest output reference under |
| 199 | + ``clearex/provenance/latest_outputs/volume_export``. |
| 200 | +- Workflow provenance stores the normalized parameter block plus resolved export |
| 201 | + metadata such as resolved source component, selected scope, resolution, file |
| 202 | + layout, and artifact paths. |
| 203 | +- OME-Zarr exports publish a public latest analysis collection; OME-TIFF |
| 204 | + exports clear any stale public ``results/volume_export/latest`` collection so |
| 205 | + the public store contract matches the selected export format. |
| 206 | + |
| 207 | +Verification |
| 208 | +------------ |
| 209 | + |
| 210 | +When this workflow changes, validation should cover: |
| 211 | + |
| 212 | +- parameter normalization for all accepted scope/format/layout values, |
| 213 | +- current-selection versus all-indices export behavior, |
| 214 | +- on-demand generation of missing resolution levels, |
| 215 | +- voxel-calibration preservation for OME-Zarr and OME-TIFF outputs, |
| 216 | +- cleanup of stale TIFF artifacts or stale public OME collections, |
| 217 | +- provenance latest-output registration. |
0 commit comments