Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
155 changes: 155 additions & 0 deletions .notes/implementation-notes.html
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,138 @@
</head>
<body>
<h1>EEGPrep Core Parity Implementation Notes</h1>
<h1>Issue #213 Statistics and ICLabel Closeout Notes</h1>
<p>Issue #213 closes the final Fable 5 architecture closeout phase for
statistics and ICLabel/viewprops ownership. The branch keeps public call
signatures, return values, command strings, GUI labels, and Matplotlib
dashboard layout behavior stable while moving implementations into focused
owner modules.</p>
<h2>Issue #213 Design Decisions</h2>
<ul>
<li>Moved statistics implementations out of
<code>functions/statistics/_core.py</code> into the existing same-name
public modules such as <code>fdr.py</code>, <code>statcond.py</code>,
<code>surrogdistrib.py</code>, and the <code>anova*_cell.py</code> and
<code>ttest*_cell.py</code> helpers. <code>_core.py</code> is now only a
compatibility re-export shell; shared array/grid helpers and two-way ANOVA
data containers live in <code>_shared.py</code>.</li>
<li>Split <code>plugins/ICLabel/pop_prop_extended.py</code> into pop/dialog
glue, <code>_prop_numerics.py</code> for classifier, DIPFIT, PVAF, spectra,
and dashboard data assembly, and <code>_prop_browser.py</code> for the
Matplotlib property-browser rendering and rejection controls.</li>
<li>Kept the public <code>pop_prop_extended</code> module as a facade that
re-exports the existing helper names used by tests and by
<code>pop_viewprops</code>, so users do not need to change imports.</li>
<li>No GUI layout, text, button order, or rendering constants were changed.
Because the visual surface was moved without visible behavior changes,
visual parity attachment is not required for this phase.</li>
</ul>
<h2>Issue #213 Closeout Mapping for Remaining #197/#205 Items</h2>
<table>
<thead>
<tr><th>Remaining item</th><th>Disposition</th><th>Rationale</th></tr>
</thead>
<tbody>
<tr><td><code>pop_load_frombids.py</code> raw-reader and montage
ownership</td><td>Fixed in PR #216 / issue #212</td><td>BIDS raw loading
and montage inference moved into EEG-BIDS helper modules while
<code>pop_load_frombids</code> stayed orchestration/history glue.</td></tr>
<tr><td><code>functions/statistics/_core.py</code> mega-module</td><td>Fixed
in issue #213</td><td>Implementations now live in same-name statistics
modules, with <code>_shared.py</code> holding only shared helpers.</td></tr>
<tr><td><code>plugins/ICLabel/pop_prop_extended.py</code> mixed
ownership</td><td>Fixed in issue #213</td><td>Pop/history/dialog glue,
numerics/data assembly, and Matplotlib browser rendering now have
separate owner modules.</td></tr>
<tr><td><code>functions/guifunc/qt.py</code> stateless renderer namespace</td><td>Fixed
in PR #216 / issue #212</td><td>Qt renderer helpers were split into
module-level helpers while preserving call-site compatibility and dialog
behavior.</td></tr>
<tr><td><code>pop_clust</code> finite-outlier robust k-means dispatch</td><td>Fixed
in PR #217 / issue #210</td><td>STUDY clustering behavior was consolidated
in the STUDY/time-frequency/statistics phase with dedicated tests.</td></tr>
<tr><td><code>pop_chanplot</code>, measure-field maps, cached measure
axes, range masks, default plot targets, <code>_trial_rows</code>, factor
matching, and <code>std_clustplot</code> history construction</td><td>Fixed
in PR #217 / issue #210</td><td>STUDY measure/cache helper ownership was
consolidated as one phase because those paths share STUDY data contracts.</td></tr>
<tr><td>Time-frequency numeric-vector parsing, bootstrap/FDR helpers,
threshold-vector helpers, and empirical p-value convention audit</td><td>Fixed
or explicitly documented in PR #214 / issue #208 and PR #217 / issue
#210</td><td>Shared parsing moved to lower-level helpers in Phase 1;
time-frequency and statistics behavior consolidation landed in Phase 3
where numerical behavior could be tested together.</td></tr>
<tr><td>Divergent <code>is_on()</code>, empty-value, Python literal,
chanloc serialization, topographic channel selection, boundary-event
detection, and <code>ConsoleEegh</code> history mutation copies</td><td>Fixed
in PR #214 / issue #208</td><td>Shared low-level contracts were centralized
before later phases consumed them.</td></tr>
<tr><td>Rejection-family browser plumbing, epoched rejection scaffolds,
component activation access, ICA finalization, and clean_rawdata
channel-removal masks</td><td>Fixed in PR #215 / issue #209</td><td>These
scientific rejection/ICA/cleaning helpers were consolidated in the phase
that could test visual rejection state and numerical side effects
together.</td></tr>
<tr><td>CLI transform/pipeline duplication, JSON detection, stale
per-module harness contracts, extension catalog split, entry-point
helpers, active-record predicate, and bundled-plugin metadata</td><td>Fixed
in PR #218 / issue #211</td><td>CLI and extension ownership was
consolidated in one agent-facing architecture phase.</td></tr>
<tr><td>FIR helper ownership and FIR GUI band-edge/shape duplication</td><td>Fixed
in PR #216 / issue #212</td><td><code>plugins/firfilt</code> now owns FIR
design helpers, and clean_rawdata imports downward from firfilt.</td></tr>
<tr><td>GUI extension pop-result STUDY return handling from #197/#205</td><td>Non-goal
for #213</td><td>The earlier tracker identified this as behavior-changing
because GUI extension <code>pop_*</code> functions returning STUDY state
would expand observable session behavior. No #213 work touched extension
GUI result semantics; this remains outside a mechanical closeout split
unless a future behavior issue requests it.</td></tr>
<tr><td>All other #197 findings not named above</td><td>Superseded by epic
#207 phase split</td><td>#197 and #205 were closed as superseded by the
replacement architecture closeout epic. PRs #214, #215, #217, #218, #216,
and this #213 branch are the final accounting set for those remaining
findings.</td></tr>
</tbody>
</table>
<h2>Issue #213 Verification Notes</h2>
<ul>
<li>Focused statistics tests cover same-name module ownership, package-level
callable exports after submodule imports, <code>_core.py</code>
compatibility re-exports, and existing statistical behavior.</li>
<li>Focused ICLabel/viewprops tests cover the preserved
<code>pop_prop_extended</code> facade, classifier/DIPFIT/PVAF data assembly,
dashboard rendering, navigation, rejection controls, packaged help, and
<code>pop_viewprops</code> delegation.</li>
<li>No packaged help text or Sphinx user docs changed because the public
behavior, GUI workflow, and help resource names did not change.</li>
</ul>
<h1>Issue #212 BIDS, Qt, and FIR Ownership Notes</h1>
<p>Issue #212 closes the Phase 5 architecture ownership work for BIDS
import helpers, Qt dialog rendering helpers, and FIR design helpers while
preserving standalone EEGPrep runtime behavior.</p>
<h2>Issue #212 Design Decisions</h2>
<ul>
<li>Kept <code>pop_load_frombids</code> as the BIDS orchestration,
metadata, event, history, and report wrapper, and moved raw EEG file
loading plus packaged montage inference into <code>plugins/EEG_BIDS</code>
helper modules.</li>
<li>Moved clean_rawdata FIR design helpers into <code>plugins/firfilt</code>
so clean_rawdata imports shared filter design behavior from the plugin
that owns FIR filtering semantics.</li>
<li>Converted the stateless Qt renderer helper bodies into module-level
functions while keeping <code>QtDialogRenderer._helper</code> aliases for
existing callers and tests.</li>
</ul>
<h2>Issue #212 Verification Notes</h2>
<ul>
<li>Focused tests cover extracted BIDS raw loading, montage inference from
packaged resources, FIR design helper ownership, and Qt helper alias
compatibility.</li>
<li>Sample-data BIDS import tests and representative Qt dialog/visual
parity tests exercise the wrapper behavior after decomposition.</li>
<li>No runtime code depends on the vendored EEGLAB reference checkout; the
EEGLAB and EEG-BIDS sources were used only as development references.</li>
</ul>
<h1>Issue #164 EEGLAB-Style Sphinx Documentation Notes</h1>
<p>Issue #164 turns the final-epic documentation into a coherent standalone
EEGPrep manual modeled after EEGLAB's learning path but written for Python,
Expand Down Expand Up @@ -566,5 +698,28 @@ <h2>Tradeoffs</h2>
dialog, while the console action boundary keeps command echo and progress
output ordered for mixed GUI-plus-console workflows.</li>
</ul>
<h1>Issue #211 CLI And Extension Architecture Notes</h1>
<h2>Design Decisions</h2>
<ul>
<li>Made the top-level <code>eegprep</code> dispatcher the canonical owner of
JSON error envelopes, including nested command parse failures such as
<code>eegprep qc report --json</code>.</li>
<li>Routed pipeline preprocessing steps through the transform command module so
CLI subcommands and YAML pipelines share option normalization and
<code>pop_*</code> invocation behavior.</li>
<li>Split runtime extension catalog loading from catalog curation validation.
The validator CLI now lives in <code>eegprep.extension_catalog_validation</code>
while <code>eegprep.extension_catalog</code> remains the manager-facing runtime
catalog module.</li>
<li>Centralized extension entry-point selection, API-version parsing, package
naming, and active/installed status predicates in <code>eegprep.extensions</code>
so the registry, validator, and plugin menu use the same metadata rules.</li>
</ul>
<h2>Verification Notes</h2>
<ul>
<li>Added regression tests for canonical JSON parse errors, transform/pipeline
option equivalence, catalog validator ownership, shared extension metadata
helpers, and bundled plugin menu projection from registry metadata.</li>
</ul>
</body>
</html>
16 changes: 11 additions & 5 deletions docs/source/api/extensions.rst
Original file line number Diff line number Diff line change
Expand Up @@ -62,11 +62,12 @@ external extension contributions follow one status and lazy-loading model.
Catalog and Governance
======================

Catalog metadata validation lives in ``eegprep.extension_catalog`` and is also
available as the ``eegprep-validate-extension-catalog`` console script. Static
validation checks JSON schema version, required metadata, naming, URLs, license,
maintainer contact, docs, conflicts, curation status, and compatibility fields
without requiring the extension package to be installed.
Extension Manager catalog loading lives in ``eegprep.extension_catalog``. Catalog
submission validation lives in ``eegprep.extension_catalog_validation`` and is
also available as the ``eegprep-validate-extension-catalog`` console script.
Static validation checks JSON schema version, required metadata, naming, URLs,
license, maintainer contact, docs, conflicts, curation status, and compatibility
fields without requiring the extension package to be installed.

Stricter validation can also check installed package versions, required
dependencies, the ``eegprep.extensions`` entry point, import failures, and
Expand Down Expand Up @@ -175,6 +176,11 @@ API Reference
:members:
:undoc-members:

.. automodule:: eegprep.extension_catalog_validation
:no-index:
:members:
:undoc-members:

.. automodule:: eegprep.extension_testing
:no-index:
:members:
Expand Down
7 changes: 7 additions & 0 deletions docs/source/user_guide/agent_cli.rst
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,13 @@ planned, reviewed, and rerun.
eegprep pipeline run preprocess.yaml --json
eegprep batch run sub-01.set sub-02.set --pipeline preprocess.yaml --output-dir derivatives/eegprep --json

Pipeline transform steps use the same defaults as the matching direct CLI
commands. In particular, ``clean`` defaults to ASR burst correction with
``burst_criterion: 20`` and leaves flatline, channel, line-noise, window, and
high-pass cleaning criteria off unless the YAML config sets them explicitly.
Set ``highpass`` as a two-value transition band, for example
``highpass: [0.25, 0.75]``.

QC And Reports
==============

Expand Down
27 changes: 23 additions & 4 deletions docs/source/user_guide/study_workflows.rst
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,9 @@ cached arrays. Cached measure fields follow EEGLAB names such as ``erpdata``,
``specdata``, ``erspdata``, and ``itcdata``. The selected ``design`` is recorded
in each measure group's metadata. EEGPrep stores dataset-level averages in the
current standalone cache rather than EEGLAB sidecar measure files.
``pop_chanplot`` reads cached channel and component measures through the same
``std_readdata``/``std_erpplot``/``std_erspplot`` cache contract used by scripts,
so GUI and console plots slice axes and cached channel groups consistently.

Use ``std_checkfiles``, ``std_checkdatasession``, ``std_uniformfiles``, and
``std_uniformsetinds`` to audit loaded dataset consistency and cached measure
Expand All @@ -129,10 +132,12 @@ Select datasets or trials from STUDY metadata:
["target"],
)

These helpers return EEGLAB-facing 1-based dataset and trial indices. Use
``std_substudy`` or ``std_rmdat`` when a workflow needs to remove datasets;
EEGPrep remaps STUDY references and invalidates cached measure arrays after
membership changes.
These helpers return EEGLAB-facing 1-based dataset and trial indices. Trial
metadata may be stored as row dictionaries or as EEGLAB-loaded columnar
``{"factor": [values...]}`` dictionaries; STUDY selectors normalize both forms
before matching factor levels and numerical ranges. Use ``std_substudy`` or
``std_rmdat`` when a workflow needs to remove datasets; EEGPrep remaps STUDY
references and invalidates cached measure arrays after membership changes.

``std_findsameica`` groups matching ICA decompositions within each subject.
This preserves the subject boundary used by STUDY designs instead of merging
Expand All @@ -153,6 +158,12 @@ Precluster and cluster ICA components:
STUDY, com = pop_clust(STUDY, ALLEEG, clus_num=4, random_state=0, return_com=True)
STUDY, com, fig = pop_clustedit(STUDY, ALLEEG, action="plot", return_com=True)

Finite ``outliers`` values in ``pop_clust`` use the EEGLAB-compatible
``robust_kmeans`` path and record ``["robust_kmeans", clus_num]`` in each
created cluster's algorithm provenance. Infinite ``outliers`` keeps the plain
k-means path. In both cases command history remains pasteable in
``eegprep-console``.

DIPFIT Source Localization
==========================

Expand Down Expand Up @@ -223,6 +234,14 @@ EEGPrep-owned ``pacdata``, ``pactimes``, and ``pacfreqs`` caches on
plots their magnitude using the same cache-reading contract as the other
STUDY measure plots.

Empirical p-value conventions are explicit. ``pac`` applies the common
``(exceedances + 1) / (permutations + 1)`` finite-sample convention,
``bootstat`` exposes exact permutation proportions for bootstrap helper tests,
and statistics helpers such as ``stat_surrogate_pvals`` follow EEGLAB's
surrogate-tail convention with FDR correction available through the statistics
module. These definitions are intentionally not collapsed when they answer
different inferential questions.

The feasible in-package LIMO-compatible layer is design preparation:
``std_limodesign`` builds categorical and continuous matrices from
``pop_listfactors`` output and trial metadata, including interaction and split
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ all = [
eegprep = "eegprep.cli.main:main"
eegprep-gui = "eegprep.functions.adminfunc.eeglab:main"
eegprep-console = "eegprep.functions.adminfunc.console:main"
eegprep-validate-extension-catalog = "eegprep.extension_catalog:main"
eegprep-validate-extension-catalog = "eegprep.extension_catalog_validation:main"

[dependency-groups]
dev = [
Expand Down
12 changes: 6 additions & 6 deletions src/eegprep/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,9 +23,9 @@
"EEGPrepSession": ("eegprep.functions.guifunc.session", "EEGPrepSession"),
"EEGobj": ("eegprep.functions.eegobj.eegobj", "EEGobj"),
"CATALOG_SCHEMA_VERSION": ("eegprep.extension_catalog", "CATALOG_SCHEMA_VERSION"),
"CatalogValidationIssue": ("eegprep.extension_catalog", "CatalogValidationIssue"),
"CatalogValidationOptions": ("eegprep.extension_catalog", "CatalogValidationOptions"),
"CatalogValidationReport": ("eegprep.extension_catalog", "CatalogValidationReport"),
"CatalogValidationIssue": ("eegprep.extension_catalog_validation", "CatalogValidationIssue"),
"CatalogValidationOptions": ("eegprep.extension_catalog_validation", "CatalogValidationOptions"),
"CatalogValidationReport": ("eegprep.extension_catalog_validation", "CatalogValidationReport"),
"EXTENSION_COMPATIBILITY_POLICY": ("eegprep.extensions", "EXTENSION_COMPATIBILITY_POLICY"),
"EXTENSION_API_VERSION": ("eegprep.extensions", "EXTENSION_API_VERSION"),
"EXTENSION_CURATION_POLICY_URL": ("eegprep.extensions", "EXTENSION_CURATION_POLICY_URL"),
Expand Down Expand Up @@ -141,7 +141,7 @@
"lat2point": ("eegprep.functions.redefine_functions", "lat2point"),
"listdlg2": ("eegprep.functions.guifunc.listdlg2", "listdlg2"),
"minphaserceps": ("eegprep.plugins.firfilt.minphaserceps", "minphaserceps"),
"load_catalog_entries": ("eegprep.extension_catalog", "load_catalog_entries"),
"load_catalog_entries": ("eegprep.extension_catalog_validation", "load_catalog_entries"),
"load_extension_catalog": ("eegprep.extension_catalog", "load_extension_catalog"),
"loadset": ("eegprep.functions.popfunc.pop_loadset", "loadset"),
"mne2eeg": ("eegprep.functions.redefine_functions", "mne2eeg"),
Expand Down Expand Up @@ -392,8 +392,8 @@
"timewarp": ("eegprep.functions.timefreqfunc.timewarp", "timewarp"),
"topoplot": ("eegprep.functions.sigprocfunc.topoplot", "topoplot"),
"validate_extension_spec": ("eegprep.extensions", "validate_extension_spec"),
"validate_catalog_entries": ("eegprep.extension_catalog", "validate_catalog_entries"),
"validate_catalog_file": ("eegprep.extension_catalog", "validate_catalog_file"),
"validate_catalog_entries": ("eegprep.extension_catalog_validation", "validate_catalog_entries"),
"validate_catalog_file": ("eegprep.extension_catalog_validation", "validate_catalog_file"),
"writelocs": ("eegprep.functions.sigprocfunc.writelocs", "writelocs"),
}

Expand Down
15 changes: 5 additions & 10 deletions src/eegprep/cli/commands/bids.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,11 @@

import argparse
import importlib.util
import json
import sys
from pathlib import Path
from typing import Any

from eegprep.cli.core import build_manifest, json_safe, now_iso, output_path, sha256_file, write_manifest_file
from eegprep.cli.core import build_manifest, now_iso, output_path, sha256_file, write_manifest_file
from eegprep.cli.dataset import dataset_summary, load_dataset
from eegprep.functions.popfunc.pop_saveset import pop_saveset
from eegprep.plugins.EEG_BIDS.bids_list_eeg_files import bids_list_eeg_files
Expand Down Expand Up @@ -243,14 +242,10 @@ def export_dataset(


def main(argv: list[str] | None = None) -> int:
"""Standalone module harness for tests and local debugging."""
parser = argparse.ArgumentParser(prog="eegprep bids")
subparsers = parser.add_subparsers(dest="command", required=True)
register(subparsers)
args = parser.parse_args(["bids", *(sys.argv[1:] if argv is None else argv)])
result = args.handler(args)
print(json.dumps(json_safe(result), sort_keys=True))
return 0 if result.get("status") in {"ok", "warning"} else int(result.get("exit_code", 1))
"""Route module execution through the canonical top-level CLI dispatcher."""
from eegprep.cli.main import main as cli_main

return cli_main(["bids", *(sys.argv[1:] if argv is None else argv)])


def _file_type(path: Path) -> str:
Expand Down
Loading
Loading