CHANGELOG.md

Changelog

v0.1.7 (06/04/2024; revised 06/19/2024

• Modified display/focus set selection behavior in the neural units table: If the Ctrl/Command is held down when a table row is left-clicked, the corresponding unit is now removed from the focus set if it was already in that set (unless it is the ONLY unit with the display focus). Otherwise, as before, it is added to the focus set if there is room (maximum of 3 units can have the display focus at any one time).
• A command-line no-GUI version of XSort is now available that builds the internal cache for a specified working directory. For detailed usage information: python -m xsort.app ?.
• All internal cache files generated by XSort are now located in a single folder $WD/.xs-cache within the current working directory. This allows the user to manually delete the internal cache with relative ease.
• Removed DC offset (template -= np.median(template)) when computing neural unit spike templates.
• The in-place edit of a neural unit label is now initiated by left-clicking on the relevant table cell while holding down the Alt(Opt) modifier key. That way the user does not need the right mouse button at all (except when hiding/showing table columns by right-clicking on the table header).
• Added inline auto-completion of a neural unit label in NeuronView. The set of "suggestions" for auto-completion are all labels previously applied to a unit by the user. The set of previous labels is maintained in the user's settings file. In addition, pressing the Tab key instead of Return/Enter will make the label change and automatically start editing the label of the unit in the next table row.
• Implemented an internal cache removal policy: Never (cache removal disabled, the default); Always (every working directory visited is marked for cache removal); or LRU (a working directory is marked for cache removal when it is evicted from the user's most recentlly visited list, and unmarked if it is put back on that list). Note that the working directory's edit history must be removed when the internal cache is destroyed, since derived units are only defined in the internal cache). Cache removal does not happen until application shutdown.
• Added a Preferences dialog that exposes two user settings: the internal cache removal policy, and a set of suggested unit labels.
• The Ctrl(Command)-Up and Ctrl(Command)-Down keyboard shortcut moves the secondary focus unit (highlighted in red) up and down in the unit table without changing the primary focus unit (blue highlight). The tertiary focus unit (yellow highlight), if any, is removed from the focus list.
• A new checkbox below the neuron table in the NeuronView allows the user to highlight the five (5) neural units most similar to the current primary focus unit. If checked, the 5 units are always listed immediately below the primary unit regardless the sort order and are highlighted with a light blue background. The state of the checkbox is a user preference saved at application exit and restored at startup.
• New feature: Multi-unit relabel and delete operations. When you left-click on a row in the neuron table with the Alt(Opt) key depressed, the corresponding unit's selection state is toggled. Units in the edit selection set are highlighted with a dark blue background in the UID cell only. You can also add a contiguous range of rows to the selection via Shift-Click.
  • To relabel all of the units in the selection set in one go, edit one of the unit labels -- upon entering the new label, all of the units in the set are relabeled accordingly and the selection is cleared.
  • To delete all of the units, choose Edit > Delete.
  • The multi-unit relabel or delete is executed and undone as a single operation, not N separate operations.
  • To clear the current edit selection set, hit the ESC key, the Spacebar, or Shift-Alt-Click.
• NOTE: If the edit selection set is empty, hitting Edit > Delete reverts to the original behavior: the current primary focus unit (royal blue) is deleted, and the next unit in the table becomes the new primary unit.
• (06/19/2024) Improved behavior of the "indeterminate" progress bar in modal blocking dialog.

v0.1.6 (05/13/2024)

• Updated About dialog to include version number and release date.
• Updated many of the in-app user guide markdowns to reflect recent changes.
• Bug fix: Upon showing the PCA view, the PCA would be recomputed from scratch even though the PCA projections were already cached.
• Changed how the in-place control is raised to edit a unit label in the neural unit table: a right-click on the relevant table cell instead of a left double-click. That way, the user can edit any unit's label without altering the current display/focus list.
• Now raises a modal error message dialog to report a failed merge or split operation or a failed background task.
• The Up and Down arrow keys can now be used to move the primary unit focus up and down the unit table, providing a way to change the focus without having to use the mouse.
• The neural unit table now displays a tool tip for a unit label if that label does not fit inside the corresponding table cell.
• Bug fix: When loading a previously edited working directory, XSort failed to apply previous Split operations from the directory's edit history.

v0.1.5 (04/29/2024)

• Extend support to non-Omniplex recordings in which the analog source is a flat binary file. The file may be interleaved (or not), and prefiltered (or not). Splitting analog channel streams into separate internal cache files is unnecessary when the analog source is a flat, prefiltered binary file.
• When "opening" a working directory for the first time, XSort will query user via dialog if there are multiple possible analog and/or unit source files, and/or if the analog source is a flat binary file. In the case of a flat binary source, the user must specify the sampling rate and number of channels recorded, the scale factor converting a raw binary sample to volts, whether or not the analog channel streams are interleaved, and whether or not the data is prefiltered. Once a valid directory configuration is specified, the configuration info is saved to an internal file (.xs.directory.txt) so that the user does not have to reenter the information the next time XSort visits the directory.
• Major redesign of the infrastructure/strategy for background task processing to improve performance and handle recording sessions with hundreds of analog channels and hundreds of neural units.
  • Divide up work among multiple processses where possible -- especially helpful for the building the internal cache for an XSort directory, and for computing statistics for units in the current display/focus list.
  • Evaluated a number of different approaches to handle the "build internal cache" task, using a Neuropixel session with 385 analog channels and 482 units.
  • We now only compute a unit's per-channel spike templates on up to 16 analog channels "in the neighborhood" of the unit's primary channel (channel exhibiting highest SNR). Since XSort does not yet support the notion of probe geometry, we use channel indices [P-8 .. P+7], where P is the primary channel index.
  • Greatly improved the algorithm for computing noise levels on each analog channel. The per-channel noise levels are calculated and cached as part of the "build internal cache" task -- after caching analog channels (if needed), but before caching neural unit metrics.
  • When there are more than 16 recorded analog channels, building the unit metrics cache files has two phases. First, a background task identifies the primary channel P for each unit. Then, a second background task computes the spike templates for the channels near each unit's primary channel and writes each unit's metrics (spike times, templates, snr, primary channel) to an internal cache file (.xs.unit.<uid>).
• Modified TemplateView implementation:
  • Precreate and reuse graphics items that render the templates and display channel labels. This is more efficient than destroying and recreating them every time -- improving overall GUI responsiveness.
  • Vertical span is auto-adjusted whenever there is a change in the displayed templates. User can still readjust the span using the slider provided.
• Modified ChannelView implementation:
  • Adjusts the visible Y-axis range whenever the user explicitly changes the vertical separation between trace so that, if the user has zoomed in on a few traces, those traces remain approximately in the same place after the change.
  • Precreate and reused graphics items that render channel traces and spike clips. Again, this is more efficient and improved overall GUI responsiveness.
  • Vertical trace offset is auto-adjusted whenever the channel traces are updated (this also resets the zoom/pan state). User can still readjust the trace offset using the slider provided.

v0.1.4 (02/08/2024)

• Finished implementation of the Help view, which offers a small user guide for XSort. The guide is divided into five chapters. A dropdown combo box selects the chapter displayed in the read-only text browser. The chapter contents are maintained in Markdown files in the assets folder.
• Minor changes to CorrelogramView: Added a horizontal line at zero correlation and a translucent white vertical band spanning the time range -1.5 to +1.5 milliseconds in each ACG or CCG plot. A checkbox at the bottom of the view lets the user toggle the visibility of the zero correlation lines.
• The TemplateView now includes a slider to set the voltage range (Y-axis) for the templates, between +/-30 and +/-250 microvolts. The voltage span is a user preference that is saved at application exit and restored on startup.
• The ChannelView now includes a slider to set the vertical separation between channel traces, between +/-50 and +/-150 microvolts. This parameter is NOT saved as a user preference and defaults to 100 microvolts at startup.
• Modified how user selects units for display in the neural units table. Clicking on any row in the table selects the corresponding unit for display while clearing any previous selection (single-selection behavior). To select multiple units for display (up to three), you must hold down the Control key (Command key on MacOS) while clicking on each table row to select.
• Implemented File | Open Recent menu with up to 5 most recently visited working directories.
• Implemented File | Save menu command. This will save the current list of neural units to a Python pickle file at a location specified by the user. The content is a List[Dict[str, Any]], with the following key-value pairs:
  1. "uid": The UID assigned to the unit in XSort (str).
  2. "spikes": A 1D Numpy array holding the unit's spike timestamps in chronological order, in seconds.
  3. "primary": The integer index identifying the primary channel on which the best signal-to-noise ratio was observed for the unit.
  4. "snr": The signal-to-noise observed on the unit's primary channel (float).
  5. "template": A 1D Numpy array holding the unit's mean spike waveform as recorded on the primary channel. The waveform spans 10-ms and is in microvolts.
• Updated setup.cfg to use ">=" rather than "==" in the install_requires section. This is needed for Numpy, in particular, as pip won't install Numpy <= 1.25 if the Python version is 3.12, as changes made in the 3.12 elease were incompatible with Numpy versions prior to 1.26.
• Introduced support for an alternative working directory configuration. Instead of the Omniplex PL2 file, a flat binary file (extensions .bin or .dat) can serve as the analog source file. This file contains the analog channel streams as raw 16-bit signed integer samples. The streams may be stored one after the other or interleaved, and the raw data may be prefiltered or not. In addition, XSort now handles situations in which the specified working directory contains multiple Python pickle files and multiple .pl2/.bin/.dat files. Whenever the identity of the neural unit and analog channel data source files is ambiguous, or the analog source requires additional configuration from the user, XSort raises a modal dialog to request this information from the user.
• Note that, if the analog source is a pre-filtered flat binary file, there is no need to generate the individual internal analog channel cache files.

v0.1.3 (01/22/2024)

• Implemented the Edit|Split operation. All editing operations are now available.
• Added modal progress dialog to block user input when waiting on a cancelled background task to stop. The dialog message reads "Please wait..." The time it takes for a background task to detect the cancel request and stop is highly variable, but will be less than 5 seconds in most cases. The dialog's progress bar animates 0-99% completion in 5 seconds, even though the wait time is unknown. Most tasks will respond to the cancel signal within that time. If not, the progress bar resets to 0% and continues animating. The same dialog will be raised at application exit if the user quits while a background task is in progress.
• When the user changes the XSort working directory, tha BUILDCACHE background task is launched to examine the directory contents, cache all Omniplex analog data channel streams in individual binary files, compute metrics (per-channel spike templates, best SNR and the analog channel on which that SNR was measured - aka the primary channel) for each unit defind in the original spike sorter pickle file, and persist unit spike trains and metrics in individual unit cache files. For a long recording session with many units, this task takes quite some time, and the user can't really do a lot of useful work until it's done. Therefore, a modal progress dialog now blocks user input until the BUILDCACHE task has finished.
• Fixed calculation of firing rate-vs-time histogram in Neuron. When not normalized, each bin value is now divided by the bin size in seconds, so the histogram reflects spikes/second (aka, firing rate) rather than raw spike count.
• FiringRateView modified so that the Y readout value indicates the firing rate at the corresponding time in Hz when the Normalized box is unchecked.
• BUG FIXED: NeuronView failed to display correctly when the neural unit list contained only a single unit (because the sort-by-column algorithm failed to initialize the list of sort indices in this scenario).

v0.1.2 (01/18/2024)

• PCAView: Implemented "lasso" interaction to define a closed polygonal region by clicking on a series of points within the view -- in order to select a subset of spikes prior to "splitting" a neural unit. This interaction is enabled only when a single unit occupies the display focus list, and only when the unit's PCA projection has been fully calculated. The "Split" operation is not yet implemented.
• BUG FIX: Release 0.1.1 failed to launch because of a configuration error in packaging the wheel file -- the markdown files were missing from the assets folder.

v0.1.1 (01/12/2024)

• NeuronView: Selected columns in the neural units table may be hidden/shown via a popup context menu raised by clicking anywhere on the table header. The UID column may not be hidden. Column visibility is saved in user settings at application shutdown and restored from settings at startup. Fixed issue with sorting on UID column.
• ACGRateView: Now renders ACG-vs-firing rate histograms as heatmap images.
• Added an "About XSort" dialog.
• Created a skeleton "user guide" which appears in a dock widget like the other XSort views. Contents TBD.
• Added user-editable label property to Neuron, distinct from its uid. The label string is restricted to 25 characters or less and may not contain a comma.NeuronView modified to display the unit labels in a Label column. Users can edit each unit's label "in place".
• Developed infrastructure for editing the neural list. Supported operations include: edit a unit label, delete a unit, or merge two units. A future release will add support for splitting a unit in two by "lassoing" a subset of that unit's spikes in the PCAView. Each individual edit is saved to an "edit history" for the neural unit list. This history is persisted in a dedicated CSV file in the current XSort working directory prior to switching directories or exiting the application. Upon returning to a given working directory, the initial state of the neural list is read from the original spike sorter file, then that list is updated by applying each of the edits in the saved edit history, in order.
• Units created by merging or splitting will have a UID ending in the letter 'x'. The integer index assigned to a unit is incremented each time a unit is created by merging or splitting, thereby ensuring that unit UIDs are always unique.
• Implemented Edit|Delete and Edit|Merge operations, along with Edit|Undo and Edit|Undo All. If you perform a "merge", you should wait until the metrics of the merged unit are calculated and cached in the background before proceeding with further edits.

v0.1.0 (11/28/2023)

• Defined similarity metric: The correlation coefficient of two vectors: the horizontal concatenation of the per-channel spike template waveforms for one unit and similarly for the other unit.
• Updated NeuronView to include the similarity metric in neural unit table. The metric always compares each unit to the so-called primary neuron, ie, the first neuron in the current display/focus list. If no units are currently selected, then the similarity metric is undefined, and all entries in the Similarity column are blank. As with any column in the table, you can sort on the new column. When similarity is undefined, sorting on that column is the same as sorting on the UID.
• Removed UMAPView. UMAP (Uniform Manifold Approximation and Projection) analysis is too slow. May revisit this decision in a future release.
• Initial implementation of ACGRateView, which plots the autocorrelogram as a function of firing rate for each of the units in the current display/focus list.
• Project dependencies now maintained in requirements.txt.

v0.0.9 (11/20/2023)

• Modified approach to principal component analysis for the PCAView. Instead of using a random sampling of 1000 spike multi-clips (horizontal concatenation of the 2ms spike clips recorded on P analog channels) across all units, we use the spike template waveform clips (first 2ms), resulting in a Kx(MxP) matrix where K is the number of units and M is the number of analog samples in 2ms. The PCA projections for each unit's spike trains are then computed as before. This achieves better separation of the unit projections in PCAView. Hoever, if only one unit is in the focus list, we revert to using the random sampling of individual clips to calculate principal components.

v0.0.8 (11/15/2023)

• Requires Python 3.9+ instead of 3.11+ (although still building on 3.11).
• A dashed green line in FiringRateView indicates the elapsed time at which the channel trace segments start in the ChannelView.
• Clicking anywhere on the FiringRateView changes the segment start time to the time under the cursor; the ChannelView updates accordingly.
• Developed code to perform principal component analysis on the spike trains of the neural units currently comprising the focus list. PCA computations take a while -- particularly when the total number of spikes processed reaches 100K or more.
• Every time the user changes the focus list, a new background task is launched to compute statistics (ISI, ACG, CCG, PCA) for the units in the list. The ISI, ACG and CCG are only computed once and cached in-memory in the relevant Neuron objects. The PCA projections are redone each time the focus list changes.
• Since the user may change the focus list rapidly and the statistics calculations are time-consuming, XSort will cancel a running task of a given type before launching a new one. This strategy also applies if the user opens a new working directory, then after a short while decides to switch again.
• The results of the PCA analysis are now rendered in the PCAView. The view includes a pushbutton to toggle the display order of the scatter plots, and a combox box to set the downsampling factor in order to speed up rendering.
• The SimilarityView was eliminated. Once we have a similarity metric defined, that will be included as a column in the NeuronView.
• Moved README and CHANGELOG from $PROJECT_ROOT/dist to $PROJECT_ROOT/ so that Gitlab picks up the README.

v0.0.7 (10/11/2023)

• Turned on antialiasing in the pyqtgraph library. Eventually, this may become a user setting, especially if it causes performance issues.
• Enhancements to FiringRateView:
  1. User can opt to display the firing rate histograms as raw bin counts or normalized.
  2. The histograms are now rendered more accurately as "staircases".
  3. Added a vertical-line "time cursor" that follow the mouse as it moves within the view. A label near the bottom shows the elapsed time in the format MM:SS.
  4. A colored, labeled marker is drawn where the time cursor intersects each displayed histogram, and the label reflects the approximate Y-coordinate value for that histogram at the elapsed time of the recording.
  5. Both X- and Y-axes are not really needed, so they are hidden.
• Bug fix in NeuronView: On Mac OS, when the system switched automatically to a night time color scheme, could no longer read the rows in the neural unit table because the background color was hard-coded to white -- and the night time default foreground color is also white!

v0.0.6 (10/09/2023)

• Initial implementation of FiringRateView, displaying firing rate over the course of the recording, normalized by dividing by the overall mean firing rate.
• Split StatisticsView into 2 separate views, ISIView and CorrelogramView. The latter renders the ACGs and CCGs for the N neurons in the current display list as a NxN matrix of subplots, with the ACGs along the major diagonal.

v0.0.5 (10/09/2023)

• Initial implementation of StatisticsView, displaying ISI histograms and ACGs for all neural units in the current display list, as well as the CCG of the first neuron in that list vs the others.
• Fixed bug in assigning display colors to neural units in the display list.
• Fixed bug in code that retrieved channel trace segments whenever user changed the segment start time using the slider in the ChannelView.
• Modified how changes in the slider position are handled in TemplateView, ChannelView, and StatisticsView since the QSlider.sliderReleased signal did not fire in RHEL.
• Known issues: (1) Keyboard interface to slider operation does not work in Mac OS. (2) The background task which generates ACGs and CCGS for neural units in the current display list is computationally intensive and causes noticeable display lags in the GUI under certain situations. May have to run these in a separate process, or compute once and cache to file. File IO retrievals do not block the GUI like heavy-duty computations -- a Python GIL issue.

v0.0.4 (10/02/2023)

• User can select up to 3 neural units in the NeuronView for inclusion in the display list, which affects what is rendered in other views. A display color is assigned to each "slot" in the display list: blue, red, yellow.
• For each neuron in the display list, ChannelView now displays 10-ms clips at every spike occurrence on the displayed trace segment for that neuron's so-called primary channel (analog channel with the highest estimated SNR for that neuron). The clips are drawn by tracing over the trace segment in the clip interval using the display color assigned to the neuron (but translucent).
• A slider in the ChannelView lets the user look at any 1-second segment within the Omniplex recording. Furthermore, the user can use the mouse wheel to zoom in/out and pan the view. Minimum time span (x-axis) is 100ms. Minimum voltage range is enough to show 2 adjacent channel traces (they're arranged in descending order from top to bottom).
• Initial implementation of TemplateView. Displays per-channel spike templates computed for each of units in the current display list. Template width set by user, between 3 and 10ms -- saved and restored as a view-specific user setting.

v0.0.3 (09/25/2023)

• Developed critical background tasks for analyzing the required source files (Omniplex PL2, spike sorter results) in the current working directory, caching analog channel traces in separates files within the directory, computing and caching neural unit metrics.
• Completed initial implementations of the NeuronView and ChannelView.

v0.0.2 (09/05/2023)

• Internal release after restructuring the application code several times.
• Added notion of the current working directory, which is also saved to user settings so that XSort can restore the working directory at app launch.

v0.0.1 (08/01/2023)

• Replaced tkinter with Qt/PySide for the GUI framework. Docking/floating functionality much improved.
• Saves and restores main window geometry and dock widgets's state via Qt's preferences support (QSettings).

v0.0.0 (07/26/2023)

• Skeleton application with no functionality except some blank views that can be docked or floated as top-level windows. To test feasibility of using Qt for cross-platform development.