(came first up in #93 (comment) and following comment in that thread.)
This isn't a PyPREP bug so much as a weirdness in the original PREP that PyPREP now matches. Essentially, during robust referencing, PREP performs an initial NoisyChannels pass and flags any bad-by-NaN, bad-by-flat, and bad-by-SNR channels as "unusable", permanently excluding them from being included in the calculated average reference signals:
|
# Determine channels to use/exclude from initial reference estimation |
|
self.unusable_channels = _union( |
|
noisy_detector.bad_by_nan + noisy_detector.bad_by_flat, |
|
noisy_detector.bad_by_SNR, |
|
) |
|
reference_channels = _set_diff(self.reference_channels, self.unusable_channels) |
|
self.reference_signal = ( |
|
np.nanmean(raw_tmp.get_data(picks=reference_channels), axis=0) * 1e6 |
|
) |
|
|
|
signal_tmp = self.remove_reference( |
|
signal, self.reference_signal, reference_index |
However, PREP doesn't include initial bad-by-SNR channels in the full set of bad channels to interpolate (see #91), so a channel that's initially bad-by-SNR prior to initial average referencing but is no longer bad afterwards will still get used for interpolating bads during the reference loop, despite being fully excluded from the average reference signal itself. This seems wrong, and like something worth fixing.
My gut instinct here would be to default to only marking initial bad-by-flat and bad-by-NaN channels as fully "unusable". I'd imagine that the original intent was to removing channels that contribute more noise than signal from contaminating the average reference, but given how sensitive the 'bad-by-correlation' default settings are (if over 1% of all correlation windows are below threshold, the whole channel is considered bad) and how that can change after average referencing, I don't think initial bad-by-SNR channels should be treated as permanently bad.
(came first up in #93 (comment) and following comment in that thread.)
This isn't a PyPREP bug so much as a weirdness in the original PREP that PyPREP now matches. Essentially, during robust referencing, PREP performs an initial NoisyChannels pass and flags any bad-by-NaN, bad-by-flat, and bad-by-SNR channels as "unusable", permanently excluding them from being included in the calculated average reference signals:
pyprep/pyprep/reference.py
Lines 230 to 235 in 1abda4d
pyprep/pyprep/reference.py
Lines 324 to 329 in 1abda4d
However, PREP doesn't include initial bad-by-SNR channels in the full set of bad channels to interpolate (see #91), so a channel that's initially bad-by-SNR prior to initial average referencing but is no longer bad afterwards will still get used for interpolating bads during the reference loop, despite being fully excluded from the average reference signal itself. This seems wrong, and like something worth fixing.
My gut instinct here would be to default to only marking initial bad-by-flat and bad-by-NaN channels as fully "unusable". I'd imagine that the original intent was to removing channels that contribute more noise than signal from contaminating the average reference, but given how sensitive the 'bad-by-correlation' default settings are (if over 1% of all correlation windows are below threshold, the whole channel is considered bad) and how that can change after average referencing, I don't think initial bad-by-SNR channels should be treated as permanently bad.