Stage 7: Apply filters

Author: Hadi Zaatiti <hadi.zaatiti@nyu.edu>

What & why

Stage 7 of the KIT general pipeline produces the cleaned continuous recording that every sensor-level and source-level analysis reads. For each stage-2 FIF it:

  1. writes the bad channels decided in stage 5 into info['bads'];

  2. removes the ICA components excluded in stage 6 from the unfiltered data (COGNESTIC recipe: the ICA is fitted on a 1 Hz high-passed copy in stage 6 and applied to the original data here);

  3. applies a notch filter at the power-line frequency and its harmonics;

  4. applies a band-pass filter;

  5. saves the result as *_desc-filtered_raw.fif with a .json sidecar that records exactly what went in.

The stage is the point where the human decisions of stages 5 and 6 enter the data. It never makes a decision of its own: no channel is added or removed from the list, no component is chosen. Downstream, stage 9 only epochs the filtered FIFs that stage 7 lists as valid, so that a bad-channel list or an ICA review that changed after filtering cannot silently reach the epochs (see How later stages use the result).

What you need to provide

  • The stage-2 outputs (2-kit_con_to_fif/kit2fiff_summary.csv and the *_desc-rawkit_raw.fif files). Only FIFs listed in the summary are filtered.

  • Optionally the stage-5 record of each recording (*_desc-badchannels.json, reviewed). Without it, the filtered FIF keeps whatever info['bads'] the stage-2 file carried (normally none) and the summary says bad_channel_source: no_record.

  • Optionally the stage-6 solution (*_desc-ica_ica.fif and *_desc-ica.json, reviewed). Without it no component is removed and the summary says ica_source: no_record.

Run stage 5 and stage 6 before stage 7 whenever the study uses them: a filtered FIF made before the reviews has to be remade (--overwrite) after them.

Configuration file

filters:
  notch:
    enabled: true          # default: true
    freqs: [50, 100, 150]  # optional explicit list; omitted = harmonics of the line frequency
    # kwargs:              # optional extra arguments passed to raw.notch_filter
    #   notch_widths: 2.0
  bandpass:
    l_freq: 1.0            # default: 1.0 Hz
    h_freq: 40.0           # default: 40.0 Hz
    # kwargs:              # optional extra arguments passed to raw.filter
    #   method: fir
    #   phase: zero
    #   fir_window: hamming

Notes:

  • The line frequency is NYUAD_KIT_CONSTANTS.LINE_FREQUENCY (50 Hz at NYUAD). When freqs is omitted the stage notches the fundamental and its harmonics below the Nyquist frequency, at most three frequencies (50, 100, 150 Hz at 1000 Hz sampling).

  • The whole filters block is optional: without it the stage uses the defaults above (notch at the 50 Hz harmonics, band-pass 1 to 40 Hz). These defaults suit event-related analyses; raise h_freq for gamma or high-frequency work and set l_freq to null for a low-pass only.

  • kwargs are passed unchanged to MNE (raw.notch_filter and raw.filter); use them to change the filter design, not to add frequencies.

  • The applied settings are part of the output’s provenance: changing any of them makes the existing FIF existing_output_unverified (see below).

How to run

conda activate neurowaves
python -m pipeline.mne_pipelines.kit_general_pipelines.7-kit_apply_filters \
    -c pipeline/mne_pipelines/kit_general_pipelines/pipeline_config_files/<project>_config.yml

Options:

  • --overwrite: replace filtered FIFs that already exist. Required after a new stage-5 or stage-6 review, after a stage-2 rerun, or after changing the filters block.

The VS Code launch entries are named <project> 7 Apply filters.

What the script does

Per recording listed in the stage-2 summary:

  1. Resolve the stage-5 bad-channel record and the stage-6 ICA record of the recording. Each comes back with a source status: ok (reviewed, made from this FIF), pending_review (computed but not reviewed: only the human-provided channels of the YAML manual list are applied, automatic candidates are not; for the ICA only the YAML manual_exclude components are removed, proposals are not), stale_input (made from an older version of the FIF; applied anyway, with a warning asking to rerun stage 5 or 6), missing_solution (ICA record without its _ica.fif; no ICA applied) or no_record.

  2. If the output exists and --overwrite is not given, check its sidecar: same input FIF (path, size, modification time), same output file, same filter settings, same bad-channel list and same ICA exclusions. All equal: the row is skipped and the file is kept. Anything differs: the row is error with existing_output_unverified: <reason>; use --overwrite and the file is left untouched, so stage 9 stops using it until it is remade.

  3. Otherwise load the stage-2 FIF, set info['bads'] to the union of the FIF’s own list and the stage-5 list (channels unknown to the FIF are dropped with a warning), apply the ICA with its exclusions, notch, then band-pass, and save with *_desc-filtered_raw.fif in place of *_desc-rawkit_raw.fif.

  4. Write the sidecar <fif>.json: identity of the input and of the output, the filter settings, bad_channels (the stage-5 decision), bad_channels_in_output (what info['bads'] contains), bad_channel_source and bad_channel_record, ica_excluded, ica_source and ica_record.

  5. Append a block to sub-<id>/filter_log.txt and a row to the summary.

Outputs

Under <bids_root>/derivatives/7-kit_apply_filters/:

filter_summary.csv
configs/config_<timestamp>.yml
sub-<id>/[ses-<id>/]
  sub-<id>..._desc-filtered_raw.fif          # info['bads'] set, ICA applied, notch + band-pass
  sub-<id>..._desc-filtered_raw.fif.json     # provenance (input, settings, bad channels, ICA)
  filter_log.txt

filter_summary.csv has one row per recording: the BIDS entities, in_fif, out_fif, notch_enabled, line_freq, bandpass_l_freq, bandpass_h_freq, bad_channels (semicolon separated), bad_channel_source, ica_excluded, ica_source, status (success | skipped | error) and error. Rows of recordings outside the run’s selection are kept.

Browsing the filtered data in MNE

The report’s stage-7 section ends with this code, filled in with one of the dataset’s recordings: it opens a filtered recording in MNE’s interactive browser with the stage-8 events as markers (named after their conditions, events.conditions) and the stage-5 bad channels in grey. Run it in a Python with MNE and a display; block=True keeps the window open.

import os
from pathlib import Path
import mne
from pipeline.mne_pipelines.kit_general_pipelines.results import Results
res = Results(bids_root=Path(os.environ["MEG_DATA"]) / "epilepsy-project")   # the dataset under $MEG_DATA

for r in res.recordings(7):                 # every filtered recording
    print(r.stem)
rec = res.recording("0989", task="flippedsquares", run="01", session="01")

raw = rec.raw("filtered", preload=True)     # bad channels marked, ICA applied, notch and band-pass
try:
    events, event_id = rec.events()         # stage-8 events with their condition names
except FileNotFoundError:
    events, event_id = None, None           # a recording without events (sleep, rest)

mne.viz.set_browser_backend("qt")           # mne-qt-browser; "matplotlib" works too
raw.plot(events=events, event_id=event_id, duration=20.0, n_channels=40,
         scalings=dict(mag=2e-12), title=rec.stem, block=True)

# before stage 7, to compare:   rec.raw("converted", preload=True).plot(...)
# spectrum after filtering:     raw.compute_psd(fmax=60).plot(picks="mag", exclude="bads")

A channel marked bad by clicking in the browser stays in that Python session: the pipeline’s bad channels come from the stage-5 review.

How later stages use the result

  • Stage 9 reads filter_summary.csv and epochs only the FIFs whose row is success or skipped. A row in error because of existing_output_unverified makes stage 9 report blocked_unverified_filter for that recording; a missing row, missing_filtered_input. Stage 9 copies bad_channels, ica_excluded and their sources into its own summary and into the *_ave.json of every recording, so an evoked response always says which cleaning it received.

  • Stages 14 and 15 take the sensor information and the noise covariance from the stage-9 outputs, which carry the same info['bads'].

  • The report shows the summary and, per recording, the applied bad channels and ICA exclusions next to the filter settings.

Edge cases & safeguards

  • Review changed after filtering: the existing FIF is reported, not reused and not overwritten; rerun with --overwrite. This is deliberate: a cleaning decision must never apply to some outputs and not others.

  • Stage 5 not reviewed (pending_review): the candidates are not applied; the summary and the console show it. Finish the review in stage 5, then rerun stage 7 with --overwrite.

  • Stage 6 solution missing (missing_solution): the .json names an _ica.fif that is gone; refit in stage 6.

  • Bad channel absent from the FIF: listed in the stage-5 record but not in raw.ch_names; dropped with a warning (unknown in the log).

  • Split recordings (*_raw-1.fif) are handled by MNE through their base file; one filtered output per recording is written.

Troubleshooting

  • ``existing_output_unverified``: see above; --overwrite.

  • Memory: the recording is loaded in full (preload=True) for the ICA and the filters; long sleep recordings at 1000 Hz need several GB. Close other MEG tools, or run subject by subject with subjects.include.

  • Filter edge artefacts in the epochs: the band-pass is applied to the continuous data, so the only edges are the start and the end of the recording; events within the first second of a recording still get a transient. Check n_dropped in stage 9.

  • No row for a recording: it is missing from the stage-2 summary or excluded by bids_selection; stage 7 never looks for FIFs on disk by itself.