Stage 3-a: Bad channel detection

Author: Hadi Zaatiti <hadi.zaatiti@nyu.edu>

What & why

Stage 3-a of the KIT general pipeline establishes, for every recording converted by stage 2, the list of bad MEG channels that the rest of the pipeline uses: stage 4 writes it into info['bads'] of the filtered FIF, and ICA, epoching and the report inherit it from there.

The stage is built on one rule: no channel is labelled bad by an algorithm. A human looks at every recording, the automatic checks only propose channels, and every proposal is shown to the reviewer with its evidence before it can enter the list. Per recording the script therefore runs three passes, always in this order:

1. Manual inspection in MNE’s raw-data browser, before any automatic result exists (unbiased).

2. Automatic checks, computed in a separate process while you inspect, which turn the MNE bad-channel detectors into a list of candidates.

3. Human confirmation, where the candidates are shown pre-marked in the browser together with their metrics and figures; what is still marked when the window closes is the decision.

Channel conventions

On the NYUAD KIT system, as read by MNE:

• the 207 head sensors are KIT axial gradiometers; MNE types them mag (coil FIFFV_COIL_KIT_GRAD, unit tesla). These are the channels inspected and audited, and the only ones that can be marked bad.

• the 16 reference magnetometers, and MEG 092 (a broken sensor retyped as a reference at acquisition), are ref_meg. They sit far from the scalp and are never part of the analysis: the stage excludes them from every pick, and so do the later stages.

The picks used everywhere are mne.pick_types(info, meg=True, ref_meg=False) which returns exactly the 207 gradiometers.

What you need to provide

• The stage-2 outputs: <bids_root>/derivatives/2-kit_con_to_fif[_manual]/ with kit2fiff_summary.csv and the *_desc-rawkit_raw.fif files. Only FIFs listed in the summary are processed; split parts (*_raw-1.fif) are read through their base file.

• A desktop session with a display, a Qt binding (PySide6 or PyQt5) and mne-qt-browser (both in the neurowaves recipe). Without mne-qt-browser MNE falls back to its matplotlib browser, which redraws every scroll and is far too slow for 207 channels; the console prints the backend used. Start the stage with Run Without Debugging (Ctrl+F5) in VS Code: under the debugger the browser is noticeably slower.

• scikit-learn for the LOF check (in the environment). Without it the LOF column is empty and the other checks still run.

Configuration file

The stage reads the bad_channels block of the same YAML as every other stage (-c / --config; default config_template.yml). All keys are optional; the values below are the defaults, which are also the KidLang settings:

bad_channels:
  enabled: true
  manual: {}                 # channels always bad, per subject: {Y0312: ["MEG 045"]}
  ignore_detected: {}        # candidates never proposed, per subject: {Y0312: ["MEG 150"]}
  segments:                  # evenly spaced segments analysed by the checks
    count: 8
    duration_s: 60
    skip_edges_s: 5          # seconds ignored at both ends of the recording
  bandpass: [1.0, 40.0]      # band for the variance, LOF and rank checks and for the browser
  resample_hz: 200           # rate for the variance and LOF checks (after band-passing)
  flat:
    threshold_t: 1.0e-16     # consecutive-sample peak-to-peak below this = flat sample
    bad_percent: 5.0         # % of a segment flat -> the segment flags the channel
    min_segment_fraction: 0.5  # fraction of segments flagging -> flat channel
  peak:
    threshold_t: 5.0e-12     # consecutive-sample step above this = jump (SQUID resets)
    bad_percent: 5.0
    min_segment_fraction: 0.5
  min_duration_s: 0.005      # annotate_amplitude: shortest flat/peak stretch considered
  variance:   {z_threshold: 5.0}
  psd:        {fmin: 1.0, fmax: 40.0, z_threshold: 5.0}
  line_noise: {enabled: true, z_threshold: 5.0, min_db: 10.0}
  lof:        {enabled: true, n_neighbors: 20, threshold: 1.5}
  maxwell:    {enabled: false, limit: 7.0, duration: 5.0, min_count: 5,
               min_segment_fraction: 0.5}   # opt-in, ~40 s per segment
  decision:
    min_detectors: 2         # flat alone proposes; other checks need this many in agreement
  save_plots: true
  review:
    scaling_t: 2.0e-12       # browser trace scaling for the gradiometers (tesla per division)
    cut_duration_s: 20       # seconds shown from the start of each segment (8 x 20 s = 2:40)
    resample_hz: 150         # rate of the browser data (0 = the recording's rate)
    window_s: 10             # seconds per browser page
    n_channels: 40           # channels per browser page
    antialiasing: false      # smoother traces; seconds per redraw with Qt 6 software rendering
    use_opengl: false        # GPU drawing, fast even with antialiasing (needs an OpenGL driver)
    whole_recording: false   # true = browser shows the entire recording (slow)

manual and ignore_detected are keyed by subject label with or without the sub- prefix. manual channels end up in the final list of every recording of that subject whatever the review decides; ignore_detected channels are removed from the candidates (useful for a known false positive) but can still be marked by hand.

The stage also uses project, subjects and bids_selection exactly as stage 2 does, and reports.scripts.3-a-kit_detect_bad_channels to enable or disable the report refresh after it.

How to run

From the repository root with the neurowaves environment activated:

python -m pipeline.mne_pipelines.kit_general_pipelines.3-a-kit_detect_bad_channels \
    -c pipeline/mne_pipelines/kit_general_pipelines/pipeline_config_files/kidlang_config.yml

Options:

Flag	Effect
-c, --config PATH	YAML config. Always pass the dataset’s file; the default is the template.
(no flag)	Interactive run: inspection, automatic checks and confirmation for every recording that has no saved review. Reviewed recordings are left as they are (their summary row is refreshed), unless new candidates appeared since the review (see below).
--no-gui	Compute the candidates, metrics, tables and figures only; label nothing. Records get review_status: pending and stage 4 applies only the human-provided channels until a reviewer runs the GUI. Needs no display; use it to precompute the evidence on a server.
--review-all	Redo the inspection and confirmation also for reviewed recordings. The browser starts from the previous decision.
--reset-review	Discard the saved reviews first; every recording becomes pending. With the GUI the reviews start from scratch, with --no-gui the reviews are just cleared.
--overwrite	Recompute the metrics even when a record with the same input and detection settings exists. Reviews are kept.
--no-plots	Skip the figures (the confirmation pass then only shows the interactive PSD and the browser).

Rerunning the stage is cheap: a recording whose FIF (path, size and modification time) and detection settings have not changed reuses the saved metrics, and only its decision rule and ignore list are re-applied. Changing any detection threshold recomputes that recording on the next run.

The VS Code launch entries (Kidlang) Bad channels: inspect + confirm (GUI), (Kidlang) Bad channels: candidates only (no GUI) and (Kidlang) Bad channels: redo all reviews (GUI) run the three usual variants.

Recommended workflow

1. Optional: run with --no-gui on all subjects (unattended) so the metrics and figures are ready.

2. Run without flags on the workstation and review subject after subject. The stage can be stopped and restarted at any time: finished recordings are kept, their summary rows are rewritten on the next run, and the report follows each review.

3. Run stage 4 (4-kit_apply_filters.py) afterwards; use --overwrite if a filtered FIF already exists, so the new list is written into it.

Pass by pass

Pass 1: manual inspection

For each recording the stage first builds a compact review copy of the data, because handing a 40-minute recording to the browser means it is preloaded and filtered in full before the window appears:

• the first review.cut_duration_s seconds (20 s) of each automatic segment are cropped, so the 8 cuts (2:40 in total) sit at the same positions along the recording as the data the checks analyse (the segment plan is described under pass 2 below);

• only the 207 gradiometers are kept, each cut is band-passed once with bandpass (1 to 40 Hz) and resampled to review.resample_hz (150 Hz; the band-passed signal has nothing above 40 Hz, so this loses no information), then the cuts are joined end to end;

• an annotation at the start of each cut tells where it comes from (cut 3/8: 10:55-11:25 of the recording) and MNE marks every join with BAD boundary / EDGE boundary annotations.

review.whole_recording: true shows the complete recording instead, with the browser filtering on the fly (several minutes to open for KidLang recordings).

The browser (raw.plot, Qt backend from mne-qt-browser) opens on this review copy, review.n_channels (40) channels and review.window_s (10 s) at a time, with the trace scaling from review.scaling_t. Every scroll or key press redraws one page, so the page size and the drawing mode decide how snappy the browser feels. The stage forces the browser’s antialiasing to review.antialiasing and passes review.use_opengl: mne-qt-browser otherwise restores antialiasing from its saved settings, and with Qt 6 software rendering that costs seconds per redraw (measured on the lab workstation, MNE 1.13 with PySide6 6.11: 3.7 s per scroll antialiased, 113 ms without antialiasing, 51 ms antialiased on the GPU with use_opengl: true). The value the stage sets is saved by the browser on close, so it also applies to any other raw.plot you open afterwards. Only human-provided marks are pre-set: info['bads'] of the stage-2 FIF and bad_channels.manual[subject]. No automatic result is visible yet.

In the browser:

• click a channel name to mark it bad (the trace turns grey) or to unmark it;

• arrow keys scroll through channels and time, Home / End change the window length, + / - the scaling, a opens the annotation tool (annotations made here are not saved);

• close the window to continue. The channels marked at that moment are the inspection decision.

Pass 2: automatic checks

In an interactive run the checks are computed in a separate process (not a thread, so they never compete with the browser for Python’s interpreter lock; its BLAS threads are capped at half the cores) while pass 1 is open, so there is normally no wait between the two windows: with the default checks a 42-minute KidLang recording takes about 40 s, and the console shows the progress and the total time. With --no-gui they run in the foreground.

Segment plan: which parts of the recording are analysed. The checks do not read the whole recording, and they do not cut it into consecutive pieces. They take segments.count (8) snapshots of segments.duration_s (60 s) each, spread at equal intervals over the recording; everything between two snapshots is skipped:

• the first and the last segments.skip_edges_s (5 s) of the recording are never used (the system settling, the participant moving at the start and the end);

• the 8 snapshots are placed so that the first starts right after the skipped start, the last ends right before the skipped end, and the gaps between them are all equal: for a recording of T seconds the starts are 5 s, 5 s + g, 5 s + 2g, … with g = (T - 10 - 60) / 7;

• a recording too short for 8 non-overlapping snapshots gets as many as fit (a 5-minute recording gets 4); a recording shorter than one snapshot is analysed whole.

Example, a 42.5-minute KidLang split (T = 2550 s): the snapshots start at 5, 360, 714, 1069, 1424, 1779, 2133 and 2488 s, one about every 5 minutes; 8 minutes of data are analysed out of 42.5:

time (s)  0    5      65          360    420               2488  2548 2550
          |----[######]-----------[######]----- ... -------[######]----|
               snapshot 1          snapshot 2               snapshot 8
          ####### = 60 s analysed by every check      ----- = skipped

The exact windows of every recording are stored in its record (segments) and drawn as a timeline in the report; the review copy of pass 1 is cut from the same windows (the first review.cut_duration_s seconds of each), so the reviewer looks where the checks looked.

Why snapshots rather than the whole recording:

1. Coverage from start to end. Channels that are intermittently bad, die part-way or start noisy are caught by the per-segment rules: flat and peak must occur in at least min_segment_fraction (0.5) of the snapshots, and the other metrics are the median over snapshots, so one transient artefact never flags a channel.

2. Bounded memory and time. One 60 s snapshot of 207 channels at 1000 Hz is about 100 MB; all checks on 8 snapshots take about half a minute per recording, whereas the whole 42-minute recording would mean several gigabytes per filtered copy.

3. Stable spectra. Welch’s method with 2 s windows on 60 s averages about 30 periodograms per snapshot.

8 x 60 s is a conservative default. Raising segments.count brings the snapshots closer together (at count = T / duration_s they become consecutive and cover everything), at a proportional cost in time.

On each segment, for each of the 207 gradiometers:

Check	Metric (what MNE provides, what the stage adds)	Flags a channel when
flat	mne.preprocessing.annotate_amplitude on the unfiltered segment (consecutive-sample peak-to-peak below flat.threshold_t for at least min_duration_s). The stage also stores the fraction of flat samples (flat_fraction) and merges the flat list of the Maxwell check.	the segment flags it (more than flat.bad_percent of the segment flat) in at least flat.min_segment_fraction of the segments, or Maxwell finds it flat in at least maxwell.min_segment_fraction
peak	same function, consecutive-sample step above peak.threshold_t (SQUID jumps, resets); peak_fraction stored.	flagged in at least peak.min_segment_fraction of the segments
variance	standard deviation of the segment band-passed with bandpass and resampled to resample_hz, as log10 (std_log10); the stage converts it to a robust z-score across channels (median and MAD, std_z).	|std_z| >= variance.z_threshold (dead channels are low, noisy ones high)
psd	Welch PSD of the unfiltered segment (Raw.compute_psd, 0.5 to 100 Hz, 2 s windows); mean log power in psd.fmin to psd.fmax (psd_db) and its robust z-score (psd_z).	|psd_z| >= psd.z_threshold
line_noise	from the same PSD: power at the mains frequency (50 Hz plus or minus 1 Hz) over the median power 3 to 10 Hz away on either side, in dB (line_noise_db), and its robust z-score (line_noise_z).	line_noise_z >= line_noise.z_threshold and line_noise_db >= line_noise.min_db
lof	mne.preprocessing.find_bad_channels_lof (Local Outlier Factor of each channel’s band-passed time course against its lof.n_neighbors most similar channels; MNE 1.7 or newer); lof_score is the outlier factor, about 1 for normal channels.	median lof_score >= lof.threshold
maxwell	mne.preprocessing.find_bad_channels_maxwell (Signal Space Separation residuals, limit / duration / min_count), with ignore_ref=True and a head-frame then device-frame attempt. Opt-in (maxwell.enabled: true): experimental on KIT data (no fine-calibration file) and about 40 s per segment, so 5 minutes per recording.	flagged noisy in at least maxwell.min_segment_fraction of the segments
rank	mne.compute_rank of the band-passed segment, for information only (estimated_rank in the record and summary).	never

Across segments the metrics are combined by their median (fractions for the per-segment flags), so a transient artefact in one segment does not flag a channel. The robust z-scores are (x - median) / (1.4826 * MAD) over the 207 channels, which keeps one dead channel from shifting the scale for the others.

A channel becomes a candidate when it is flat, or when at least decision.min_detectors (2) of the other checks agree; channels in ignore_detected[subject] are removed. Candidates are proposals and nothing more: they are never written as bad on their own.

Pass 3: human confirmation

If every candidate was already marked in pass 1, nothing is asked. Otherwise the stage:

1. prints each new candidate with the checks that flagged it and its metrics (std z, psd z, line noise in dB, LOF, flat and peak fractions);

2. opens the saved PSD, metric and trace figures and an interactive PSD of the first segment (click a trace to print its channel name);

3. reopens the browser on the review copy, pre-marked with your inspection marks plus the candidates.

Keep a candidate marked to confirm it, click its name to reject it; you can also mark further channels. Closing the window records the decision: the channels still marked, plus the manual list, are the final bad channels (review_status: reviewed), and the unmarked candidates are stored as rejected so they are not asked again.

When candidates appear for an already reviewed recording (a looser decision.min_detectors, a channel removed from ignore_detected), an interactive run opens pass 3 for just those channels, starting from the previous decision; a --no-gui run reports them as unseen_candidates and leaves the decision untouched.

With --no-gui pass 1 and pass 3 do not run: the record keeps review_status: pending and final_bads holds only the human-provided channels; the console prints PENDING REVIEW for every recording with candidates.

Outputs

Everything is written under the stage’s own folder; the folders of the other stages are not touched.

<bids_root>/derivatives/3-a-kit_detect_bad_channels/
  config_<timestamp>.yml                 # copy of the config used by each run
  bad_channels_summary.csv               # one row per recording, whole cohort
  sub-<id>/
    bad_channel_log.txt                  # one block per processed recording
    [ses-<id>/]
      sub-<id>..._desc-badchannels.json          # decision, review, settings, provenance
      sub-<id>..._desc-badchannels_metrics.tsv   # one row per gradiometer
      figures/
        sub-<id>..._desc-badchannels_psd.png     # Welch PSD, candidates and bads coloured
        sub-<id>..._desc-badchannels_metrics.png # variance z vs LOF, band power z vs line noise
        sub-<id>..._desc-badchannels_sensors.png # sensor layout, final bad channels in red
        sub-<id>..._desc-badchannels_traces.png  # bad channels against their 3 nearest neighbours

The JSON record (<entities>_desc-badchannels.json) is what later stages read. Its main fields:

• final_bads, review_status (reviewed | pending);

• candidates, by_detector (channels per check), unseen_candidates;

• review: inspection (pre-marked channels, marks, timestamp, browser backend, data shown) and confirmation (candidates, confirmed, rejected, marks, timestamp), bads, started_from (fresh | previous_review);

• pre_existing_bads (info['bads'] of the FIF), manual, ignored, unknown_manual (manual channels absent from the recording);

• input (path, size, modification time of the FIF, used to detect a changed input), detection_settings, decision_settings, segments, estimated_rank, channels_evaluated, detector_errors, figures, metrics_tsv, mne_version.

The metrics TSV has, per channel: flat_fraction, peak_fraction, std_log10, std_z, psd_db, psd_z, line_noise_db, line_noise_z, lof_score, maxwell_noisy_fraction, maxwell_flat_fraction, detectors (semicolon list), n_detectors, one flag_<check> column per check, and the booleans pre_existing_bad, candidate, manual, ignored, inspection_bad, confirmed_bad, rejected_candidate, final_bad.

bad_channels_summary.csv has one row per recording: the BIDS entities, in_fif, n_channels, estimated_rank, n_segments, one n_<check> count per check, pre_existing_bads, manual, ignored, inspection_bads, candidates, confirmed, rejected, review_status, final_bads, n_final_bads, record_json, metrics_tsv, status (success | error), error and mne_version. Channel lists are semicolon separated. The file is rewritten after every recording: the row of that recording replaces the previous one and all other rows are kept, so the summary always covers the whole cohort whatever subset the run processed, and an interrupted run loses nothing.

How later stages use the result

utilities.final_bad_channels_for_recording(bids_root, entities, source_fif) returns (bad_channels, status, record_path) with status:

• ok: reviewed record, computed from the current FIF;

• pending_review: candidates exist but nobody confirmed them; the list holds only human-provided channels;

• stale_input: the record was computed from a different version of the FIF (stage 2 was rerun); the list is returned anyway;

• no_record: the stage has not run for this recording.

Stage 4 calls it for every recording, warns on pending_review and stale_input, sets info['bads'] before filtering, stores the list in the .json provenance sidecar (bad_channels, bad_channel_source, bad_channel_record) and in the bad_channels column of filter_summary.csv, and reports an existing filtered FIF whose list differs from the current decision as existing_output_unverified instead of reusing it (rerun stage 4 with --overwrite to refilter it). ICA and epoching read info['bads'] from the filtered FIF, so the whole pipeline shares one list.

Report

The section 3-a. Bad channel detection of the Quarto report explains the three passes and every check with the parameter values of the run, then shows the cohort table (reviewed and pending counts) and, per recording, a collapsed block with the review outcome, the metrics of the flagged channels and the four figures. The report is refreshed in the background after each reviewed recording (and after each subject in --no-gui runs), so it follows the reviews without waiting for the whole cohort; the Watch Quarto reports VS Code task renders it as well whenever a record or the summary changes.

Edge cases & safeguards

• No stage-2 output for a subject: skipped with a message.

• Short recordings: shorter than 2 s, the recording is refused with a clear error (status: error in the summary); shorter than two edge skips plus one window (130 s by default), the edge skipping is dropped or reduced and the whole recording is analysed as a single window (a 10 s file gives one 10 s window); shorter than count windows, as many non-overlapping windows as fit. The review copy follows the same windows, so it is never longer than the recording.

• FIF changed since the review (stage 2 rerun): the review is kept and a note suggests --review-all; stage 4 reports stale_input.

• Manual channel not in the recording: warning, recorded in unknown_manual, otherwise ignored.

• A check fails (for example Maxwell on a short segment): the error is stored in detector_errors and the error column, the other checks still count, status stays success.

• Interrupted run: finished recordings keep their record and summary row; rerun the stage to continue with the others.

• No Qt binding / no display: the interactive run stops at start with a message; use --no-gui or a machine with a display.

Troubleshooting

• Every click or scroll takes seconds: that is antialiased software rendering on Qt 6 (the Browser drawing: console line shows what the stage set). Keep review.antialiasing: false, or set review.use_opengl: true on a machine with a working GPU driver to have smooth traces at no cost. Also check the backend printed with the inspection result: matplotlib means mne-qt-browser is missing from the interpreter you selected (the stage warns at start; install it with pip install mne-qt-browser). Running under the VS Code debugger makes every click several times slower; the stage warns at start when it detects one, the GUI launch entries set noDebug, and Ctrl+F5 does the same for any entry.

• The browser takes long to open: check the Shown: line (review.whole_recording should be false). With the Qt browser and the default cuts the window opens about 10 s after the recording starts. For an even lighter page lower review.window_s, review.n_channels or review.resample_hz.

• Long wait after closing the inspection window: the automatic checks are still running; the console shows the segment progress. Keep maxwell.enabled: false unless you need that check, or lower segments.count.

• Traces look flat or clipped: adjust review.scaling_t (larger value = smaller traces) or use + / - in the browser.

• Too many or too few candidates: raise or lower the z_threshold values, lof.threshold or decision.min_detectors; the metrics TSV shows where each channel stands. Changing a threshold recomputes the affected recordings on the next run; changing decision.min_detectors or ignore_detected only re-applies the rule to the saved metrics.

• The report does not show a recording: check that its row is in bad_channels_summary.csv and that its status is success.

• Stage 4 warns about pending candidates: run stage 3-a with the GUI for that subject, then stage 4 with --overwrite.