Stage 9: Epochs and evoked responses

Author: Hadi Zaatiti <hadi.zaatiti@nyu.edu>

What & why

Stage 9 of the KIT general pipeline cuts the cleaned continuous recording of stage 7 into epochs around the events written by stage 8 and averages them into one evoked response per event code. It is the last sensor-level stage before the anatomy and the source stages: stages 14 and 15 take the sensor information, the noise covariance and the evoked responses from its outputs, stage 17 runs the sensor-level cluster tests on its evoked files and stage 18 decodes from its epochs.

Everything the epochs carry was decided upstream and is only recorded here, never recomputed:

  • the bad channels of stage 5, written by stage 7 into info['bads']. MNE keeps them in the epochs (marked bad) and ignores them in the peak-to-peak rejection;

  • the ICA components of stage 6, removed by stage 7 before filtering;

  • the notch and band-pass of stage 7.

Stage 9 therefore accepts only the filtered FIFs that stage 7 lists as valid in filter_summary.csv. A FIF whose bad-channel list or ICA changed since it was filtered is reported, not epoched, and the stage never falls back to the unfiltered stage-2 file by itself.

What you need to provide

  • The stage-7 outputs (7-kit_apply_filters/filter_summary.csv and the *_desc-filtered_raw.fif files with their .json sidecars).

  • The stage-8 event files of the same recordings: one *_desc-<desc>_events.eve per recording in 8-kit_compute_events_single_channel/, 8-kit_compute_events_multi_channel/ or 8-kit_compute_events_STI014/ (the triggers_to_events folder of older runs is read as legacy). The third column of the .eve is the event code; one evoked response is computed per distinct code.

  • For an explicit unfiltered analysis only, the stage-2 outputs (epoching.input: raw): the stage-5 bad channels are still applied, no ICA and no filter.

Configuration file

events:
  desc: autopulses       # stage 8 writes this desc; stage 9 reads the matching .eve files
  source: null           # null = search the stage-8 branches; or single_channel | multi_channel | sti014 | legacy

epoching:
  input: filtered        # filtered (stage 7, default) | raw (stage 2, bad channels only)
  tmin: -0.2             # global defaults, seconds relative to the event
  tmax: 0.8
  baseline: [-0.2, 0.0]  # null = no baseline correction
  preload: true
  picks: []              # e.g. ["meg"], ["mag"]; empty = all MEG channels
  decim:                 # e.g. 4 to keep every 4th sample
  reject:                # peak-to-peak rejection, e.g. {mag: 4.0e-12} (decimal point: YAML reads 4e-12 as text)
  flat:                  # flatness rejection, e.g. {mag: 1.0e-15}
  windows:               # per-event-code overrides (keys are codes as written in the .eve)
    "224":
      tmin: -0.1
      tmax: 0.5
      baseline: [-0.1, 0.0]
    "231":
      tmax: 0.7            # tmin and baseline fall back to the global values
      baseline: null       # null = no baseline correction for this code
  tasks:                 # per-task settings (keys are BIDS task labels)
    twotones:
      tmin: -0.1
      tmax: 0.5
      baseline: [-0.1, 0.0]
    buttonlr:
      tmin: -0.8
      tmax: 0.5
      baseline: [-0.8, -0.6]
      reject: {mag: 5.0e-12}
    sleep:
      tmin: -1.0
      tmax: 3.0
      baseline: null
      windows:
        "224": {tmax: 5.0}

Notes:

  • events.desc must match what stage 8 wrote (autopulses for the analog variants, sti014 for the STI 014 variant by default); stage 9 looks for *_desc-<desc>_events.eve.

  • events.source: null searches all stage-8 branches and skips a recording that has an .eve in more than one of them (ambiguous_events); pin the branch to resolve it.

  • windows lets conditions have different epochs (a short window for a tone, a longer one for a picture). Codes not listed use the global values. Because windows may differ, epochs are saved per code.

  • tasks is the way to give different trial lengths to different tasks of one study (an auditory task, a motor task and a sleep recording rarely want the same window, and the same event code often means something else in each task). Each task may set tmin, tmax, baseline, reject, flat, decim, picks and its own windows per code. Every field is resolved on its own, from the most specific level to the global one: tasks.<task>.windows.<code> > windows.<code> > tasks.<task> > epoching. A window given for a code applies in every task (run markers, say) and a task block only supplies the fields it leaves out; a task that needs another window for that code gives it in its own windows. A field a level does not give falls through to the next, so sleep: {tmax: 3.0} keeps the global tmin. Write thresholds with a decimal point (reject: {mag: 5.0e-12}): YAML reads 5e-12 as text and the stage refuses it with a message. The level that was used is written to the summary and to the *_ave.json record (window_source), so a surprising window can be traced to its line in the YAML.

  • Keep the window of one condition the same across the subjects and recordings that stages 16 and 17 will average or test: those stages crop the estimates to the common time range, so a subject epoched with a longer window only loses its extra samples, but a different tmin shifts the baseline and changes what is compared.

  • The epoching block is optional; without it the defaults are -0.2 to 0.8 s with a baseline of -0.2 to 0 s, no rejection, all MEG channels.

  • The names of the codes (events.conditions) are used by the report and by stages 15 to 18; stage 9 itself works with the numeric codes.

How to run

conda activate neurowaves
python -m pipeline.mne_pipelines.kit_general_pipelines.9-kit_evoked_response_single_channel \
    -c pipeline/mne_pipelines/kit_general_pipelines/pipeline_config_files/<project>_config.yml

Options:

  • --overwrite: recompute recordings whose *_ave.fif already exists (otherwise they are reported as skipped_existing_output). A change of the epoching block or of the stage-8 .eve file is detected from the .json record and recomputes the recording without it (records written before 2026-10-21 lack the events identity: use --overwrite once); a stage-7 rerun still needs it.

The VS Code launch entries are named <project> 9 Epochs + evoked .... Despite its name, 9-kit_evoked_response_single_channel.py serves all three stage-8 variants; the name refers to the original trigger mode.

What the script does

Per recording of the stage-2 summary that matches the selection:

  1. Find the filtered FIF in filter_summary.csv. A row in error (existing_output_unverified) gives blocked_unverified_filter; no row gives missing_filtered_input. Read the stage-7 sidecar to learn the bad channels, the ICA exclusions and the filter settings that the file carries.

  2. Find the .eve of the recording in the stage-8 branch(es): none gives missing_events, more than one gives ambiguous_events.

  3. Skip the recording if its *_ave.fif exists, --overwrite is not given and neither the epoch settings, the .eve file nor the input FIF (size and content digest: a stage-7 rerun) changed since; when one did, the epochs and evoked files are replaced (no --overwrite needed) and the record is written after them.

  4. Load the FIF (preload=True) and the events. For every distinct event code: select its events, resolve the window for this code in this recording’s task (tasks.<task>.windows.<code>, windows.<code>, tasks.<task>, then the global values), build mne.Epochs with the resolved baseline, picks, decim, reject and flat, call drop_bad() and count the dropped epochs per reason (the first channel over the threshold, or TOO_SHORT / NO_DATA for windows that fall outside the recording). A code whose epochs are all dropped is no_usable_epochs.

  5. Average the epochs of the code into an Evoked with comment = code-<code>, save the epochs of the code as *_desc-epo_raw_ev<code>_epo.fif and append the evoked to the list of the recording.

  6. Save all evoked responses of the recording in one *_desc-evk_raw_ave.fif (mne.write_evokeds) and write the *_desc-evk_raw_ave.json record: input identity, events file, bad channels in the data and their source, ICA exclusions and their source, filter settings, the stage-7 sidecar path, the epoch settings, and per condition the window, n_events, n_epochs, n_dropped, drop_reasons and the epochs file.

  7. Write one summary row per event code and a block in sub-<id>/epochs_evoked_log.txt.

Outputs

Under <bids_root>/derivatives/9-kit_evoked_response_single_channel/:

epochs_evoked_summary.csv
configs/config_<timestamp>.yml
sub-<id>/[ses-<id>/]
  sub-<id>..._desc-epo_raw_ev<code>_epo.fif # one epochs file per event code
  sub-<id>..._desc-evk_raw_ave.fif          # all evoked responses of the recording
  sub-<id>..._desc-evk_raw_ave.json         # what went in, what came out, peaks, figures
  figures/sub-<id>..._ev<code>_desc-evokedjoint.png   # butterfly, GFP, topographies at the peaks
  figures/sub-<id>..._ev<code>_desc-epochsimage.png   # single epochs as an image (GFP over sensors)
  figures/sub-<id>..._ev<code>_desc-droplog.png       # rejected epochs per sensor (when any)

epochs_evoked_summary.csv has one row per recording and event code: the BIDS entities, input_kind (filtered | raw), raw_in, eve_in, bad_channels, ica_excluded, ev_code, tmin, tmax, baseline, window_source (the YAML level that set the window), n_events, n_epochs, n_dropped, drop_reasons, the peaks (peak_channel, peak_time_s, peak_ft: the largest absolute value over the KIT gradiometers; gfp_peak_time_s, gfp_peak_ft: the peak of the global field power, the standard deviation across sensors), epochs_out, evoked_out, status and message. Statuses: created, skipped_existing_output, no_usable_epochs, missing_events, ambiguous_events, missing_filtered_input, blocked_unverified_filter, failed_loading_input, failed_epoching, failed_evoked.

Report

The report’s stage-9 section explains epochs, baseline, rejection and averaging for a reader new to MEG, shows the epoching block of the latest run, and lists the full summary table: one row per recording and event code with the windows actually used, the number of events, kept and dropped epochs with the drop reasons, and the bad channels and ICA exclusions inherited from stage 7, then the sensor-level results: the peak table and, per recording, the joint plot of every event code (evoked.plot_joint: every sensor as a line, the GFP, and the field topography at the automatically found peaks), the single epochs as an image (epochs.plot_image) and the drop log when epochs were rejected (epoching.plots; --no-plots skips them). Each figure carries the code that reproduces it (see Working with the results in Python (the results module and the report’s code blocks)). The pipeline console shows per recording the number of codes and the total number of kept epochs, and names the blocking status (blocked_unverified_filter, missing_*) when there is one.

Edge cases & safeguards

  • Many dropped epochs with a channel name as the reason: the reject threshold is too strict for that channel, or the channel should have been marked bad in stage 5 (bad channels are excluded from the rejection). Check the stage-5 metrics for that channel before loosening reject.

  • ``TOO_SHORT`` / ``NO_DATA``: events closer to the recording edges than tmin / tmax; expected for the first and last events of a run.

  • Events but no epochs for a code present in .eve: look at drop_reasons; no_usable_epochs keeps a row so the report shows the condition as attempted.

  • Changed stage-8 output: the .eve identity is in the .json record and a changed file recomputes the recording.

  • Baselines: MNE’s form with null for the epoch start or end (baseline: [null, 0]) is accepted; a baseline that does not lie inside its epoch gives failed_epoching for that code, with the reason.

  • Mixed windows across codes: allowed; the evoked responses in one *_ave.fif may have different time axes. Stages 15 to 18 crop to the common window they need.

Troubleshooting

  • ``blocked_unverified_filter``: rerun stage 7 with --overwrite, then stage 9.

  • ``missing_filtered_input``: stage 7 never produced this recording (not in its selection, or an error); check filter_summary.csv.

  • ``ambiguous_events``: the same recording has an .eve in two stage-8 branches (for example an old single_channel run and a new sti014 one); set events.source.

  • ``missing_events``: events.desc does not match what stage 8 wrote, or stage 8 did not run for this recording.

  • Memory: the full recording is loaded; run subject by subject with subjects.include on small machines.