Stage 1: trigger count sanity check (single and multi channel)

Author: Hadi Zaatiti <hadi.zaatiti@nyu.edu>

What & why

1-kit_count_sanity_check_single_channel.py validates that the trigger pulses recorded on the analog MISC lines of a NYUAD-KIT MEG dataset match the event annotations in the BIDS *_events.csv / *_events.tsv table: same number of pulses per trigger channel, same chronological order. It detects lost, duplicated or misaligned triggers before any further processing.

The check is intended for datasets acquired in single-channel trigger mode (each event type on its own trigger line, KIT channels 224 to 231). A companion script, 1-kit_count_sanity_check_multi_channel.py, performs the equivalent check for binary-coded (multi-channel) trigger mode, where one event is encoded by several lines rising together.

Datasets whose triggers are read from the digital STI 014 channel (such as KidLang) do not go through this check; their events are extracted in stage 8 with 8-kit_compute_events_STI014.py.

What you need to provide

  • MEG recordings

    sub-<id>/[ses-<id>/]meg/sub-<id>[_ses-<id>]_task-<task>[_run-<n>]_meg.con

  • A paired events table and JSON sidecar at the same scope

    ..._events.csv   (or .tsv)
..._events.json

    The table must contain a channel column holding the KIT trigger channel number (224 to 231) of each event, in presentation order. The JSON must declare the trigger mode:

    { "TriggerMode": "single_channel" }

    Runs whose JSON does not declare single_channel (case, spaces and hyphens are normalised) are reported in the summary as skipped.

Fallback hierarchy for events files

Both files must exist at the same level. The search goes from the most specific to the least specific scope and stops at the first level that provides both:

  1. Exact match: sub-<id>[_ses-<id>]_task-<task>_run-<n>_events.*

  2. Subject + task: sub-<id>[_ses-<id>]_task-<task>_events.*

  3. Subject only: sub-<id>[_ses-<id>]_events.*

  4. Dataset root: events.csv + events.json next to dataset_description.json (applies to every subject)

A level that provides only one of the two files is ignored. The same policy is used by the stage-8 event-extraction scripts.

Configuration file

The stage reads the shared pipeline YAML (-c / --config; default pipeline_config_files/config_template.yml). Keys used:

project:
  name: "script-testing-dataset"
  root_env: "MEG_DATA"        # or root_override

subjects:
  include: []
  exclude: []

bids_selection:
  sessions: []
  tasks: []
  runs: []
  processings: []             # [null] = unprocessed .con only
  exclude: {}

events:
  multi_channel:
    max_onset_skew_ms: 2.0    # multi-channel variant only

reports:
  enabled: true

The pulse-detection thresholds are fixed in the script (see Tuning below); there are no YAML keys for them.

How to run

From the repository root, neurowaves environment activated:

python -m pipeline.mne_pipelines.kit_general_pipelines.1-kit_count_sanity_check_single_channel \
    -c pipeline/mne_pipelines/kit_general_pipelines/pipeline_config_files/config_template.yml

# binary-coded datasets
python -m pipeline.mne_pipelines.kit_general_pipelines.1-kit_count_sanity_check_multi_channel \
    -c <your_config.yml>

The VS Code launch entry KIT Sanity Check Single Channel runs the same command on the template config.

What the script does

  1. Loads the config and resolves the BIDS root (<parent>/<project.name>).

  2. Discovers subjects and .con recordings, applying bids_selection.

  3. Resolves the paired events table and JSON; skips the run if the pair is missing or TriggerMode is not single_channel.

  4. Reads the KIT recording (no preload) and the events table; checks that every channel value is one of the KIT trigger channels.

  5. Detects pulses on each of MISC 001 to MISC 008 with a robust threshold: baseline from the lower tail of the first 10 s, median + MAD threshold, hysteresis and debouncing. Spans annotated as bad are ignored without shifting sample indices.

  6. Compares detected pulses with the table: count per KIT channel and the chronological channel sequence.

  7. Computes pulse statistics (amplitude and width mean/variance), writes a per-run log and the summary CSV, saves a timestamped copy of the config and requests a report refresh.

Outputs

<bids_root>/derivatives/1-kit_count_sanity_check_single_channel/
  sanity_check_overview.csv
  configs/config_<timestamp>.yml
  sub-<id>/
    sub-<id>[_ses-<id>]_task-<task>[_run-<n>]_desc-sanitycheck_log.txt

The multi-channel variant writes to derivatives/1-kit_count_sanity_check_multi_channel/ with the same layout (its summary columns differ, see the multi-channel section below).

Per-run log

  • Raw file, events table and JSON paths, TriggerMode

  • Threshold per channel (thr_hi, thr_lo, pulse count)

  • Count comparison per KIT channel (csv_count / detected_count / diff)

  • Sequence check with the first mismatching position

  • Amplitude and width statistics

  • Final PASS/FAIL line (ASCII OK/FAIL on Windows consoles)

Summary CSV

sanity_check_overview.csv columns: subject, file, trigger_mode, csv_events, detected_events, counts_match, row_order_match, reference_channels_valid, pass, log_file, comments, mne_version. Skipped runs keep a row with the reason in comments.

Pass/Fail criteria

A run passes only if every KIT channel count matches the table exactly and the detected chronological channel order equals the table order.

Tuning & interpretation

The detector is detect_pulses_on_channel in kit_general_pipelines/utilities.py. The stage calls it with baseline_s=(0.0, 10.0) and the function defaults:

  • absolute_floor = 0.3 V minimum deviation from baseline

  • mad_mult = 12.0 (threshold = baseline + 12 * MAD)

  • hysteresis_frac = 0.8 (pulse ends below 80 % of the threshold)

  • min_width_ms = 3.0, min_distance_ms = 6.0

  • baseline_q = 0.7 (baseline from the lower 70 % of samples)

Interpretation:

  • Stable amplitude and width across pulses: reliable triggering.

  • Large width variance: noisy line or timing drift; inspect the stim plots saved by stage 8.

  • Count mismatch on one channel: an event was dropped or doubled on that line, or the events table is wrong.

  • Order mismatch with equal counts: the table and the acquisition disagree about presentation order.

Troubleshooting

  • Run skipped, “no paired events”: both .csv/.tsv and .json must exist at the same scope; check the fallback hierarchy above.

  • Run skipped, wrong TriggerMode: the JSON declares another mode. Use the multi-channel script, or stage 8 for STI 014 datasets.

  • “Events table is missing the required channel column”: the table needs a channel column with KIT channel numbers.

  • Too many or too few pulses: the pulses may not start within the first 10 s used as baseline, or their amplitude may be below 0.3 V; adjust the call in the script for that dataset and document it in the config copy.

Multi-channel (binary-coded) variant

1-kit_count_sanity_check_multi_channel.py is the same check for datasets acquired in binary-coded trigger mode: one event is encoded as a combination of the eight trigger lines rising together, and the code is the sum of the bit weights of the lines that are high (KIT channel 224 = bit 0, weight 1; 225 = bit 1, weight 2; … 231 = bit 7, weight 128). Where the single-channel script counts pulses per line, this one groups the rising edges of all lines into events and compares the number of events with the number of rows of the events table.

Inputs and selection are the same as above. The run is skipped unless the events JSON declares a multi-channel TriggerMode; the accepted spellings are multi_channel, multi_channel_trigger_mode, multiple_channels, binary_coded, binary_coded_trigger_mode and their variants without underscores (case and punctuation are ignored). The events table needs no channel column: only its number of non-empty rows is used.

Configuration:

events:
  multi_channel:
    max_onset_skew_ms: 2.0   # lines rising within this delay belong to the same event

What the script does, per recording:

  1. Detects the high intervals of every trigger line (MISC 001 to MISC 008) with the same robust threshold as the single-channel script (baseline from the first 10 s, median + MAD, hysteresis, minimum width 3 ms). A missing line is kept as an empty slot so that the bit weights stay tied to the KIT channel numbers.

  2. Builds the code at every sample from the lines that are high and splits the occupied stretches into events: every rising edge that comes more than max_onset_skew_ms after the first rising edge of the current event starts a new event, even if other lines are still high; rising edges within the delay are the same event (the lines of one code never rise on exactly the same sample). A falling edge never starts an event. A gap of up to 2 ms where every line is low is tolerated inside an event.

  3. Records, per event, the onset sample and time, the width, the code, the number of active lines, whether an earlier line was still high at the onset (held_channels_at_onset), whether the grouped lines were ever high together (grouped_bits_never_coactive) and whether the code changed without a new rising edge (late_falling_code_transitions).

  4. Compares the number of events with the number of table rows and writes the log and the summary row.

PASS requires all of: at least one table row and one detected event, equal counts, all eight trigger lines present, no non-finite samples, no event starting while another line was held high, no grouped code whose lines were never simultaneously high, and no late code transition. Any of these produces a warning in the log and FAIL. A recording in which every event has a single active line is flagged as a possible single-channel dataset labelled as multi-channel.

The summary sanity_check_overview.csv of this variant has the columns subject, file, trigger_mode, expected_events, detected_events, max_onset_skew_ms, held_onset_events, noncoactive_event_codes, late_falling_code_transitions, count_match, pass, warnings, comments, log_file and mne_version; the per-run log lists the threshold per line and every detected event with its code.

Interpretation:

  • Fewer events than rows with held_onset_events above zero: two stimuli were presented while the lines of the first were still high; the second was absorbed. Check the stimulus script’s pulse duration.

  • More events than rows with late_falling_code_transitions: the lines of one code fell at different times and the intermediate codes were counted; raise max_onset_skew_ms slightly only if the lines also rise with that skew, otherwise fix the trigger wiring or the script.

  • grouped_bits_never_coactive: lines grouped into one event never overlapped in time, so the code is a guess; the onset delay between the lines is longer than their pulses.

Stage 8’s 8-kit_compute_events_multi_channel.py uses the same grouping rule and the same max_onset_skew_ms, so a PASS here means stage 8 will produce one event per table row.

Summary

Run this stage immediately after BIDS conversion of a single-channel dataset. It confirms that the analog trigger stream and the event annotations agree, and its summary table becomes the first section of the pipeline report.