Stage 1: trigger count sanity check (single channel)

Author: Hadi Zaatiti <hadi.zaatiti@nyu.edu>

What & why

1-kit_count_sanity_check_single_channel.py validates that the trigger pulses recorded on the analog MISC lines of a NYUAD-KIT MEG dataset match the event annotations in the BIDS *_events.csv / *_events.tsv table: same number of pulses per trigger channel, same chronological order. It detects lost, duplicated or misaligned triggers before any further processing.

The check is intended for datasets acquired in single-channel trigger mode (each event type on its own trigger line, KIT channels 224 to 231). A companion script, 1-kit_count_sanity_check_multi_channel.py, performs the equivalent check for binary-coded (multi-channel) trigger mode, where one event is encoded by several lines rising together.

Datasets whose triggers are read from the digital STI 014 channel (such as KidLang) do not go through this check; their events are extracted in stage 5 with 5-a-kit_compute_events_STI014.py.

What you need to provide

  • MEG recordings

    sub-<id>/[ses-<id>/]meg/sub-<id>[_ses-<id>]_task-<task>[_run-<n>]_meg.con

  • A paired events table and JSON sidecar at the same scope

    ..._events.csv   (or .tsv)
..._events.json

    The table must contain a channel column holding the KIT trigger channel number (224 to 231) of each event, in presentation order. The JSON must declare the trigger mode:

    { "TriggerMode": "single_channel" }

    Runs whose JSON does not declare single_channel (case, spaces and hyphens are normalised) are reported in the summary as skipped.

Fallback hierarchy for events files

Both files must exist at the same level. The search goes from the most specific to the least specific scope and stops at the first level that provides both:

  1. Exact match: sub-<id>[_ses-<id>]_task-<task>_run-<n>_events.*

  2. Subject + task: sub-<id>[_ses-<id>]_task-<task>_events.*

  3. Subject only: sub-<id>[_ses-<id>]_events.*

  4. Dataset root: events.csv + events.json next to dataset_description.json (applies to every subject)

A level that provides only one of the two files is ignored. The same policy is used by the stage-5 event-extraction scripts.

Configuration file

The stage reads the shared pipeline YAML (-c / --config; default pipeline_config_files/config_template.yml). Keys used:

project:
  name: "script-testing-dataset"
  root_env: "MEG_DATA"        # or root_override

subjects:
  include: []
  exclude: []

bids_selection:
  sessions: []
  tasks: []
  runs: []
  processings: []             # [null] = unprocessed .con only
  exclude: {}

events:
  multi_channel:
    max_onset_skew_ms: 2.0    # multi-channel variant only

reports:
  enabled: true

The pulse-detection thresholds are fixed in the script (see Tuning below); there are no YAML keys for them.

How to run

From the repository root, neurowaves environment activated:

python -m pipeline.mne_pipelines.kit_general_pipelines.1-kit_count_sanity_check_single_channel \
    -c pipeline/mne_pipelines/kit_general_pipelines/pipeline_config_files/config_template.yml

# binary-coded datasets
python -m pipeline.mne_pipelines.kit_general_pipelines.1-kit_count_sanity_check_multi_channel \
    -c <your_config.yml>

The VS Code launch entry KIT Sanity Check Single Channel runs the same command on the template config.

What the script does

  1. Loads the config and resolves the BIDS root (<parent>/<project.name>).

  2. Discovers subjects and .con recordings, applying bids_selection.

  3. Resolves the paired events table and JSON; skips the run if the pair is missing or TriggerMode is not single_channel.

  4. Reads the KIT recording (no preload) and the events table; checks that every channel value is one of the KIT trigger channels.

  5. Detects pulses on each of MISC 001 to MISC 008 with a robust threshold: baseline from the lower tail of the first 10 s, median + MAD threshold, hysteresis and debouncing. Spans annotated as bad are ignored without shifting sample indices.

  6. Compares detected pulses with the table: count per KIT channel and the chronological channel sequence.

  7. Computes pulse statistics (amplitude and width mean/variance), writes a per-run log and the summary CSV, saves a timestamped copy of the config and requests a report refresh.

Outputs

<bids_root>/derivatives/1-kit_count_sanity_check_single_channel/
  sanity_check_overview.csv
  configs/config_<timestamp>.yml
  sub-<id>/
    sub-<id>[_ses-<id>]_task-<task>[_run-<n>]_desc-sanitycheck_log.txt

The multi-channel variant writes to derivatives/1-kit_count_sanity_check_multi_channel/ with the same layout.

Per-run log

  • Raw file, events table and JSON paths, TriggerMode

  • Threshold per channel (thr_hi, thr_lo, pulse count)

  • Count comparison per KIT channel (csv_count / detected_count / diff)

  • Sequence check with the first mismatching position

  • Amplitude and width statistics

  • Final PASS/FAIL line (ASCII OK/FAIL on Windows consoles)

Summary CSV

sanity_check_overview.csv columns: subject, file, trigger_mode, csv_events, detected_events, counts_match, row_order_match, reference_channels_valid, pass, log_file, comments, mne_version. Skipped runs keep a row with the reason in comments.

Pass/Fail criteria

A run passes only if every KIT channel count matches the table exactly and the detected chronological channel order equals the table order.

Tuning & interpretation

The detector is detect_pulses_on_channel in kit_general_pipelines/utilities.py. The stage calls it with baseline_s=(0.0, 10.0) and the function defaults:

  • absolute_floor = 0.3 V minimum deviation from baseline

  • mad_mult = 12.0 (threshold = baseline + 12 * MAD)

  • hysteresis_frac = 0.8 (pulse ends below 80 % of the threshold)

  • min_width_ms = 3.0, min_distance_ms = 6.0

  • baseline_q = 0.7 (baseline from the lower 70 % of samples)

Interpretation:

  • Stable amplitude and width across pulses: reliable triggering.

  • Large width variance: noisy line or timing drift; inspect the stim plots saved by stage 5.

  • Count mismatch on one channel: an event was dropped or doubled on that line, or the events table is wrong.

  • Order mismatch with equal counts: the table and the acquisition disagree about presentation order.

Troubleshooting

  • Run skipped, “no paired events”: both .csv/.tsv and .json must exist at the same scope; check the fallback hierarchy above.

  • Run skipped, wrong TriggerMode: the JSON declares another mode. Use the multi-channel script, or stage 5-a for STI 014 datasets.

  • “Events table is missing the required channel column”: the table needs a channel column with KIT channel numbers.

  • Too many or too few pulses: the pulses may not start within the first 10 s used as baseline, or their amplitude may be below 0.3 V; adjust the call in the script for that dataset and document it in the config copy.

Summary

Run this stage immediately after BIDS conversion of a single-channel dataset. It confirms that the analog trigger stream and the event annotations agree, and its summary table becomes the first section of the pipeline report.