Stage 2: KIT .con to .fif

Author: Hadi Zaatiti <hadi.zaatiti@nyu.edu>

What & why

Stage 2 of the KIT general pipeline converts NYUAD-KIT .con recordings into MNE .fif files while preserving every BIDS entity in the output filename, pairing the head-position marker (.mrk) files deterministically, attaching the laser head shape and digitizer points, and recording the provenance of each conversion under BIDS derivatives/.

2-kit_con_to_fif.py opens the KIT-to-FIFF GUI (mne-kit-gui 2.x) once per recording, preloaded with the resolved files, so that a person inspects the marker alignment and presses Save FIFF…. The conversion is always done this way: an operator checks every coregistration (an earlier automatic conversion was removed on 2026-10-20). Recordings are processed one after the other.

Run this stage after BIDS-structuring the dataset (see Data Naming and Storing) and before every other MEG stage: stages 3 to 7 and the STI 014 variant of stage 8 read the FIFs it produces, stages 12 and 13 their digitisation, and every later stage inherits from those.

What you need to provide

A BIDS-organised KIT dataset under <MEG_DATA>/<project.name>/ containing, per subject (and optionally per session):

sub-<id>/[ses-<id>/]meg/
  sub-<id>[_ses-<id>]_task-<task>[_acq-...][_run-<n>][_split-<n>][_proc-<label>]_meg.con
  sub-<id>[_ses-<id>]_task-rest_acq-<k>_space-ALS_markers.mrk         (k = chronological number)
  sub-<id>[_ses-<id>]_acq-head_headshape.txt                          (laser head surface)
  sub-<id>[_ses-<id>]_acq-points_headshape.txt                        (laser stylus points)
  sub-<id>[_ses-<id>]_scans.tsv                                       (optional: declared marker pairs)

Rules applied by the script:

  • MRK pairing (utilities.pair_mrks_per_con), per session, on every CON on disk (so that excluding a recording cannot shift the pairs of the others). Marker files are named after their chronological order in the session; which recordings a measurement bounds is decided in this order:

    1. Declared in scans.tsv. When the session’s BIDS acquisition table sub-<id>[_ses-<id>]_scans.tsv has the two additional columns marker_before and marker_after (paths relative to the subject or session folder, meg/..., or bare file names; n/a for none), every recording of the session takes the files it lists, one or two. This is how a measurement shared by several recordings is expressed: for marker, task A, task B, marker both rows list the same two files. The table is the declaration of record: a recording missing from it, or a listed file that does not exist, is an error row (no conversion), and two rows of one recording (raw and proc variant) that disagree are an error too. Describe the columns in scans.json for the BIDS validator (the draft writer below produces one).

    2. The chain (no table, or a table without the marker columns): CONs are ordered by (run, split, acq), MRKs by acq. With n CONs and n+1 MRKs, CON i uses MRK i (before) and MRK i+1 (after); the after of one recording is the before of the next. Any other count is paired best effort and flagged with a warning in the summary (no_mrk, single_mrk_for_N_cons, non_ideal_mrk_count, non_ideal_mrk_count_trailing_con: that CON uses only the last MRK); a CON with no MRK is skipped. The chain needs session-wide run numbers in recording order and a marker measurement between every two consecutive files, splits included (each split is a chain slot).

    The mrk_pairing column of the summary names the source and the counts used for every recording (scans.tsv (2 MRK), chain (3 CON, 4 MRK)). --write-scans-draft writes a draft scans.tsv (and scans.json) per subject and session under the stage’s derivatives folder, listing every recording with its header time (acq_time) and the pair the rule in force assigns to it; correct the rows that need it and copy both files next to the data. The raw dataset is never written by the stage.

  • Head shape / points: resolved per subject and session, never filtered by task or run. A session-level file wins; otherwise a subject-level file is used. Another session’s digitisation is never borrowed. If several candidates match, the run fails with an “Ambiguous digitizer files” error.

  • Edited points: the stylus points file is rewritten once per scope as *_edited.txt (last three columns dropped) and reused for every CON of that subject/session.

Configuration file

Every stage reads one YAML config given with -c / --config. Without the option the template pipeline_config_files/config_template.yml is used. The keys used by stage 2:

project:
  name: "kidlang"
  root_env: "MEG_DATA"        # or root_override: "D:/MEG_Data"

subjects:
  include: []                 # [] = all subjects
  exclude: []

bids_selection:               # optional include filters (empty = all)
  sessions: []
  tasks: []
  runs: []
  splits: []
  processings: []             # [null] = no proc entity; [CALMnoisereduction] = CALM only
  exclude: {}                 # e.g. {processings: [CALMnoisereduction]}

kit2fiff:
  stim: ">"                   # '>' little-endian, '<' big-endian, or explicit channel list
  slope: "+"                  # '+' peak (0 to 5 V), '-' trough (5 to 0 V)
  stimthresh: 1.0
  stim_code: "binary"         # or 'channel'

The stim / slope / stimthresh / stim_code values are set on the GUI’s model (its stim panel) before the window opens, so the STI 014 channel of the saved FIF is built with them; the values the model held at the save are written to the marker-selection sidecar and the summary (stim_settings). Set slope according to the polarity of the trigger pulses sent by the experiment-control software ("+" for 0 to 5 V peaks, "-" for 5 to 0 V troughs; the KidLang and epilepsy configs use "+"). With the wrong slope STI 014 holds the inverted code and mne.find_events places events at the pulse offsets with the wrong value (events.source: sti014 projects). Check the stim plots of stage 8 or the GUI’s “Find Events” if unsure.

How to run

From the repository root, with the neurowaves environment activated (see the environment setup page):

python -m pipeline.mne_pipelines.kit_general_pipelines.2-kit_con_to_fif \
    -c pipeline/mne_pipelines/kit_general_pipelines/pipeline_config_files/kidlang_config.yml

Options:

  • --overwrite: reopen the GUI for recordings whose FIF already exists. Existing outputs are otherwise skipped.

  • --write-scans-draft: write a draft scans.tsv (marker pairs) per subject and session under the stage’s derivatives, to review and copy next to the data.

The GUI in practice

  1. One GUI window opens per recording, preloaded with the CON, the paired MRKs, the head shape and the edited points. The output path is pre-filled.

  2. Rotate the head in the 3-D view, check that the markers sit where expected, adjust the marker selection if needed, then press Save FIFF… and accept the pre-filled path. You can change the selection and save to the same path again before closing; the last completed save is the recorded result.

  3. Wait for the GUI to report the completed save, then close the window. Closing without saving creates no file.

All file dialogs of the GUI start where the pipeline expects them: each Browse button opens in the folder of the file currently loaded in that field (the subject’s BIDS meg folder when the field is empty), and the marker Save as buttons (source markers and New Marker) start in the recording’s folder under derivatives/2-kit_con_to_fif/sub-<id>/, next to the FIF, with a .txt name derived from the marker or the recording. Only Save FIFF... is recorded in the pipeline summary; marker files saved this way are side outputs for your own inspection.

The VS Code launch entries (Kidlang) CON TO FIF Manual Alignment and KIT to FIF Manual Alignment (choose config) run this script directly. The GUI walkthrough with screenshots is on the “Kit2Fiff Tutorial” page.

After every GUI window the manual script rebuilds a cumulative coregistration overview from the saved FIFs: coreg_rms_overview.csv (one row per subject), coreg_rms_recordings.csv (one row per recording), coreg_rms_group_stats.csv (cohort mean, SD, median and range of both quantities at subject and recording level) plus a PNG/SVG plot of the RMS residual of the HPI marker-coil fit (the coils measured by the MEG against their digitised positions; the laser head shape does not enter it) against the pre/post HPI coil displacement, with the subject-level mean and ± 1 SD band drawn in. A recording bounded by one marker measurement has no measurable displacement and is left out of the displacement averages (drawn hollow). The console prints the mean ± SD after every refresh and the report shows the plot and the statistics table.

What the script does

  1. Load the YAML and resolve the BIDS root (<parent>/<project.name>).

  2. Discover subjects (include empty = all; exclude removed) and apply bids_selection.

  3. Pair the MRKs with the CONs: the session’s scans.tsv when it declares marker_before/marker_after, else the chronological chain.

  4. Resolve head shape and points per subject/session; write the edited points once per scope.

  5. Read the KIT data with the configured stim settings, open the GUI, and record the FIF the operator saves.

  6. Append one row per CON to kit2fiff_summary.csv and one block to the per-subject kit2fiff_log.txt; save a timestamped copy of the config.

  7. Request a refresh of the Quarto report (reports.enabled).

File naming

All present entities are serialised in a stable order, then desc-rawkit and the _raw.fif suffix:

sub-<id>[_ses-<id>][_task-<task>][_acq-<label>][_run-<n>][_split-<n>]
[_rec-<label>][_space-<label>][_proc-<label>]_desc-rawkit_raw.fif

Examples:

  • sub-test1_task-400events_desc-rawkit_raw.fif

  • sub-test1_task-400events_proc-CALMnoisereduction_desc-rawkit_raw.fif

  • sub-01_task-rest_run-02_split-01_desc-rawkit_raw.fif

Processed (CALM) and unprocessed recordings, and split parts, therefore never overwrite each other.

Outputs

<bids_root>/derivatives/2-kit_con_to_fif/
  kit2fiff_summary.csv
  configs/config_<timestamp>.yml
  coreg_rms_overview.csv, coreg_rms_recordings.csv,
  coreg_rms_group_stats.csv, coreg_rms_overview.png/.svg
  sub-<id>/
    kit2fiff_log.txt
    [ses-<id>/]
      sub-<id>..._desc-rawkit_raw.fif
      sub-<id>..._desc-rawkit_raw_marker_selection.json   # GUI marker selection of that save
      sub-<id>[_ses-<id>]_acq-points_headshape_edited.txt

kit2fiff_summary.csv has one row per CON with: timestamp, subject, session, run, split, con_path, mrk_paths (semicolon separated), hsp_path, elp_points_edited, fif_out, status (success | skipped | error), error, warning, the coregistration columns (coreg_decision, coreg_dropped, coreg_all_rms_mm, coreg_chosen_rms_mm, coreg_improvement_mm, coreg_improvement_rel, coreg_plot, coreg_plot_all, coreg_info) and mne_version.

Later stages read this CSV to know which FIFs are the current, verified outputs of stage 2; files that are not listed in it are ignored.

Edge cases & safeguards

  • Missing MRKs: the CON is skipped with status=skipped.

  • Short MRK list: best-effort pairing with a warning in the summary.

  • scans.tsv problems (recording not listed, listed marker file missing, both marker cells empty, raw and proc rows disagreeing): status=error with the reason in error; nothing is converted for that recording until the table is fixed.

  • Missing head shape or points: the CON is skipped with a warning.

  • Ambiguous digitizer files for one scope: error, nothing is written for that subject until the dataset is fixed.

  • Existing FIF: skipped unless --overwrite.

Troubleshooting

  • No FIFs / empty summary: check that .con files are discoverable under the selected entities, that MRKs exist, and that MEG_DATA points to the parent of the dataset folder.

  • Wrong head shape used: confirm the session-level files exist if you expect per-session digitisation; otherwise the subject-level file is used.

  • GUI does not open: run on a machine with a display and select the neurowaves interpreter, which contains mne-kit-gui 2.x and PySide6.

Next stages

  • Stage 3: Head shape and sensor alignment plots saves a picture of the accepted head-shape and sensor alignment of every FIF (the QC of this stage).

  • Stage 4: Head movement between marker measurements turns the marker pairing recorded in kit2fiff_summary.csv into a head movement timeline per session.

  • Stages 5 (bad channels), 6 (ICA) and 7 (filters) read the FIFs listed in kit2fiff_summary.csv; stage 12 takes the digitisation of the first FIF of each subject for the MEG-MRI coregistration.

Testing

tests/test_pipelines_kit_mne.py runs the stage-1 sanity check on the script-testing-dataset fetched from Box. Use the same dataset with config_template.yml to try stage 2 on a small example before converting a full study.