KIT general pipeline: environment setup and running

Author: Hadi Zaatiti <hadi.zaatiti@nyu.edu>

This page is the entry point for the NeuroWaves KIT general pipeline, the YAML-driven set of numbered scripts under pipeline/mne_pipelines/kit_general_pipelines/. It explains how to create the Python environment, how to run each stage, and where the outputs land. The following pages describe individual stages in more detail.

Preparing the BIDS dataset and its YAML config is described in Starting a project with the general MNE-KIT pipeline. The KidLang study is the worked example used throughout (pipeline_config_files/kidlang_config.yml).

Important

Everything in this section is derived from the scripts currently in the repository. If a command here disagrees with --help of a script, trust the script and open an issue.

What the pipeline does

Each numbered script is one stage. Stages 1 to 6 operate on the MEG recordings, stage 7 reconstructs the anatomy with FreeSurfer and builds the scalp surface, stage 7-a puts every reconstruction in front of a reviewer, and stage 8 performs the manual MEG-MRI coregistration.

#	Script	Reads	Writes (under `<bids_root>/derivatives/`)
1	`1-kit_count_sanity_check_single_channel.py` or `1-kit_count_sanity_check_multi_channel.py`	raw `.con` + paired `_events.csv\|.tsv` and `_events.json`	`1-kit_count_sanity_check_<mode>/sanity_check_overview.csv` and per-run logs
2	`2-kit_con_to_fif.py` (automatic) or `2-kit_con_to_fif_manual.py` (GUI)	`.con`, `.mrk`, laser head shape and digitizer points `.txt`	`2-kit_con_to_fif/` or `2-kit_con_to_fif_manual/` with `sub-/_desc-rawkit_raw.fif` and `kit2fiff_summary.csv`
2-a	`2-a-kit_fif_plot_headshape_sensor_alignment.py` (QC of stage 2)	stage-2 `.fif`	`2-a-kit_fif_plot_headshape_sensor_alignment/sub-/.png`, `alignment_summary.csv`
3-a	`3-a-kit_detect_bad_channels.py`	stage-2 `.fif`	`3-a-kit_detect_bad_channels/sub-/_desc-badchannels.json`, `_desc-badchannels_metrics.tsv`, `figures/.png`, `bad_channels_summary.csv`
3-b	`3-b-kit_ica.py` (interactive: COGNESTIC ICA, proposals reviewed in MNE’s source browser)	stage-2 `.fif` + stage 3-a record	`3-b-kit_ica/sub-/_desc-ica_ica.fif` (solution incl. exclusions), `_desc-ica.json`, `_desc-ica_scores.tsv`, `figures/*.png`, `ica_summary.csv`
4	`4-kit_apply_filters.py`	stage-2 `.fif` + stage 3-a and 3-b records	`4-kit_apply_filters/sub-/_desc-filtered_raw.fif` with `info['bads']` set and the reviewed ICA applied (+ `.json` provenance), `filter_summary.csv`
5	`5-kit_compute_events_single_channel.py`, `5-kit_compute_events_multi_channel.py` or `5-a-kit_compute_events_STI014.py`	raw `.con` (single/multi channel) or stage-2 `.fif` (STI 014)	`5-.../sub-/_desc-<desc>_events.eve` + `*_detail.tsv`, `auto_events_index.csv` or `events_summary.csv`
6	`6-kit_evoked_response_single_channel.py`	stage-4 `.fif` + stage-5 `.eve`	`6-kit_evoked_response_single_channel/sub-/_epo.fif`, `*_ave.fif`, `epochs_evoked_summary.csv`
7	`7-freesurfer_reconstruction.py`	`sub-/anat/_T1w.nii[.gz]` (Linux/WSL/HPC)	`7-freesurfer_reconstruction/subjects/sub-*` (recon-all and the `surf/lh.seghead` scalp from `mkheadsurf`), `freesurfer_summary.csv`, `runs/<run-id>/`
7-a	`7-a-freesurfer_qc.py`	stage-7 reconstructions	MNE scalp surfaces `bem/sub--head-.fif`; `7-a-freesurfer_qc/sub-/sub-_desc-reconqc.json` (reviewer’s verdict), figures, `freesurfer_qc_summary.csv`
8	`8-kit_mri_coreg.py`	stage-2 `.fif` + 7-a-approved reconstruction or `fsaverage`	manual GUI; `8-kit_mri_coreg/sub-/sub--trans.fif` with provenance and QA, `coreg_summary.csv`

Run stages 1 to 6 in order against the same YAML. For the anatomy run 7, then 7-a, then 8: coregistration needs the reconstruction and its scalp surface, and stage 8 uses an individual MRI only after a pass or pass_with_note verdict in 7-a. Stage 8 falls back to the fsaverage template for subjects without a T1w image (or with a use_fsaverage verdict). After every successful stage the pipeline re-renders an incremental Quarto report under <bids_root>/derivatives/pipeline_report/_output/report.html.

Prerequisites

Git and access to the private repository (neurowaves-lab-documentation). Email nyuad.meg@nyu.edu to request access.
Conda. If you do not have one, install Miniforge (conda-forge first, all platforms). Check with conda --version.
A desktop session with working OpenGL for the manual GUIs (stage 2 manual mode, stages 7-a and 8) and the 3-D plots of stage 2-a.
Disk space: several GB for the environment plus space for the datasets, their derivatives/ and FreeSurfer outputs.

Create the `neurowaves` environment

The dependency recipe is pipeline/mne_pipelines/kit_general_pipelines/environment.yml. It is the only environment file you need for the pipeline. (The environment.yml at the repository root is the Windows export used to build this documentation site and does not contain the MEG stack.)

From the repository root:

conda env create -f pipeline/mne_pipelines/kit_general_pipelines/environment.yml
conda activate neurowaves

The recipe installs, from conda-forge only:

Layer	Packages
Interpreter	Python 3.11 or 3.12 (KIT GUI 2.x needs 3.11+)
MEG / BIDS	`mne-base` (imports as `mne`), `mne-bids`, `mne-kit-gui` 2.x, `nibabel`, `scikit-learn` (optional speed-up for the coregistration GUI)
Numerics	`numpy`, `scipy`, `pandas`, `joblib`, OpenBLAS (avoids an MKL crash on Windows while reading KIT sensor positions)
GUIs and plots	`pyside6`, `qtpy`, `mne-qt-browser`, `matplotlib`, `pyvista`, `pyvistaqt`, `vtk`
Config and tables	`pyyaml`, `tabulate`
Reports	Quarto CLI (bundles Pandoc), `ipython`, `ipykernel`, `jupyter_client`, `nbclient`, `nbformat`

The environment also sets QT_API=pyside6 on activation so MNE and the KIT GUI pick the installed Qt binding.

To update an existing environment after the recipe changed:

conda env update -f pipeline/mne_pipelines/kit_general_pipelines/environment.yml --prune

--prune removes packages that are no longer in the recipe, so keep this environment dedicated to the pipeline. Before updating an environment that was used for a published analysis, record it with conda env export -n neurowaves > neurowaves-resolved.yml.

Verify the installation

With neurowaves activated, from the repository root:

# (1) Python dependencies import
python -c "import mne, mne_bids, mne_kit_gui, nibabel, pandas, pyvista, pyvistaqt, qtpy, yaml, joblib, tabulate, IPython, ipykernel, nbclient, nbformat; print('Python deps OK'); mne.sys_info()"

# (2) Quarto is on PATH (1.4 or newer)
quarto --version

# (3) Report toolchain self-check (kernel, LaTeX)
quarto check

Optional: PDF reports

HTML reports work out of the box. PDF rendering needs a LaTeX distribution. Let Quarto install a small one, once per machine:

quarto install tinytex

Optional: FreeSurfer for stage 7

FreeSurfer is not installed by Conda. Install it on the Linux host that will run recon-all (Linux workstation, HPC node, or WSL Ubuntu-22.04 on Windows), following FreeSurfer on WSL, and provide its license (FS_LICENSE or freesurfer.license_file in the YAML).

On Windows, 7-freesurfer_reconstruction.py is launched with the Windows neurowaves interpreter and forwards the work into WSL, where python3 and PyYAML must exist:

# inside the WSL distribution
sudo apt update && sudo apt install python3 python3-yaml
python3 -c "import sys, yaml; print(sys.version)"

Set FREESURFER_WSL_DISTRO before starting VS Code if FreeSurfer lives in a distribution other than Ubuntu-22.04. The distribution’s files are reachable from the Windows explorer at \\wsl$\<distribution>.

Stage 7 runs recon-all for every selected subject, plans jobs and threads from the available CPUs and RAM, resumes incomplete subjects, creates the scalp surface with mkheadsurf and records an audit CSV. freeview (part of FreeSurfer) can be opened from the stage 7-a review window through WSLg.

IDE setup

In VS Code select the neurowaves interpreter with Python: Select Interpreter. The repository ships .vscode/launch.json entries for every stage (including the KidLang-specific ones) and .vscode/tasks.json tasks that refresh the Quarto report. All launch entries set PYTHONPATH=${workspaceFolder} so the pipeline package is importable.

Run the stages

Every stage takes the dataset’s YAML config with -c / --config (see Starting a project with the general MNE-KIT pipeline for creating it).

Run the scripts as modules from the repository root so that the local pipeline package is importable (no pip install of the repository is required):

conda activate neurowaves
MODULE=pipeline.mne_pipelines.kit_general_pipelines
CONFIG=pipeline/mne_pipelines/kit_general_pipelines/pipeline_config_files/kidlang_config.yml

python -m $MODULE.1-kit_count_sanity_check_single_channel -c $CONFIG
python -m $MODULE.2-kit_con_to_fif -c $CONFIG
python -m $MODULE.2-a-kit_fif_plot_headshape_sensor_alignment -c $CONFIG
python -m $MODULE.3-a-kit_detect_bad_channels -c $CONFIG
python -m $MODULE.3-b-kit_ica -c $CONFIG
python -m $MODULE.4-kit_apply_filters -c $CONFIG
python -m $MODULE.5-a-kit_compute_events_STI014 -c $CONFIG
python -m $MODULE.6-kit_evoked_response_single_channel -c $CONFIG

# Anatomy (Linux/WSL/HPC), reconstruction QC, manual MEG-MRI coregistration
python -m $MODULE.7-freesurfer_reconstruction -c $CONFIG
python -m $MODULE.7-a-freesurfer_qc -c $CONFIG
python -m $MODULE.8-kit_mri_coreg -c $CONFIG

The variable assignments are Bash syntax; in PowerShell spell out the module and config paths. Running a script by file path also works when PYTHONPATH contains the repository root (this is what the VS Code launch entries do).

Common options:

--overwrite (stages 2, 4, 5, 6): rebuild outputs that already exist. Without it, existing outputs are skipped and reported as such.
--desc <label> (stage 5 variants): override events.desc.
--show (stage 2-a): open the interactive 3-D alignment plots instead of only saving PNGs.
--no-gui, --review-all, --reset-review, --overwrite, --no-plots, --save-cleaned (stage 3-b): fit and propose without reviewing, redo or discard reviews, refit, skip figures, also write the cleaned recording. Details on the ICA page.
--no-gui (stage 3-a): compute the automatic candidates only and label nothing (review_status: pending); the default run opens the inspection and confirmation browsers for every recording not yet reviewed. --review-all redoes the review of reviewed recordings, --reset-review discards the saved reviews, --no-plots skips the audit figures. Rerunning stage 3-a with unchanged input and detection settings reuses the saved metrics. Details on the bad channel page.
--dry-run, --force, --jobs, --threads (stage 7).
--no-gui, --review-all, --reset-review, --overwrite, --no-plots (stage 7-a): as in stage 3-a; --no-gui computes metrics, flags and figures and leaves the verdicts pending.
--redo, --qa-only, --no-plots (stage 8): reopen the GUI for verified transforms, or only recompute the distance QA.

Each Python stage writes a timestamped copy of the YAML it used into <stage>/configs/config_<timestamp>.yml (the report reads the newest one to show the parameters of the run; copies that older runs left in the stage root are moved into configs/ on the next run).

The KidLang configuration

kidlang_config.yml differs from the template in a few deliberate ways:

kit2fiff.slope: "+" (triggers are 0 to 5 V peaks) and kit2fiff.coreg.mode: manual: stage 2 opens the KIT-to-FIFF GUI once per recording so the head-shape alignment can be checked by eye.
Events come from the STI 014 channel, so the stage-5 variant is 5-a-kit_compute_events_STI014.py with events.desc: sti014, events.source: sti014 and events.find_events kwargs (stim_channel: "STI 014", shortest_event: 1, min_duration: 0.002).
freesurfer.auto_cw256: true retries subjects whose field of view exceeds 256 with -cw256 (stage 7-a checks the crop); missing_anat: fsaverage lets stage 8 use the template for subjects without a T1w.
reports.formats: ["html"] so no LaTeX is needed.
There is no filters or epoching block, so stages 4 and 6 use their built-in defaults (notch at 50 Hz harmonics, band-pass 1 to 40 Hz, epochs from -0.2 s to 0.8 s with a baseline of -0.2 to 0 s). Add those blocks to the config to change them.

Where the outputs land

The scripts write only under <bids_root>/derivatives/; the source tree is never modified.

<bids_root>/derivatives/
  1-kit_count_sanity_check_single_channel/
    sanity_check_overview.csv
    sub-*/sub-*_..._desc-sanitycheck_log.txt
  2-kit_con_to_fif/                      # or 2-kit_con_to_fif_manual/
    kit2fiff_summary.csv
    sub-*/[ses-*/]
      sub-*_..._desc-rawkit_raw.fif
      sub-*_..._desc-coreg.png           # automatic mode
      *_edited.txt                       # digitizer points without the last 3 columns
      kit2fiff_log.txt
  2-a-kit_fif_plot_headshape_sensor_alignment/
    alignment_summary.csv
    sub-*/...png (+ .json)
  3-a-kit_detect_bad_channels/
    bad_channels_summary.csv
    sub-*/sub-*_..._desc-badchannels.json          # final bad-channel list + settings
    sub-*/sub-*_..._desc-badchannels_metrics.tsv   # one row per channel
    sub-*/figures/*.png                            # PSD, metrics, sensor map, traces
  3-b-kit_ica/
    ica_summary.csv
    sub-*/sub-*_..._desc-ica_ica.fif               # ICA solution incl. reviewed exclusions
    sub-*/sub-*_..._desc-ica.json                  # proposals, review, settings
    sub-*/sub-*_..._desc-ica_scores.tsv            # one row per component
    sub-*/figures/*.png                            # components, scores, properties, overlays
  4-kit_apply_filters/
    filter_summary.csv                             # bad_channels, ica_excluded columns
    sub-*/sub-*_..._desc-filtered_raw.fif (+ .json provenance; info['bads'] from 3-a, ICA from 3-b applied)
  5-a-kit_compute_events_STI014/         # or 5-kit_compute_events_<mode>/
    events_summary.csv                   # auto_events_index.csv for the analog variants
    sub-*/sub-*_..._desc-sti014_events.eve
    sub-*/sub-*_..._desc-sti014_events_detail.tsv
  6-kit_evoked_response_single_channel/
    epochs_evoked_summary.csv
    sub-*/sub-*_..._desc-epo_raw_ev<code>_epo.fif     # one epochs file per event code
    sub-*/sub-*_..._desc-evk_raw_ave.fif               # all evoked responses of the recording
  7-freesurfer_reconstruction/
    subjects/sub-*/                      # SUBJECTS_DIR (surf/lh.seghead, bem/*-head-*.fif)
    freesurfer_summary.csv
    runs/<run-id>/{freesurfer_summary.csv, run_metadata.txt, config.yml, logs/}
  7-a-freesurfer_qc/
    freesurfer_qc_summary.csv
    freesurfer_qc_metrics.tsv
    sub-*/sub-*_desc-reconqc.json        # metrics, flags, verdict
    sub-*/figures/*.png
  8-kit_mri_coreg/
    coreg_summary.csv
    sub-*/sub-*-trans.fif                # saved from the GUI
    sub-*/sub-*-trans.json               # provenance and QA distances
    subjects/fsaverage/                  # template copy when needed
  pipeline_report/
    _output/report.html [report.pdf]       # + source/report.qmd, _quarto.yml
    logs/render_log_<timestamp>.txt        # newest 30 kept
    state/render.lock, report_refresh.json
    staging/                               # transient, empty between renders

Report

The report (report_generation/report.qmd) is rendered with Quarto after each stage when reports.enabled: true. Quarto is found on PATH, through reports.quarto_path, or via conda run -n <reports.quarto_env>. To re-render without re-running any processing (for example after pulling a new report template, or on a workstation after stage 7 ran on a compute node):

python -m pipeline.mne_pipelines.kit_general_pipelines.report_generation.refresh --config $CONFIG
# keep watching the inputs and refresh on change
python -m pipeline.mne_pipelines.kit_general_pipelines.report_generation.refresh --watch

Only one render runs per report at a time. A stage that finds another render in progress (for example the VS Code task Watch Quarto reports, which starts with the workspace) waits at most reports.lock_wait_s seconds (default 30), then continues without waiting: the running render re-checks the inputs when it finishes and renders again if the stage added outputs. A render that runs longer than reports.render_timeout_s seconds (default 600) is killed, the previous report is kept, and the next stage or the watcher renders again. A stage whose outputs were already rendered by the watcher reports report already up to date and skips.

Troubleshooting

Symptom	Cause	Fix
`No module named 'pipeline'`	Script launched without the repository on the import path	Run with `python -m pipeline.mne_pipelines.kit_general_pipelines.<script>` from the repository root, or set `PYTHONPATH` to it
`MEG_DATA is not set and project.root_override not provided`	Dataset parent folder unknown	Export `MEG_DATA` or set `project.root_override`
`ScannerError: while scanning a double-quoted scalar`	Backslash in a double-quoted Windows path	Use forward slashes or single quotes
Quarto cannot be found	Not on PATH and not in the configured Conda env	`conda run -n neurowaves quarto --version`; set `reports.quarto_env` or `reports.quarto_path`
`tlmgr: command not found` while rendering PDF	No LaTeX	`quarto install tinytex` or set `reports.formats: ["html"]`
GUI cannot initialise Qt / OpenGL	No desktop session or wrong interpreter	Select the `neurowaves` interpreter and run on a machine with a display
Stage 7 cannot import `yaml` in WSL	PyYAML installed on Windows only	`sudo apt install python3-yaml` inside the WSL distribution
Stage 4 reports `existing_output_unverified`	Input, settings or output changed since the FIF was filtered	Rerun stage 4 with `--overwrite`
Report shows old data	Not refreshed after the inputs changed	Run the `report_generation.refresh` command above
`another render is in progress ... Not waiting.`	The watcher task or another stage holds the render lock	Nothing to do; that render includes this stage’s outputs
`quarto render ... exceeded N s and was killed`	Quarto or its Jupyter kernel hung, often on a slow or syncing drive	Check the latest `logs/render_log_*.txt` and run the refresh command

If a render fails, the Quarto stderr is in <bids_root>/derivatives/pipeline_report/logs/render_log_<timestamp>.txt. The stages continue after a failed render unless reports.fail_on_error: true.