Stage 0: CALM noise reduction in MEG160

Author: Hadi Zaatiti <hadi.zaatiti@nyu.edu>

What & why

The KIT system records three reference magnetometers (channels 208, 209 and 210) placed away from the head. They pick up almost only environmental magnetic noise: power lines, lifts, cars, building vibrations. The same noise reaches the head sensors. CALM (Continuously Adjusted Least-squares Method, Adachi et al., 2001) fits, for every head sensor, the weighted sum of the reference channels that best explains its signal over a time range, and subtracts it:

\[\hat{x}_i(t) = x_i(t) - \sum_{k \in \{208, 209, 210\}} w_{ik}\, r_k(t) - c_i\]

The vendor’s MEG160 program (MEG Laboratory) implements it. Until now an operator opened each recording in MEG160, clicked Edit > Noise Reduction, accepted the pre-filled CALM values, pressed Execute and saved the result under a new name. Stage 0 does exactly that for every recording of a dataset, driven by the same YAML config as the other stages, and records what was done.

MEG160 has no batch command for the noise reduction: its command-line batch mode (-b<macro> -o<output>) runs macros, and macros cannot contain the noise reduction; its automation interface has no noise-reduction call either. The stage therefore operates MEG160’s own windows with Windows messages, the way a person would click them (details in pipeline/mne_pipelines/kit_general_pipelines/meg160/README.md).

What you need to provide

  • Windows. MEG160 is a Windows program.

  • MEG160. The executable is bundled in the repository (kit_general_pipelines/meg160/Meg160_20260723). MEG160 keeps its settings in C:\Meg160 whatever folder it is started from; the pre-filled noise-reduction values come from there. On a computer without C:\Meg160, run the stage once with --install-meg160-home to create it from the bundled copy.

  • A BIDS dataset with raw .con files (without a proc entity) and the usual YAML config (project, subjects, bids_selection).

  • MEG160 closed. The stage refuses to start when MEG160 is open (two instances would share C:\Meg160\TempFiles).

Which files are processed

Every .con without a proc entity that subjects and bids_selection select. bids_selection.processings (and its exclude) is ignored here, because a project config usually sets it to [CALMnoisereduction] for the later stages; every other selector applies, including bids_selection.exclude:

subjects:
  include: []          # empty = all subjects
  exclude: [pilot01]
bids_selection:
  tasks: [kidlang]
  exclude: {runs: ["03"]}

sub-emptyroom is processed too, so the empty-room recordings get the same noise reduction as the data; add it to subjects.exclude to leave it out.

Settings

calm:
  enabled: true
  meg160_exe: null                      # null = the bundled offlineMeg160E.exe
  processing_label: "CALMnoisereduction"
  parameters:                           # null = keep MEG160's pre-filled value
    reference_channels: null            # e.g. "208,209,210"
    start_ms: null
    end_ms: null                        # MEG160 pre-fills the recording length
    range: null                         # fixed | moving
    moving_width_ms: null
  timeouts_s: {open: 900, noise_reduction: 7200, save: 1800}
  verify: true
  on_overflow: warn                     # warn | fail (see "16-bit overflow" below)

With every parameter null the stage reproduces the manual routine: on the NYUAD KIT MEG160 pre-fills CALM on the environmental noise of the MEG channels, references 208, 209, 210, and a fixed range over the whole recording. The method is always CALM (environmental noise, MEG channels); the stage corrects the dialog if it was left on TSPCA or sensor noise and records that it did.

Running it

python -m pipeline.mne_pipelines.kit_general_pipelines.0-kit_meg160_calm_noise_reduction \
    -c pipeline/mne_pipelines/kit_general_pipelines/pipeline_config_files/<project>_config.yml

or the VS Code entries (Epilepsy) 0 CALM noise reduction in MEG160 (Windows), … dry run (list files only) and CALM noise reduction in MEG160 (choose config), or the pipeline console (stage 0).

MEG160 windows open and close by themselves, one recording after the other. The keyboard and mouse are not used, so the computer stays usable, but do not work in MEG160 meanwhile. A 14-minute recording takes about 30 s.

Flags:

  • --dry-run: list the recordings and the output names; run nothing.

  • --overwrite: redo recordings that already have a CALM file. The old file is moved aside and deleted only after the new one passed the check; otherwise it is put back.

  • --no-verify: skip the MNE comparison of output and input.

  • --check-existing: also run the check on CALM files that already exist (for example made by hand), without redoing them. A file that is not noise-reduced is reported as existing_output_not_calm.

  • --install-meg160-home: create C:\Meg160 from the bundled copy when it is missing (an existing C:\Meg160 is never touched).

What it writes

The noise-reduced recording goes into the dataset, next to the original (which is never changed), with the BIDS processing entity:

sub-<id>[_ses-<id>]_task-<task>[_acq-<a>][_run-<r>]_proc-CALMnoisereduction[_split-<s>]_meg.con

That is the file the later stages select with bids_selection.processings: [CALMnoisereduction]. Under <bids_root>/derivatives/0-kit_meg160_calm_noise_reduction/:

calm_summary.csv                         one row per recording, merged across runs
configs/config_<timestamp>.yml
sub-*/[ses-*/]<output stem>_calm.json        provenance
sub-*/[ses-*/]<output stem>_calmweights.txt  CALM weights per channel (MEG160's NoiseReduct.txt)
failed/                                  outputs that failed the check (out of the dataset)
replaced/                                old CALM files during an --overwrite

The provenance JSON holds the input and output size and modification time, the values in the dialog when Execute was pressed and the values MEG160 had pre-filled, which settings the stage changed, the MEG160 time estimate, timings, the SHA-256 of the executable and of C:\Meg160\AppInfo\DenoiseConfig.txt, and the check results.

How each output is checked

The output and its input are read in MNE and compared on the first and the last 10 s:

  • same channel names, sampling rate and number of samples;

  • every head sensor changed (at least 90 %, a warning below 100 %);

  • reference and trigger/misc channels identical;

  • the CALM model holds: for every head sensor, input minus output is fitted as a linear combination of the reference channels entered in the dialog plus a constant. What the fit leaves must be the rounding to integers MEG160 does when it saves: at most 0.5 A/D counts RMS on every sensor (calm_residual_rms_counts; real CALM files give 0.29, i.e. 1/sqrt(12)), with the same weights at the start and at the end (calm_weights_drift, a warning above 1 %). calm_fit_r2_min is recorded for information only: it is low on sensors where CALM removes just a few counts;

  • the median MEG standard deviation before and after (rms_before_ft, rms_after_ft, rms_reduction_pct). CALM mostly removes low-frequency and line noise, so the broadband reduction is often only a few percent.

An output that fails, or an incomplete output after a MEG160 error, is moved to failed/: later stages select files by name and must never find it.

16-bit overflow

MEG160 writes 16-bit samples and does not saturate them. When the noise is large (some head sensors already at the A/D limit of about plus or minus 33 pT in the raw file), subtracting it can push a sample beyond the range; MEG160 then stores it shifted by 65536 counts, that is with the opposite sign (+39.6 pT stored as -26.5 pT). The same file results when the noise reduction is done by hand. The check recognises these samples (exactly 65536 counts off the CALM fit), and, with the fitted weights, counts them over the whole recording: overflow_channels and overflow_samples in the summary, the count per channel in the provenance JSON, and a warning. With calm.on_overflow: warn (default) the file is kept, so the result is the one the manual routine gives, and the listed channels should be marked bad in stage 5; fail moves such files to failed/. In the test data, one empty-room recording of 2024-10-08 had 9535 such samples on 52 channels.

Statuses in calm_summary.csv

  • success: made by this run and checked.

  • skipped_existing_output: a CALM file was already there. origin is stage_0 when this stage made it, outside_stage_0 when it was made by hand (no provenance, settings unknown; --overwrite redoes it).

  • existing_output_unverified: the raw file changed after its CALM file was made; rerun with --overwrite.

  • existing_output_not_calm (with --check-existing): an existing CALM file failed the check, for example a renamed copy of the raw file.

  • error: the error column has MEG160’s message or the failed check.

A recording whose settings differ from the first one of the run gets a warning (a pre-filled time range is compared as “whole recording”, not by its length).

Note on existing CALM files

A file named proc-CALMnoisereduction is not necessarily noise-reduced. When this stage was tested, the script-testing-dataset CALM file of sub-test1_task-400events turned out to be byte-for-byte the raw recording renamed (MEG160 adds a 256-byte history entry when it saves, so a real CALM file is slightly larger than its raw file). --check-existing reports such files as existing_output_not_calm; --overwrite redoes them.