milestone A: add benchmark and notebook scaffolding

Author: snesmaeili

.gitignore

@@ -39,3 +39,16 @@ htmlcov/
 /figures/
 /outputs/
 /scratch/
+/results/
+
+# Benchmark artifacts
+benchmarks/reports/generated/
+
+# Notebook and generated example artifacts
+notebooks/.ipynb_checkpoints/
+**/.ipynb_checkpoints/
+examples/output/
+
+# Local logs
+*.log
+tests/jdss_log.txt

README.md

@@ -154,6 +154,22 @@ pytest
 pytest --cov=mne_denoise --cov-report=html
 ```
 
+## Benchmarking and notebook structure
+
+The repository now includes scaffolding for reproducible benchmarking and
+dataset tutorial notebooks:
+
+- `benchmarks/` contains config-driven benchmarking modules.
+- `benchmarks/configs/` contains dataset, method, metric, and experiment YAMLs.
+- `notebooks/` contains tutorial tracks split by dataset and step.
+- `scripts/benchmark/run_benchmark.ps1` runs a benchmark experiment config.
+
+Example entrypoint:
+
+```powershell
+python -m benchmarks.runners.run_experiment --config benchmarks/configs/experiments/pilot_milestone_a.yaml
+```
+
 ## Contributing
 
 We welcome contributions! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.

benchmarks/README.md

@@ -0,0 +1,23 @@
+# Benchmark Suite
+
+This folder contains the reproducible benchmarking framework for `mne-denoise`.
+
+## Scope
+
+- Dataset-specific loading and preprocessing wrappers.
+- Method registry including baselines and `mne-denoise` methods.
+- Standard metric computation for denoising quality and preservation.
+- Experiment runners that produce machine-readable outputs.
+
+## Conventions
+
+- Config-first runs: all experiments are defined in YAML under `benchmarks/configs/`.
+- Determinism: all methods should accept a `random_state`.
+- Outputs: write per-run artifacts to `results/` (ignored by git).
+
+## Next steps
+
+- Implement loaders in `benchmarks/datasets/registry.py`.
+- Implement method wrappers in `benchmarks/methods/registry.py`.
+- Implement metrics in `benchmarks/metrics/core.py`.
+- Wire runner entrypoint in `benchmarks/runners/run_experiment.py`.

benchmarks/__init__.py

@@ -0,0 +1 @@
+"""Benchmarking utilities for mne-denoise."""

benchmarks/configs/README.md

@@ -0,0 +1,8 @@
+# Benchmark Configs
+
+Config groups:
+
+- `datasets/`: dataset roots, modalities, and split policies.
+- `methods/`: parameter grids and default method settings.
+- `metrics/`: enabled metrics and thresholds.
+- `experiments/`: explicit combinations of dataset x methods x metrics.

benchmarks/configs/datasets/ds003620_runabout_mobi.yaml

@@ -0,0 +1,18 @@
+name: ds003620_runabout_mobi
+modality: eeg_mobile
+description: OpenNeuro Runabout MoBI dataset for nonstationary line and motion noise.
+dataset_root: data/raw/openneuro/ds003620
+fetch:
+  enabled: false
+  source: openneuro
+  dataset: ds003620
+splits:
+  type: subject
+  subjects: []
+preprocessing:
+  l_freq: 1.0
+  h_freq: 120.0
+  resample_sfreq: 250.0
+metadata:
+  task: oddball
+  primary_use_case: zapline_plus_stress_test

benchmarks/configs/datasets/eegdenoisenet.yaml

@@ -0,0 +1,17 @@
+name: eegdenoisenet
+modality: eeg
+description: Artifact-focused benchmark for EOG/EMG denoising performance.
+dataset_root: data/raw/eegdenoisenet
+fetch:
+  enabled: false
+  source: gin_or_github
+splits:
+  type: predefined
+  subjects: []
+preprocessing:
+  l_freq: 1.0
+  h_freq: 100.0
+  resample_sfreq: 256.0
+metadata:
+  task: artifact_removal
+  primary_use_case: eog_emg_benchmark

benchmarks/configs/datasets/mne_sample.yaml

@@ -0,0 +1,17 @@
+name: mne_sample
+modality: meg_eeg
+description: MNE sample dataset for ERP/ERF preservation and baseline comparisons.
+dataset_root: data/raw/mne_sample
+fetch:
+  enabled: true
+  source: mne.datasets.sample
+splits:
+  type: subject_session
+  subjects: [sample]
+preprocessing:
+  l_freq: 1.0
+  h_freq: 100.0
+  resample_sfreq: 250.0
+metadata:
+  task: auditory_visual
+  primary_use_case: evoked_preservation

benchmarks/configs/datasets/ssvep_template.yaml

@@ -0,0 +1,16 @@
+name: ssvep_template
+modality: eeg_ssvep
+description: Template for SSVEP benchmark datasets (MAMEM/Tsinghua/BETA).
+dataset_root: data/raw/ssvep
+fetch:
+  enabled: false
+splits:
+  type: subject
+  subjects: []
+preprocessing:
+  l_freq: 2.0
+  h_freq: 90.0
+  resample_sfreq: 250.0
+metadata:
+  task: ssvep
+  primary_use_case: periodic_dss

benchmarks/configs/experiments/pilot_milestone_a.yaml

@@ -0,0 +1,17 @@
+name: pilot_milestone_a
+seed: 42
+output_root: results/benchmarks/pilot_milestone_a
+datasets:
+  - benchmarks/configs/datasets/mne_sample.yaml
+  - benchmarks/configs/datasets/ds003620_runabout_mobi.yaml
+methods:
+  - benchmarks/configs/methods/baselines.yaml
+  - benchmarks/configs/methods/mne_denoise.yaml
+metrics:
+  - benchmarks/configs/metrics/primary.yaml
+runtime:
+  max_subjects_per_dataset: 2
+  save_intermediate: true
+reporting:
+  export_markdown_summary: true
+  export_csv_metrics: true