[ENH] microelectrode electrophysiology specification (BEP032) #1705

yarikoptic · 2024-02-14T15:24:13Z

Replaces #1352 submitted from a fork outside of bids-specification.

Add specification for microelectrode electrohpysiology datasets based on the BEP032 proposal

Note

We meet regularly and everyone is welcome

Next meeting: insert date on URL to join

Communication channel: https://framalistes.org/sympa/info/neuroscience-data-structure

Tip

HTML preview of this BEP

To use this WiP schema on sample datasets, see https://deno.land/x/[email protected]#modifying-and-building-a-new-schema on how to use "stock" bids-validator with a custom schema. (attn @TheChymera)

DONEs

TODOs

Issues this PR would likely to address

Fixes run node tests in github actions legacy-validator#1375

Issues to see being addressed while working on this BEP (likely to move above) or not (moved below):

Other issues which relate but not in scope here and provided for reference/backreference

… _acq-

src/schema/rules/files/raw/channels.yaml

TheChymera · 2024-03-06T16:39:46Z

src/schema/objects/columns.yaml

@@ -42,6 +42,38 @@ age:
    for privacy purposes.
  type: number
  unit: year
+alpha_rotation:


I'd really recommend renaming the rotation axes to yaw, roll, and pitch (that would be the analogous angle order). There was no consensus either way on the google docs discussion. Someone said both are confusing, which I guess might be expected, but alpha, beta, gamma, are just more confusing...

I agree -- very adhoc. Let's discuss in google doc

https://docs.google.com/document/d/1oG-C8T-dWPqfVzL2W8HO3elWK8NIh2cOCPssRGv23n0/edit?disco=AAAA4fkI4eY

Could you chime in? I think the other guy commenting might be amenable to accepting this as well.

I agree that both are confusing, but at least alpha, beta and gamma are SO confusing that everyone realizes that additional specification is needed to define them properly. With roll, yaw and pitch it seems at first that all is clear, until you have a number of different people go through different use cases. See the more challenging examples that I posted on the google doc under https://docs.google.com/document/d/1oG-C8T-dWPqfVzL2W8HO3elWK8NIh2cOCPssRGv23n0/edit?disco=AAAA4fkI4eY

@robertoostenveld sorry, I only now saw the update, reply there. I don't think confusion on purpose is good in this case, because the documentation is very eclectic so we might be sending people down rabbit holes. Wiki, where people will invariably go first, does a particularly poor job explaining both euler and TB angles for the casual non-mathematics-versed user. The only thing that wiki has going for it here are the aircraft animations on the TB page. Yaw, Pitch, Roll, will be intuited correctly as long as we specify the starting postion. That we can do (1) as (I think, it's pretty vague) is currently proposed, aligning the implant with the world coordinate system (meaning most implants will be at yaw 0 pitch -90 roll 0) or (2) relative to the implantation stereotax normal (meaning most implants will be at 0 0 0).

For comparison, Euler commonly has the normal pointing up so most implants will be.... 0 -180 0 🤔

in the last BEP032 meeting we discussed further and wanted to follow this approach now (based on the Allen Inst. standard and IBL standard):

Assumption: x,y,z is posterior, ventral, right (unit needs to be specified).

Translational origin (0,0,0) needs to be defined (typically Bregma for rodents).

Rotational origin (0,0,0) is the probe facing up with the tip facing forward. Rotations are all clockwise in degrees and around the tip. For multi-shank probes, the “tip” of the probe is defined as the end of the left shank if you are looking at the electrodes.

yaw: clockwise when looking down

pitch: In the direction of the electrode face

roll: clockwise when looking down at the probe

The depth (unit needs to be specified) is a translation in the direction that the tip is pointing.

We need to add a “probe_model”, which references probeinterface_library

versioning of probe library files (raised here)

annotation of “tip” position (raised here)

NOTE: We need to change the electrode x,y,z.
X,y,z in BIDS refers to location in brain, not on probe.

TheChymera · 2024-03-06T16:46:27Z

src/schema/objects/columns.yaml

@@ -391,6 +462,12 @@ reference__ieeg:
    - type: string
      enum:
        - n/a
+reference_atlas:


The “reference atlas” if you visually verify the area is basically the atlas you looked at before you nod and said “eh, good enough”. This seems at once an overly detailed (does this really matter? the coordinates are commonly given with sub-mm precision) and underdetermined (what coordinates did you look at in the atlas?). I think it's best just to drop this.

We need to double check the setting again, but I think the point is that in some animal studies there are no real coordinates but only semantic statements about the location of a probe/electrode. In these cases it is important to know which atlas they are semantically referring to. Also: in rodents (as far as I know) the atlases are used to plan the surgery. So here it identifies the target area (in coordinates and semantics of the atlas), while the actual location might of cause differ (hopefully only a bit).

TheChymera · 2024-03-12T01:33:27Z

Also relevant if you'd like to comment. → https://docs.google.com/document/d/1oG-C8T-dWPqfVzL2W8HO3elWK8NIh2cOCPssRGv23n0/edit?disco=AAABIzHGpUU

TheChymera · 2024-03-12T01:57:39Z

Also I forgot to link to this when I wrote it → https://docs.google.com/document/d/1oG-C8T-dWPqfVzL2W8HO3elWK8NIh2cOCPssRGv23n0/edit?disco=AAABIGPAMOw

src/schema/rules/files/raw/ephys.yaml

…t was missing

src/schema/objects/datatypes.yaml

src/schema/objects/modalities.yaml

src/schema/objects/suffixes.yaml

bendichter · 2024-12-19T07:31:22Z

src/modality-specific-files/microelectrode-electrophysiology.md

+Task Events documentation.
+Note that this file can also be used to describe stimulation performed during the recording. For this,
+please follow the iEEG stimulation documentation.


To do: add links here.

src/modality-specific-files/microelectrode-electrophysiology.md

bendichter · 2024-12-19T07:32:52Z

src/modality-specific-files/microelectrode-electrophysiology.md

+in the `*_events.tsv` file.
+These two options are made available to support different usages and habits of the experimenters, as
+well as to benefit from the capability of the supported data formats (NWB and NIX).
+They are described in the following subsections, and made explicit through some of the example data sets.


My read of the BIDS standard is that (i) _scans.tsv is the more canonical approach. What is the justification for supporting both methods?

also unclear to me on why _scans.tsv to be used here at all for "events" related info.

also smells very related to https://bids.neuroimaging.io/bep037 "Non-Invasive Brain Stimulation (NIBS)" ... and questions on how Deep Brain Stimulation (no BEP yet) would do it...

In that bep037 I referred to

device/dev- entity + devices.{tsv,json} #2027

where to potential describe some common to devices metadata.

bendichter · 2024-12-19T07:34:03Z

src/modality-specific-files/microelectrode-electrophysiology.md

+session (for example, recording start times in case the recording was paused and restarted)
+when the data from each of these different recordings is stored in separate files.
+Each data file should have a name that contains a `_task-XX` and/or `_run-XX` suffix, and
+should be described by at most one row in the `*_scans.tsv` file. See also the BIDS Scans


Suggested change

should be described by at most one row in the `*_scans.tsv` file. See also the BIDS Scans

should be described by one row in the `*_scans.tsv` file. See also the BIDS Scans

why "at most"? Shouldn't it be exactly 1, so this scan file can specify the start time?

bendichter · 2024-12-19T07:37:12Z

src/modality-specific-files/microelectrode-electrophysiology.md

+be expressed in the following format 2009-06-15T13:45:30 (year, month, day, hour (24h),
+minute, second; this is equivalent to the RFC3339 “date-time” format, time zone is always
+assumed as local time).


Suggested change

be expressed in the following format 2009-06-15T13:45:30 (year, month, day, hour (24h),

minute, second; this is equivalent to the RFC3339 “date-time” format, time zone is always

assumed as local time).

be expressed in the RFC3339 “date-time” format, for example 2009-06-15T13:45:30 (year, month, day, hour (24h), minute, second. Time zone is always assumed as local time.

Fixing up a bit:

Suggested change

be expressed in the following format 2009-06-15T13:45:30 (year, month, day, hour (24h),

minute, second; this is equivalent to the RFC3339 “date-time” format, time zone is always

assumed as local time).

be expressed in the RFC3339 "date-time" format, for example `2009-06-15T13:45:30` (year, month, day, hour (24h), minute, second). Time zone is always assumed as local time.

?

lzehl · 2024-10-16T12:51:55Z

src/schema/objects/columns.yaml

@@ -391,6 +462,12 @@ reference__ieeg:
    - type: string
      enum:
        - n/a
+reference_atlas:


We need to double check the setting again, but I think the point is that in some animal studies there are no real coordinates but only semantic statements about the location of a probe/electrode. In these cases it is important to know which atlas they are semantically referring to. Also: in rodents (as far as I know) the atlases are used to plan the surgery. So here it identifies the target area (in coordinates and semantics of the atlas), while the actual location might of cause differ (hopefully only a bit).

lzehl · 2025-01-22T10:49:18Z

src/schema/objects/entities.yaml

@@ -342,12 +342,12 @@ split:
  display_name: Split
  description: |
    In the case of long data recordings that exceed a file size of 2Gb,
-    `.fif` files are conventionally split into multiple parts.
-    Each of these files has an internal pointer to the next file.
+    `.fif`, `.nwb`, `.nix` files are conventionally split into multiple parts.


@bendichter is NWB also providing internal pointers to the next split file? The next comments only explain the .fif split files further.

@ree-gupta is this split also true for .nix files?

lzehl · 2025-01-22T10:58:34Z

src/modality-specific-files/microelectrode-electrophysiology.md

+It follows the [general BIDS specifications to describe participants](../modality-agnostic-files.md#participants-file).
+
+On top of the existing columns that can be present in this file and that are described in the BIDS specifications (`participant_id`, `species`, `strain`, `strain_rrid`, `sex`, `handedness`, and `age`), we propose to allow adding the following ones:
+


@Peyman-N there is no issue with the participant sections regarding sensitivity in the model agnostic files section. This can be handled through REQUIRED, RECOMMENDED, OPTIONAL plus an awareness comment in the Description of the general column names. So I would agree in removing this section from the modality-specific section and updating instead the model agnostic files section

lzehl · 2025-01-22T11:02:46Z

src/modality-specific-files/microelectrode-electrophysiology.md

+
+
+## Coordinate System JSON (`*_coordsystem.json`) & Photos of electrode positions (`_photo.jpg`)
+


@bendichter I agree with you that probes and electrodes should be discussed first. And in iEEG and EEG channels and electrodes descriptions also come prior to the coordinate system description. Feel free to move it.

src/modality-specific-files/microelectrode-electrophysiology.md

lzehl · 2025-01-22T11:37:40Z

src/modality-specific-files/microelectrode-electrophysiology.md

+Example of * _electrodes.tsv:
+
+```tsv
+probe_id	impedance	x	y	z	material	location


electrode_id is missing form the table

lzehl · 2025-01-22T11:38:09Z

src/modality-specific-files/microelectrode-electrophysiology.md

+The electrode positions and properties are stored in a `.tsv` file (amplifier information is in `channels.tsv`).
+
+This file contains the following information:
+- The electrode name


electrode name or ID ?

lzehl · 2025-01-22T14:33:51Z

src/modality-specific-files/microelectrode-electrophysiology.md

+
+
+```tsv
+channel_id	electrode_id	gain	type	units	sampling_frequency	status


sampling_frequency has a set unit?

lzehl · 2025-01-22T14:35:29Z

src/modality-specific-files/microelectrode-electrophysiology.md

+
+**Recommended Channel Type Values**
+
+For the `type` column we recommend to use the following terms (adopted from [iEEG](intracranial-electroencephalography.md#channelselectrode-description-_channelselectrodestsv))


types of example are not occurring in that table. should be added??

if this is an enum list we may not need this table hardcoded but have it in a schema and pulled in?

replace with macro (if macro is there)

replace EXT in example above with more specific channel type (e.g. LFP, MUA, SPIKES,etc )

lzehl · 2025-01-22T14:35:57Z

src/modality-specific-files/microelectrode-electrophysiology.md

+
+There should be one such JSON file for each data file.
+
+The `*_ephys.json` file can be used to store any ephys-specific metadata for the dataset. We recommend storing all setup-related metadata in a dedicated node of the JSON file called `Setup`.


Suggested change

The `*_ephys.json` file can be used to store any ephys-specific metadata for the dataset. We recommend storing all setup-related metadata in a dedicated node of the JSON file called `Setup`.

The `*_microephys.json` file can be used to store any microephys-specific metadata for the dataset. We recommend storing all setup-related metadata in a dedicated node of the JSON file called `Setup`.

we have no _microephys suffix but two _ecephys and _icephys, see e.g. examples in rendered version: https://bids-specification--1705.org.readthedocs.build/en/1705/modality-specific-files/microelectrode-electrophysiology.html#primary-data-file-formats .

[FIX] Bep032 - fixing Markdown

[BEP032] [SCHEMA] Channel enums for microephys

yarikoptic · 2025-02-05T23:50:03Z

src/modality-specific-files/microelectrode-electrophysiology.md

+
+## General ephys metadata (*_ephys.json)
+
+We propose to store all metadata that is not directly related to one of the other metadata files (probe/electrode/channel information) into a single JSON file: `_ephys.json`.


we have no _ephys suffix defined any longer -- for better or for worse separating out _icephys and _ecephys, so this whole section should be re-reviewed and adjusted?

yarikoptic · 2025-02-05T23:57:30Z

src/modality-specific-files/microelectrode-electrophysiology.md

+If acquisition time is included, it should be with the “acq_time” header. Datetime should
+be expressed in the following format 2009-06-15T13:45:30 (year, month, day, hour (24h),
+minute, second; this is equivalent to the RFC3339 “date-time” format, time zone is always
+assumed as local time).


I recalled we had some discussion but -- shouldn't we allow here timezone specification and assume "local" (which would be what? any placement to state?) only if not specified?

Does this even play a role? Important is that within the same BIDS dataset time is specified within the same time zone, no?

we should probably raise awareness that this information might be GDPR sensitive?

src/modality-specific-files/microelectrode-electrophysiology.md

Co-authored-by: Lyuba Zehl <[email protected]> Co-authored-by: Ben Dichter <[email protected]>

lzehl · 2025-02-19T14:43:25Z

src/schema/objects/enums.yaml

@@ -763,6 +763,7 @@ MISC:
    - ieeg
    - fnirs
    - motion
+    - microephys


maybe I'm confusing things but should this not be icephys and ecephys? microephys was just the abbreviation we used for the overall specification chapter no?

lzehl · 2025-02-19T14:53:33Z

src/modality-specific-files/microelectrode-electrophysiology.md

+
+{{ MACROS___make_sidecar_table("microephys.microephysTaskInformation") }}
+
+### Example of * _ephys.json:


_ephys.json does not exist anymore as suffix, no? should be icephys and ecephys

src/modality-specific-files/microelectrode-electrophysiology.md

yarikoptic · 2025-02-19T15:30:34Z

src/modality-specific-files/microelectrode-electrophysiology.md

+For the `type` column we recommend to use the following terms (adopted from [iEEG](intracranial-electroencephalography.md#channelselectrode-description-_channelselectrodestsv))
+
+
+| **Keyword**   | **Description**                                                        |


TODO for @yarikoptic -- replace with a macro (if possible)

Co-authored-by: Ben Dichter <[email protected]>

Also used appropriate suffix in a few spots when encountered

Also extended for microephys for channels and coordsystems

Add first BEP032 objects

7ec5d5e

yarikoptic requested review from tsalo and erdalkaraca as code owners February 14, 2024 15:24

yarikoptic mentioned this pull request Feb 14, 2024

[ENH] Add BEP032 (microelectrode electrophysiology) specification #1352

Closed

3 tasks

Add ephys modality, start adding _channels but ran into a question on…

2f393ca

… _acq-

yarikoptic commented Feb 14, 2024

View reviewed changes

src/schema/rules/files/raw/channels.yaml Show resolved Hide resolved

tsalo added the BEP label Feb 15, 2024

yarikoptic mentioned this pull request Feb 29, 2024

Allow for using an arbitrary suffix and/or arbitrary entity bids-standard/bids-2-devel#63

Open

yarikoptic and others added 2 commits March 5, 2024 19:56

Add _acq- to ephys, add _channels and _coords

648781a

Draft for probes TSV file

d3c9996

TheChymera reviewed Mar 6, 2024

View reviewed changes

TheChymera closed this Mar 7, 2024

TheChymera deleted the bep032 branch March 7, 2024 22:13

TheChymera restored the bep032 branch March 7, 2024 22:15

TheChymera reopened this Mar 11, 2024

Fix preferably typo

08c92cd

yarikoptic mentioned this pull request Mar 15, 2024

[SCHEMA] Remove stale test comment -- test no longer fails! #1727

Merged

yarikoptic commented Mar 15, 2024

View reviewed changes

src/schema/rules/files/raw/ephys.yaml Outdated Show resolved Hide resolved

robertoostenveld mentioned this pull request Mar 21, 2024

Draft examples for BEP032 on animal electrophysiology bids-standard/bids-examples#430

Draft

yarikoptic mentioned this pull request Apr 2, 2024

Automatic generation of extra-fields template forms from yaml-based schema (BIDS standard) elabftw/elabftw#5023

Open

Comment out not yet defined iephys and add ephys to datatypes where i…

e2030ac

…t was missing

yarikoptic commented Apr 17, 2024

View reviewed changes

src/schema/objects/datatypes.yaml Outdated Show resolved Hide resolved

dorahermes reviewed Apr 17, 2024

View reviewed changes

src/schema/objects/modalities.yaml Outdated Show resolved Hide resolved

dorahermes reviewed Apr 17, 2024

View reviewed changes

src/schema/objects/suffixes.yaml Outdated Show resolved Hide resolved

TheChymera force-pushed the bep032 branch from cde80a6 to e2030ac Compare April 17, 2024 21:26

Added probes file

53f1087

Remi-Gau changed the title ~~[ENH] Add BEP032 (microelectrode electrophysiology) specification~~ [ENH] microelectrode electrophysiology specification (BEP032) Apr 18, 2024

bendichter reviewed Dec 19, 2024

View reviewed changes

src/modality-specific-files/microelectrode-electrophysiology.md Outdated Show resolved Hide resolved

bendichter reviewed Dec 19, 2024

View reviewed changes

Peyman-N and others added 3 commits January 17, 2025 15:20

using MACROS___make_sidecar_table

8e65a63

numbering

2e3be31

Adding icephys and ecephys to filename template

1762f6f

lzehl reviewed Jan 22, 2025

View reviewed changes

yarikoptic and others added 4 commits January 22, 2025 10:35

Merge pull request #2034 from Peyman-N/bep032-peyman

1229a4a

[FIX] Bep032 - fixing Markdown

[ENH] Added microephys tag to electrode related enums

04c1b93

[ENH] Added additional channel enums for microphys

1fd8139

[ENH] Enlisted microphys channel types in type__channels

94de4e5

ree-gupta mentioned this pull request Jan 28, 2025

[BEP032] [SCHEMA] Channel enums for microephys #2038

Merged

Merge pull request #2038 from ree-gupta/bep032

5bc95b0

[BEP032] [SCHEMA] Channel enums for microephys

yarikoptic mentioned this pull request Feb 5, 2025

[ENH] channels.tsv and electrodes.tsv should have a clear relationship #2041

Open

yarikoptic commented Feb 5, 2025

View reviewed changes

src/modality-specific-files/microelectrode-electrophysiology.md Outdated Show resolved Hide resolved

Various small fixes from code-review

b171eee

Co-authored-by: Lyuba Zehl <[email protected]> Co-authored-by: Ben Dichter <[email protected]>

neuromechanist mentioned this pull request Feb 7, 2025

BEP042 surface electromyography (HDsEMG/EMG) #1371

Open

lzehl reviewed Feb 19, 2025

View reviewed changes

yarikoptic commented Feb 19, 2025

View reviewed changes

src/modality-specific-files/microelectrode-electrophysiology.md Outdated Show resolved Hide resolved

yarikoptic commented Feb 19, 2025

View reviewed changes

yarikoptic and others added 5 commits February 19, 2025 10:33

Minor tuneups from code review (mostly on wording)

50a38db

Co-authored-by: Ben Dichter <[email protected]>

A pass to harmonize to use "microephys" in favor over "ephys"

73976f1

Also used appropriate suffix in a few spots when encountered

Encode in schema that ATM we do require _electodes and _probes

387b924

Also extended for microephys for channels and coordsystems

One more ephys -> microephys

d791dc5

Add .json to data (nwb and nix) and _probes files

41dfec7

	should be described by at most one row in the `*_scans.tsv` file. See also the BIDS Scans
	should be described by one row in the `*_scans.tsv` file. See also the BIDS Scans

		It follows the [general BIDS specifications to describe participants](../modality-agnostic-files.md#participants-file).

		On top of the existing columns that can be present in this file and that are described in the BIDS specifications (`participant_id`, `species`, `strain`, `strain_rrid`, `sex`, `handedness`, and `age`), we propose to allow adding the following ones:



		## Coordinate System JSON (`*_coordsystem.json`) & Photos of electrode positions (`_photo.jpg`)



		```tsv
		channel_id electrode_id gain type units sampling_frequency status


		Recommended Channel Type Values

		For the `type` column we recommend to use the following terms (adopted from [iEEG](intracranial-electroencephalography.md#channelselectrode-description-_channelselectrodestsv))


		There should be one such JSON file for each data file.

		The `*_ephys.json` file can be used to store any ephys-specific metadata for the dataset. We recommend storing all setup-related metadata in a dedicated node of the JSON file called `Setup`.

	The `*_ephys.json` file can be used to store any ephys-specific metadata for the dataset. We recommend storing all setup-related metadata in a dedicated node of the JSON file called `Setup`.
	The `*_microephys.json` file can be used to store any microephys-specific metadata for the dataset. We recommend storing all setup-related metadata in a dedicated node of the JSON file called `Setup`.


		## General ephys metadata (*_ephys.json)

		We propose to store all metadata that is not directly related to one of the other metadata files (probe/electrode/channel information) into a single JSON file: `_ephys.json`.


		{{ MACROS___make_sidecar_table("microephys.microephysTaskInformation") }}

		### Example of * _ephys.json:

		For the `type` column we recommend to use the following terms (adopted from [iEEG](intracranial-electroencephalography.md#channelselectrode-description-_channelselectrodestsv))


		\| Keyword \| Description \|

[ENH] microelectrode electrophysiology specification (BEP032) #1705

Are you sure you want to change the base?

[ENH] microelectrode electrophysiology specification (BEP032) #1705

Conversation

yarikoptic commented Feb 14, 2024 • edited Loading

TODOs

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TheChymera Apr 17, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TheChymera commented Mar 12, 2024 • edited Loading

TheChymera commented Mar 12, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yarikoptic commented Feb 14, 2024 •

edited

Loading

TheChymera Apr 17, 2024 •

edited

Loading

TheChymera commented Mar 12, 2024 •

edited

Loading