Post-session retrieval and BIDS

Within 48h after the FIRST session

Anatomical images must be screened for incidental findings within 48h after the first session

• Send the T1-weighted and T2-weighted scan to ███ for screening and incidental findings.
• Indicate on our recruits spreadsheet that the participant's first session has been submitted for screening.
• Wait for response from ███ and note down the result of the screening in our our recruits spreadsheet.

To do so, you'll need to first download the data from PACS and then convert the data into BIDS.

What to do when there are incidental findings

• Discuss with ███ how to proceed with the participant.
• Exclude the participant from the study if ███ evaluates they don't meet the participation (inclusion and exclusion) criteria.

Within one week after the completed session

Download the data from the PACS with PACSMAN (only authorized users)

• Log-in into the PACSMAN computer (███)

• Mount a remote filesystem through sshfs:

  sshfs <hostname>:/data/datasets/hcph-pilot-sourcedata \
                 $HOME/data/hcph-pilot \
        <args>
  

• Edit the query file vim $HOME/queries/last-session.csv (most likely, just update with the session's date)

  mydata-onesession.csv

  PatientID,StudyDate
  2022_11_07*,20230503
  

• Prepare and run PACSMAN, pointing the output to the mounted directory.

  pacsman --save -q $HOME/queries/last-session.csv \
         --out_directory $HOME/data/hcph-pilot/ \
         --config /opt/PACSMAN/files/config.json
  

• Unmount the remote filesystem:

  sudo umount $HOME/data/hcph-pilot
  

Retrieve physiological recordings

• Check that the AcqKnowledge file(s) corresponding to the session were added to the Dropbox shared folder and completely uploaded from ████.
• Check that Psychopy's logs and ET's .EDF files corresponding to the session were added to the Dropbox shared folder and completely uploaded from ████.

Within two weeks after the completed session

Convert imaging data to BIDS with HeuDiConv

We use HeuDiConv to convert from the DICOM format generated by the scanner. In addition, starting from the piloting session five, we abide by ReproIn conventions. To support backward compatibility (and some extra, currently unsupported features by the original heuristic file), we have our own heuristic file.

Our custom heuristic file

Our heuristic file largely derives from ReproIn's at the time of writing. The heuristic has a a protocols2fix: dict[str | re.Pattern[str], list[tuple[str, str]]] (lines 113-148), where replacement patterns to permit backward compatibility are written.

# emacs: -*- mode: python; py-indent-offset: 4; indent-tabs-mode: nil -*-
# vi: set ft=python sts=4 ts=4 sw=4 et:
#
# Copyright 2023 The Axon Lab <theaxonlab@gmail.com>
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# We support and encourage derived works from this project, please read
# about our expectations at
#
#     https://www.nipreps.org/community/licensing/
#
# STATEMENT OF CHANGES: This file is derived from sources licensed under the Apache-2.0 terms,
# and this file has been changed.
# The original file this work derives from is found at:
# https://github.com/nipy/heudiconv/blob/55524168b02519bbf0a3a1c94cafb29a419728a0/heudiconv/heuristics/reproin.py
#
# ORIGINAL WORK'S ATTRIBUTION NOTICE:
#
#     Copyright [2014-2019] [Heudiconv developers]
#
#     Licensed under the Apache License, Version 2.0 (the "License");
#     you may not use this file except in compliance with the License.
#     You may obtain a copy of the License at
#
#         http://www.apache.org/licenses/LICENSE-2.0
#
#     Unless required by applicable law or agreed to in writing, software
#     distributed under the License is distributed on an "AS IS" BASIS,
#     WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#     See the License for the specific language governing permissions and
#     limitations under the License.
"""Reproin heuristic."""

from __future__ import annotations

from warnings import warn
from collections import Counter
import logging
import re

import pydicom as dcm

from heudiconv.utils import SeqInfo
from heudiconv.heuristics.reproin import (
    _apply_substitutions,
    get_study_hash,
    get_study_description,
)

lgr = logging.getLogger("heudiconv")


DWI_RES = {
    "1.6mm-iso": "highres",
    "2mm-iso": "lowres",
}


IGNORE_PROTOCOLS = (
    "DEV",
    "LABEL",
    "REPORT",
    "ADC",
    "TRACEW",
    "FA",
    "ColFA",
    "B0",
    "TENSOR",
    "10meas",  # dismiss a trial of fmap acquisition
    "testJB",  # dismiss a test trial of the cmrr sequence
)

bids_regex = re.compile(r"_(?=(dir|acq|task|run)-([A-Za-z0-9]+))")


# Terminology to harmonise and use to name variables etc
# experiment
#  subject
#   [session]
#    exam (AKA scanning session) - currently seqinfo, unless brought together from multiple
#     series  (AKA protocol?)
#      - series_spec - deduced from fields the spec (literal value)
#      - series_info - the dictionary with fields parsed from series_spec

# Which fields in seqinfo (in this order) to check for the ReproIn spec
series_spec_fields = ("protocol_name", "series_description")

# dictionary from accession-number to runs that need to be marked as bad
# NOTE: even if filename has number that is 0-padded, internally no padding
# is done
fix_accession2run: dict[str, list[str]] = {
    # e.g.:
    # 'A000035': ['^8-', '^9-'],
}

# A dictionary containing fixes/remapping for sequence names per study.
# Keys are md5sum of study_description from DICOMs, in the form of PI-Experimenter^protocolname
# You can use `heudiconv -f reproin --command ls --files  PATH
# to list the "study hash".
# Values are list of tuples in the form (regex_pattern, substitution).
# If the  key is an empty string`''''`, it would apply to any study.
protocols2fix: dict[str | re.Pattern[str], list[tuple[str, str]]] = {
    "": [
        ("t1_mprage_pre_Morpho", "anat-T1w_acq-morphobox__mprage"),
        (
            "micro_struct_137dir_BIPOLAR_b3000_1.6mm-iso",
            "dwi-dwi_acq-highres_dir-unknown__137dir_bipolar",
        ),
        (
            "micro_struct_137dir_BIPOLAR_b3000_1.6mm-iso",
            "dwi-dwi_acq-highres_dir-unknown__137dir_bipolar",
        ),
        (
            "micro_struct_137dir_BIPOLAR_b3000_2mm-iso",
            "dwi-dwi_acq-lowres_dir-unknown__137dir_bipolar",
        ),
        ("gre_field_mapping_1.6mmiso", "fmap-phasediff__gre"),
        (
            "cmrr_mbep2d_bold_me4_sms4_fa75_750meas",
            "func-bold_task-rest__750meas",
        ),
        (
            "cmrr_mbep2d_bold_me4_sms4_fa80",
            "func-bold_task-rest_acq-fa80__cmrr",
        ),
        (
            "cmrr_mbep2d_bold_me4_testJB",
            "func-bold_task-rest_acq-testJB__cmrr",
        ),
        (
            "cmrr_mbep2d_bold_fmap_fa80",
            "fmap-epi_acq-bold_dir-unknown__cmrr_mbepd2d_fa80",
        ),
        ("cmrr_mbep2d_bold_me4_sms4", "func-bold_task-qct__cmrr"),
        ("_task-qc_", "_task-qct_"),
        ("anat-T2w__flair", "anat-FLAIR__spcir"),
        ("AAHead_Scout_.*", "anat-scout"),
        ("_dir_RL", "_dir-RL"),
        ("_dir_LR", "_dir-LR"),
        ("_dir_AP", "_dir-AP"),
        ("_dir_PA", "_dir-PA"),
    ]
    # e.g., QA:
    # '43b67d9139e8c7274578b7451ab21123':
    #     [
    #      ('BOLD_p2_s4_3\.5mm', 'func_task-rest_acq-p2-s4-3.5mm'),
    #      ('BOLD_', 'func_task-rest'),
    #      ('_p2_s4',        '_acq-p2-s4'),
    #      ('_p2', '_acq-p2'),
    #     ],
    # '':  # for any study example with regexes used
    #     [
    #         ('AAHead_Scout_.*', 'anat-scout'),
    #         ('^dti_.*', 'dwi'),
    #         ('^.*_distortion_corr.*_([ap]+)_([12])', r'fmap-epi_dir-\1_run-\2'),
    #         ('^(.+)_ap.*_r(0[0-9])', r'func_task-\1_run-\2'),
    #         ('^t1w_.*', 'anat-T1w'),
    #         # problematic case -- multiple identically named pepolar fieldmap runs
    #         # I guess we will just sacrifice ability to detect canceled runs here.
    #         # And we cannot just use _run+ since it would increment independently
    #         # for ap and then for pa.  We will rely on having ap preceding pa.
    #         # Added  _acq-mb8  so they match the one in funcs
    #         ('func_task-discorr_acq-ap', r'fmap-epi_dir-ap_acq-mb8_run+'),
    #         ('func_task-discorr_acq-pa', r'fmap-epi_dir-pa_acq-mb8_run='),
    # ]
}

# list containing StudyInstanceUID to skip -- hopefully doesn't happen too often
dicoms2skip: list[str] = [
    # e.g.
    # '1.3.12.2.1107.5.2.43.66112.30000016110117002435700000001',
]

DEFAULT_FIELDS = {
    # Let it just be in each json file extracted
    "Acknowledgements": "See README.md.",
}

POPULATE_INTENDED_FOR_OPTS = {
    "matching_parameters": ["ImagingVolume", "Shims"],
    "criterion": "Closest",
}


def filter_dicom(dcmdata: dcm.dataset.Dataset) -> bool:
    """Return True if a DICOM dataset should be filtered out, else False"""
    return True if dcmdata.StudyInstanceUID in dicoms2skip else False


def filter_files(_fn: str) -> bool:
    """Return True if a file should be kept, else False.

    ATM reproin does not do any filtering. Override if you need to add some
    """
    return not _fn.endswith((".csv", ".dvs"))


def fix_canceled_runs(seqinfo: list[SeqInfo]) -> list[SeqInfo]:
    """Function that adds cancelme_ to known bad runs which were forgotten"""
    if not fix_accession2run:
        return seqinfo  # nothing to do
    for i, s in enumerate(seqinfo):
        accession_number = s.accession_number
        if accession_number and accession_number in fix_accession2run:
            lgr.info(
                "Considering some runs possibly marked to be "
                "canceled for accession %s",
                accession_number,
            )
            # This code is reminiscent of prior logic when operating on
            # a single accession, but left as is for now
            badruns = fix_accession2run[accession_number]
            badruns_pattern = "|".join(badruns)
            if re.match(badruns_pattern, s.series_id):
                lgr.info("Fixing bad run {0}".format(s.series_id))
                fixedkwargs = dict()
                for key in series_spec_fields:
                    fixedkwargs[key] = "cancelme_" + getattr(s, key)
                seqinfo[i] = s._replace(**fixedkwargs)
    return seqinfo


def fix_dbic_protocol(seqinfo: list[SeqInfo]) -> list[SeqInfo]:
    """Ad-hoc fixup for existing protocols.

    It will operate in 3 stages on `protocols2fix` records.
    1. consider a record which has md5sum of study_description
    2. apply all substitutions, where key is a regular expression which
       successfully searches (not necessarily matches, so anchor appropriately)
       study_description
    3. apply "catch all" substitutions in the key containing an empty string

    3. is somewhat redundant since `re.compile('.*')` could match any, but is
    kept for simplicity of its specification.
    """

    study_hash = get_study_hash(seqinfo)
    study_description = get_study_description(seqinfo)

    # We will consider first study specific (based on hash)
    if study_hash in protocols2fix:
        _apply_substitutions(
            seqinfo, protocols2fix[study_hash], "study (%s) specific" % study_hash
        )
    # Then go through all regexps returning regex "search" result
    # on study_description
    for sub, substitutions in protocols2fix.items():
        if isinstance(sub, re.Pattern) and sub.search(study_description):
            _apply_substitutions(
                seqinfo, substitutions, "%r regex matching" % sub.pattern
            )
    # and at the end - global
    if "" in protocols2fix:
        _apply_substitutions(seqinfo, protocols2fix[""], "global")

    return seqinfo


def fix_seqinfo(seqinfo: list[SeqInfo]) -> list[SeqInfo]:
    """Just a helper on top of both fixers"""
    # add cancelme to known bad runs
    seqinfo = fix_canceled_runs(seqinfo)
    seqinfo = fix_dbic_protocol(seqinfo)
    return seqinfo


def create_key(template, outtype=("nii.gz",), annotation_classes=None):
    if template is None or not template:
        raise ValueError("Template must be a valid format string")
    return template, outtype, annotation_classes


def infotodict(seqinfo):
    """Heuristic evaluator for determining which runs belong where

    allowed template fields - follow python string module:

    item: index within category
    subject: participant id
    seqitem: run number during scanning
    subindex: sub index within group
    """
    seqinfo = fix_seqinfo(seqinfo)
    lgr.info("Processing %d seqinfo entries", len(seqinfo))

    t1w = create_key(
        "sub-{subject}/{session}/anat/sub-{subject}_{session}_acq-{acquisition}{run_entity}_T1w"
    )
    t2w = create_key(
        "sub-{subject}/{session}/anat/sub-{subject}_{session}_acq-{acquisition}{run_entity}_T2w"
    )
    t2_flair = create_key(
        "sub-{subject}/{session}/anat/sub-{subject}_{session}{run_entity}_FLAIR"
    )
    dwi = create_key(
        "sub-{subject}/{session}/dwi/sub-{subject}_{session}_acq-{acq}_dir-{dir}{run_entity}_dwi"
    )
    mag = create_key(
        "sub-{subject}/{session}/fmap/sub-{subject}_{session}{run_entity}_magnitude"
    )
    phdiff = create_key(
        "sub-{subject}/{session}/fmap/sub-{subject}_{session}{run_entity}_phasediff"
    )
    epi = create_key(
        "sub-{subject}/{session}/fmap/sub-{subject}_{session}"
        "_acq-{acquisition}_dir-{dir}{run_entity}{part_entity}_epi"
    )
    func = create_key(
        "sub-{subject}/{session}/func/sub-{subject}_{session}"
        "_task-{task}{acq_entity}{dir_entity}{run_entity}{part_entity}_bold"
    )
    sbref = create_key(
        "sub-{subject}/{session}/func/sub-{subject}_{session}_task-{task}{run_entity}_sbref"
    )

    info = {
        t1w: [],
        t2w: [],
        t2_flair: [],
        dwi: [],
        mag: [],
        phdiff: [],
        epi: [],
        func: [],
        sbref: [],
    }
    epi_mags = []
    bold_mags = []

    for s in seqinfo:
        """
        The namedtuple `s` contains the following fields:

        * total_files_till_now
        * example_dcm_file
        * series_id
        * dcm_dir_name
        * unspecified2
        * unspecified3
        * dim1
        * dim2
        * dim3
        * dim4
        * TR
        * TE
        * protocol_name
        * is_motion_corrected
        * is_derived
        * patient_id
        * study_description
        * referring_physician_name
        * series_description
        * image_type
        """

        # Ignore derived data and reports
        if (
            s.is_derived == "True"
            or s.is_derived is True
            or s.dcm_dir_name.split("_")[-1] in IGNORE_PROTOCOLS
        ):
            continue

        thisitem = {
            "item": s.series_id,
        }
        thiskey = None
        thisitem.update({k: v for k, v in bids_regex.findall(s.protocol_name)})
        thisitem["run_entity"] = f"{thisitem.pop('run', '')}"

        if s.protocol_name.lower().startswith("anat-t1w"):
            thiskey = t1w
            acquisition_present = thisitem.pop("acq", None)
            thisitem["acquisition"] = (
                ("original" if s.dcm_dir_name.endswith("_ND") else "undistorted")
                if not acquisition_present
                else acquisition_present
            )
        elif s.protocol_name.lower().startswith("anat-t2w"):
            thiskey = t2w
            acquisition_present = thisitem.pop("acq", None)
            thisitem["acquisition"] = (
                ("original" if s.dcm_dir_name.endswith("_ND") else "undistorted")
                if not acquisition_present
                else "unspecified"
            )
        elif s.protocol_name.lower().startswith("anat-flair"):
            thiskey = t2_flair
        elif s.protocol_name.startswith("dwi-dwi"):
            thiskey = dwi
        elif s.protocol_name.startswith("fmap-phasediff"):
            thiskey = phdiff if "P" in s.image_type else mag
        elif s.protocol_name.startswith("fmap-epi"):
            thiskey = epi
            thisitem["part_entity"] = ""
            thisitem["acquisition"] = (
                "b0" if s.sequence_name.endswith("ep_b0") else "bold"
            )

            # Check whether phase was written out:
            # 1. A magnitude needs to exist immediately before in the dicom info
            # 2. Magnitude and phase must have the same number of volumes
            series_id_idx, series_id_name = s.series_id.split("-", 1)
            prev_series_id = f"{int(series_id_idx) - 1}-{series_id_name}-{s.series_files}"
            if prev_series_id in epi_mags:
                thisitem["part_entity"] = "_part-phase"
                info[thiskey][epi_mags.index(prev_series_id)]["part_entity"] = "_part-mag"

            epi_mags.append(f"{s.series_id}-{s.series_files}")

        elif s.protocol_name.startswith("func-bold"):
            # Likely an error
            if s.series_files < 100:
                warn(
                    f"Dropping exceedingly short BOLD file with {s.series_files} time points."
                )
                continue

            thiskey = func

            thisitem["part_entity"] = ""
            # Some functional runs may come with acq
            func_acq = thisitem.pop("acq", None)
            thisitem["acq_entity"] = "" if not func_acq else f"_acq-{func_acq}"

            # Some functional runs may come with dir
            func_dir = thisitem.pop("dir", None)
            thisitem["dir_entity"] = "" if not func_dir else f"_dir-{func_dir}"

            # Check whether phase was written out:
            # 1. A magnitude needs to exist immediately before in the dicom info
            # 2. Magnitude and phase must have the same number of volumes
            series_id_idx, series_id_name = s.series_id.split("-", 1)
            prev_series_id = f"{int(series_id_idx) - 1}-{series_id_name}-{s.series_files}"
            if prev_series_id in bold_mags:
                thisitem["part_entity"] = "_part-phase"
                info[thiskey][bold_mags.index(prev_series_id)]["part_entity"] = "_part-mag"

            bold_mags.append(f"{s.series_id}-{s.series_files}")

        if thiskey is not None:
            info[thiskey].append(thisitem)

    for mod, items in info.items():
        if len(items) < 2:
            continue

        info[mod] = _assign_run_on_repeat(items)

    return info


def _assign_run_on_repeat(modality_items):
    """
    Assign run IDs for repeated inputs for a given modality.

    Examples
    --------
    >>> _assign_run_on_repeat([
    ...     {"item": "discard1", "acq": "bold", "dir": "PA"},
    ...     {"item": "discard2", "acq": "bold", "dir": "AP"},
    ...     {"item": "discard3", "acq": "bold", "dir": "PA"},
    ... ])  # doctest: +NORMALIZE_WHITESPACE
    [{'item': 'discard1', 'acq': 'bold', 'dir': 'PA', 'run_entity': '_run-1'},
     {'item': 'discard2', 'acq': 'bold', 'dir': 'AP'},
     {'item': 'discard3', 'acq': 'bold', 'dir': 'PA', 'run_entity': '_run-2'}]

    >>> _assign_run_on_repeat([
    ...     {"item": "discard1", "acq": "bold", "dir": "PA"},
    ...     {"item": "discard2", "acq": "bold", "dir": "AP"},
    ...     {"item": "discard3", "acq": "bold", "dir": "PA"},
    ...     {"item": "discard4", "acq": "bold", "dir": "AP"},
    ... ])  # doctest: +NORMALIZE_WHITESPACE
    [{'item': 'discard1', 'acq': 'bold', 'dir': 'PA', 'run_entity': '_run-1'},
     {'item': 'discard2', 'acq': 'bold', 'dir': 'AP', 'run_entity': '_run-1'},
     {'item': 'discard3', 'acq': 'bold', 'dir': 'PA', 'run_entity': '_run-2'},
     {'item': 'discard4', 'acq': 'bold', 'dir': 'AP', 'run_entity': '_run-2'}]

    >>> _assign_run_on_repeat([
    ...     {"item": "discard1", "acq": "bold", "dir": "PA", "run": "1"},
    ...     {"item": "discard2", "acq": "bold", "dir": "AP"},
    ...     {"item": "discard3", "acq": "bold", "dir": "PA", "run": "2"},
    ... ])  # doctest: +NORMALIZE_WHITESPACE
    [{'item': 'discard1', 'acq': 'bold', 'dir': 'PA', 'run': '1'},
     {'item': 'discard2', 'acq': 'bold', 'dir': 'AP'},
     {'item': 'discard3', 'acq': 'bold', 'dir': 'PA', 'run': '2'}]

    >>> _assign_run_on_repeat([
    ...     {"item": "discard1", "acq": "bold", "dir": "PA", "run_entity": "_run-1"},
    ...     {"item": "discard2", "acq": "bold", "dir": "AP"},
    ...     {"item": "discard3", "acq": "bold", "dir": "PA", "run_entity": "_run-2"},
    ... ])  # doctest: +NORMALIZE_WHITESPACE
    [{'item': 'discard1', 'acq': 'bold', 'dir': 'PA', 'run_entity': '_run-1'},
     {'item': 'discard2', 'acq': 'bold', 'dir': 'AP'},
     {'item': 'discard3', 'acq': 'bold', 'dir': 'PA', 'run_entity': '_run-2'}]

    >>> _assign_run_on_repeat([
    ...     {"item": "discard1", "acq": "bold", "dir": "PA", "part_entity": "_part-mag"},
    ...     {"item": "discard2", "acq": "bold", "dir": "PA", "part_entity": "_part-phase"},
    ...     {"item": "discard3", "acq": "bold", "dir": "AP", "part_entity": "_part-mag"},
    ...     {"item": "discard4", "acq": "bold", "dir": "AP", "part_entity": "_part-phase"},
    ... ])  # doctest: +NORMALIZE_WHITESPACE
    [{'item': 'discard1', 'acq': 'bold', 'dir': 'PA', 'part_entity': '_part-mag'},
     {'item': 'discard2', 'acq': 'bold', 'dir': 'PA', 'part_entity': '_part-phase'},
     {'item': 'discard3', 'acq': 'bold', 'dir': 'AP', 'part_entity': '_part-mag'},
     {'item': 'discard4', 'acq': 'bold', 'dir': 'AP', 'part_entity': '_part-phase'}]

    """
    modality_items = modality_items.copy()

    str_patterns = [
        "_".join([f"{s[0]}-{s[1]}" for s in item.items() if s[0] != "item"])
        for item in modality_items
    ]
    strcount = Counter(str_patterns)

    for string, count in strcount.items():
        if count < 2:
            continue

        runid = 1

        for index, item_string in enumerate(str_patterns):
            if string == item_string:
                modality_items[index].update(
                    {
                        "run_entity": f"_run-{runid}",
                    }
                )
                runid += 1

    return modality_items

During piloting, we changed a number of settings

For example, the first four sessions did not follow Reproin conventions and filenames varied substantially. Please note the protocols2fix variable in our heuristic file, where the compatibility is implemented.

• Run HeuDiConv with our heuristic file <path>/code/heudiconv/reproin.py:

  Executing HeuDiConv

  #!/bin/bash
  heudiconv -s "001" -ss "pilot001" -b -l . -o /data/datasets/hcph/ \
            -f <sops_clone_path>/code/heudiconv/reproin.py \
            --files /data/datasets/hcph-pilot-sourcedata/\
                    sub-01/\
                    ses-18950702/
  

  Session number MUST be updated manually

  Example of the dataset organization

  Piloting sessions 15 and 16 look like this:

  ├── ses-pilot015
  │   ├── anat
  │   │   ├── sub-001_ses-pilot015_acq-original_T1w.json
  │   │   ├── sub-001_ses-pilot015_acq-original_T1w.nii.gz
  │   │   ├── sub-001_ses-pilot015_acq-undistorted_T1w.json
  │   │   ├── sub-001_ses-pilot015_acq-undistorted_T1w.nii.gz
  │   │   ├── sub-001_ses-pilot015_T2w.json
  │   │   └── sub-001_ses-pilot015_T2w.nii.gz
  │   ├── dwi
  │   │   ├── sub-001_ses-pilot015_acq-highres_dir-LR_dwi.bval
  │   │   ├── sub-001_ses-pilot015_acq-highres_dir-LR_dwi.bvec
  │   │   ├── sub-001_ses-pilot015_acq-highres_dir-LR_dwi.json
  │   │   └── sub-001_ses-pilot015_acq-highres_dir-LR_dwi.nii.gz
  │   ├── fmap
  │   │   ├── sub-001_ses-pilot015_acq-b0_dir-RL_epi.json
  │   │   ├── sub-001_ses-pilot015_acq-b0_dir-RL_epi.nii.gz
  │   │   ├── sub-001_ses-pilot015_acq-bold_dir-RL_part-mag_epi.json
  │   │   ├── sub-001_ses-pilot015_acq-bold_dir-RL_part-mag_epi.nii.gz
  │   │   ├── sub-001_ses-pilot015_acq-bold_dir-RL_part-phase_epi.json
  │   │   ├── sub-001_ses-pilot015_acq-bold_dir-RL_part-phase_epi.nii.gz
  │   │   ├── sub-001_ses-pilot015_magnitude1.json
  │   │   ├── sub-001_ses-pilot015_magnitude1.nii.gz
  │   │   ├── sub-001_ses-pilot015_magnitude2.json
  │   │   ├── sub-001_ses-pilot015_magnitude2.nii.gz
  │   │   ├── sub-001_ses-pilot015_phasediff.json
  │   │   └── sub-001_ses-pilot015_phasediff.nii.gz
  │   ├── func
  │   │   ├── sub-001_ses-pilot015_task-bht_echo-1_part-mag_bold.json
  │   │   ├── sub-001_ses-pilot015_task-bht_echo-1_part-mag_bold.nii.gz
  │   │   ├── sub-001_ses-pilot015_task-bht_echo-1_part-phase_bold.json
  │   │   ├── sub-001_ses-pilot015_task-bht_echo-1_part-phase_bold.nii.gz
  │   │   ├── sub-001_ses-pilot015_task-bht_echo-2_part-mag_bold.json
  │   │   ├── sub-001_ses-pilot015_task-bht_echo-2_part-mag_bold.nii.gz
  │   │   ├── sub-001_ses-pilot015_task-bht_echo-2_part-phase_bold.json
  │   │   ├── sub-001_ses-pilot015_task-bht_echo-2_part-phase_bold.nii.gz
  │   │   ├── sub-001_ses-pilot015_task-bht_echo-3_part-mag_bold.json
  │   │   ├── sub-001_ses-pilot015_task-bht_echo-3_part-mag_bold.nii.gz
  │   │   ├── sub-001_ses-pilot015_task-bht_echo-3_part-phase_bold.json
  │   │   ├── sub-001_ses-pilot015_task-bht_echo-3_part-phase_bold.nii.gz
  │   │   ├── sub-001_ses-pilot015_task-bht_echo-4_part-mag_bold.json
  │   │   ├── sub-001_ses-pilot015_task-bht_echo-4_part-mag_bold.nii.gz
  │   │   ├── sub-001_ses-pilot015_task-bht_echo-4_part-phase_bold.json
  │   │   ├── sub-001_ses-pilot015_task-bht_echo-4_part-phase_bold.nii.gz
  │   │   ├── sub-001_ses-pilot015_task-bht_part-mag_events.tsv
  │   │   ├── sub-001_ses-pilot015_task-bht_part-phase_events.tsv
  │   │   ├── sub-001_ses-pilot015_task-qct_echo-1_part-mag_bold.json
  │   │   ├── sub-001_ses-pilot015_task-qct_echo-1_part-mag_bold.nii.gz
  │   │   ├── sub-001_ses-pilot015_task-qct_echo-1_part-phase_bold.json
  │   │   ├── sub-001_ses-pilot015_task-qct_echo-1_part-phase_bold.nii.gz
  │   │   ├── sub-001_ses-pilot015_task-qct_echo-2_part-mag_bold.json
  │   │   ├── sub-001_ses-pilot015_task-qct_echo-2_part-mag_bold.nii.gz
  │   │   ├── sub-001_ses-pilot015_task-qct_echo-2_part-phase_bold.json
  │   │   ├── sub-001_ses-pilot015_task-qct_echo-2_part-phase_bold.nii.gz
  │   │   ├── sub-001_ses-pilot015_task-qct_echo-3_part-mag_bold.json
  │   │   ├── sub-001_ses-pilot015_task-qct_echo-3_part-mag_bold.nii.gz
  │   │   ├── sub-001_ses-pilot015_task-qct_echo-3_part-phase_bold.json
  │   │   ├── sub-001_ses-pilot015_task-qct_echo-3_part-phase_bold.nii.gz
  │   │   ├── sub-001_ses-pilot015_task-qct_echo-4_part-mag_bold.json
  │   │   ├── sub-001_ses-pilot015_task-qct_echo-4_part-mag_bold.nii.gz
  │   │   ├── sub-001_ses-pilot015_task-qct_echo-4_part-phase_bold.json
  │   │   ├── sub-001_ses-pilot015_task-qct_echo-4_part-phase_bold.nii.gz
  │   │   ├── sub-001_ses-pilot015_task-qct_part-mag_events.tsv
  │   │   ├── sub-001_ses-pilot015_task-qct_part-phase_events.tsv
  │   │   ├── sub-001_ses-pilot015_task-rest_echo-1_part-mag_bold.json
  │   │   ├── sub-001_ses-pilot015_task-rest_echo-1_part-mag_bold.nii.gz
  │   │   ├── sub-001_ses-pilot015_task-rest_echo-1_part-phase_bold.json
  │   │   ├── sub-001_ses-pilot015_task-rest_echo-1_part-phase_bold.nii.gz
  │   │   ├── sub-001_ses-pilot015_task-rest_echo-2_part-mag_bold.json
  │   │   ├── sub-001_ses-pilot015_task-rest_echo-2_part-mag_bold.nii.gz
  │   │   ├── sub-001_ses-pilot015_task-rest_echo-2_part-phase_bold.json
  │   │   ├── sub-001_ses-pilot015_task-rest_echo-2_part-phase_bold.nii.gz
  │   │   ├── sub-001_ses-pilot015_task-rest_echo-3_part-mag_bold.json
  │   │   ├── sub-001_ses-pilot015_task-rest_echo-3_part-mag_bold.nii.gz
  │   │   ├── sub-001_ses-pilot015_task-rest_echo-3_part-phase_bold.json
  │   │   ├── sub-001_ses-pilot015_task-rest_echo-3_part-phase_bold.nii.gz
  │   │   ├── sub-001_ses-pilot015_task-rest_echo-4_part-mag_bold.json
  │   │   ├── sub-001_ses-pilot015_task-rest_echo-4_part-mag_bold.nii.gz
  │   │   ├── sub-001_ses-pilot015_task-rest_echo-4_part-phase_bold.json
  │   │   ├── sub-001_ses-pilot015_task-rest_echo-4_part-phase_bold.nii.gz
  │   │   ├── sub-001_ses-pilot015_task-rest_part-mag_events.tsv
  │   │   └── sub-001_ses-pilot015_task-rest_part-phase_events.tsv
  │   └── sub-001_ses-pilot015_scans.tsv
  └── ses-pilot016
      ├── anat
      │   ├── sub-001_ses-pilot016_acq-original_T1w.json
      │   ├── sub-001_ses-pilot016_acq-original_T1w.nii.gz
      │   ├── sub-001_ses-pilot016_acq-undistorted_T1w.json
      │   ├── sub-001_ses-pilot016_acq-undistorted_T1w.nii.gz
      │   ├── sub-001_ses-pilot016_T2w.json
      │   └── sub-001_ses-pilot016_T2w.nii.gz
      ├── dwi
      │   ├── sub-001_ses-pilot016_acq-highres_dir-RL_dwi.bval
      │   ├── sub-001_ses-pilot016_acq-highres_dir-RL_dwi.bvec
      │   ├── sub-001_ses-pilot016_acq-highres_dir-RL_dwi.json
      │   └── sub-001_ses-pilot016_acq-highres_dir-RL_dwi.nii.gz
      ├── fmap
      │   ├── sub-001_ses-pilot016_acq-b0_dir-AP_epi.json
      │   ├── sub-001_ses-pilot016_acq-b0_dir-AP_epi.nii.gz
      │   ├── sub-001_ses-pilot016_acq-b0_dir-LR_epi.json
      │   ├── sub-001_ses-pilot016_acq-b0_dir-LR_epi.nii.gz
      │   ├── sub-001_ses-pilot016_acq-b0_dir-PA_epi.json
      │   ├── sub-001_ses-pilot016_acq-b0_dir-PA_epi.nii.gz
      │   ├── sub-001_ses-pilot016_acq-b0_dir-RL_epi.json
      │   ├── sub-001_ses-pilot016_acq-b0_dir-RL_epi.nii.gz
      │   ├── sub-001_ses-pilot016_acq-bold_dir-AP_part-mag_epi.json
      │   ├── sub-001_ses-pilot016_acq-bold_dir-AP_part-mag_epi.nii.gz
      │   ├── sub-001_ses-pilot016_acq-bold_dir-AP_part-phase_epi.json
      │   ├── sub-001_ses-pilot016_acq-bold_dir-AP_part-phase_epi.nii.gz
      │   ├── sub-001_ses-pilot016_acq-bold_dir-LR_part-mag_epi.json
      │   ├── sub-001_ses-pilot016_acq-bold_dir-LR_part-mag_epi.nii.gz
      │   ├── sub-001_ses-pilot016_acq-bold_dir-LR_part-phase_epi.json
      │   ├── sub-001_ses-pilot016_acq-bold_dir-LR_part-phase_epi.nii.gz
      │   ├── sub-001_ses-pilot016_acq-bold_dir-PA_part-mag_epi.json
      │   ├── sub-001_ses-pilot016_acq-bold_dir-PA_part-mag_epi.nii.gz
      │   ├── sub-001_ses-pilot016_acq-bold_dir-PA_part-phase_epi.json
      │   ├── sub-001_ses-pilot016_acq-bold_dir-PA_part-phase_epi.nii.gz
      │   ├── sub-001_ses-pilot016_acq-bold_dir-RL_part-mag_epi.json
      │   ├── sub-001_ses-pilot016_acq-bold_dir-RL_part-mag_epi.nii.gz
      │   ├── sub-001_ses-pilot016_acq-bold_dir-RL_part-phase_epi.json
      │   ├── sub-001_ses-pilot016_acq-bold_dir-RL_part-phase_epi.nii.gz
      │   ├── sub-001_ses-pilot016_magnitude1.json
      │   ├── sub-001_ses-pilot016_magnitude1.nii.gz
      │   ├── sub-001_ses-pilot016_magnitude2.json
      │   ├── sub-001_ses-pilot016_magnitude2.nii.gz
      │   ├── sub-001_ses-pilot016_phasediff.json
      │   └── sub-001_ses-pilot016_phasediff.nii.gz
      ├── func
      │   ├── sub-001_ses-pilot016_task-bht_echo-1_part-mag_bold.json
      │   ├── sub-001_ses-pilot016_task-bht_echo-1_part-mag_bold.nii.gz
      │   ├── sub-001_ses-pilot016_task-bht_echo-1_part-phase_bold.json
      │   ├── sub-001_ses-pilot016_task-bht_echo-1_part-phase_bold.nii.gz
      │   ├── sub-001_ses-pilot016_task-bht_echo-2_part-mag_bold.json
      │   ├── sub-001_ses-pilot016_task-bht_echo-2_part-mag_bold.nii.gz
      │   ├── sub-001_ses-pilot016_task-bht_echo-2_part-phase_bold.json
      │   ├── sub-001_ses-pilot016_task-bht_echo-2_part-phase_bold.nii.gz
      │   ├── sub-001_ses-pilot016_task-bht_echo-3_part-mag_bold.json
      │   ├── sub-001_ses-pilot016_task-bht_echo-3_part-mag_bold.nii.gz
      │   ├── sub-001_ses-pilot016_task-bht_echo-3_part-phase_bold.json
      │   ├── sub-001_ses-pilot016_task-bht_echo-3_part-phase_bold.nii.gz
      │   ├── sub-001_ses-pilot016_task-bht_echo-4_part-mag_bold.json
      │   ├── sub-001_ses-pilot016_task-bht_echo-4_part-mag_bold.nii.gz
      │   ├── sub-001_ses-pilot016_task-bht_echo-4_part-phase_bold.json
      │   ├── sub-001_ses-pilot016_task-bht_echo-4_part-phase_bold.nii.gz
      │   ├── sub-001_ses-pilot016_task-bht_part-mag_events.tsv
      │   ├── sub-001_ses-pilot016_task-bht_part-phase_events.tsv
      │   ├── sub-001_ses-pilot016_task-qct_echo-1_part-mag_bold.json
      │   ├── sub-001_ses-pilot016_task-qct_echo-1_part-mag_bold.nii.gz
      │   ├── sub-001_ses-pilot016_task-qct_echo-1_part-phase_bold.json
      │   ├── sub-001_ses-pilot016_task-qct_echo-1_part-phase_bold.nii.gz
      │   ├── sub-001_ses-pilot016_task-qct_echo-2_part-mag_bold.json
      │   ├── sub-001_ses-pilot016_task-qct_echo-2_part-mag_bold.nii.gz
      │   ├── sub-001_ses-pilot016_task-qct_echo-2_part-phase_bold.json
      │   ├── sub-001_ses-pilot016_task-qct_echo-2_part-phase_bold.nii.gz
      │   ├── sub-001_ses-pilot016_task-qct_echo-3_part-mag_bold.json
      │   ├── sub-001_ses-pilot016_task-qct_echo-3_part-mag_bold.nii.gz
      │   ├── sub-001_ses-pilot016_task-qct_echo-3_part-phase_bold.json
      │   ├── sub-001_ses-pilot016_task-qct_echo-3_part-phase_bold.nii.gz
      │   ├── sub-001_ses-pilot016_task-qct_echo-4_part-mag_bold.json
      │   ├── sub-001_ses-pilot016_task-qct_echo-4_part-mag_bold.nii.gz
      │   ├── sub-001_ses-pilot016_task-qct_echo-4_part-phase_bold.json
      │   ├── sub-001_ses-pilot016_task-qct_echo-4_part-phase_bold.nii.gz
      │   ├── sub-001_ses-pilot016_task-qct_part-mag_events.tsv
      │   ├── sub-001_ses-pilot016_task-qct_part-phase_events.tsv
      │   ├── sub-001_ses-pilot016_task-rest_echo-1_part-mag_bold.json
      │   ├── sub-001_ses-pilot016_task-rest_echo-1_part-mag_bold.nii.gz
      │   ├── sub-001_ses-pilot016_task-rest_echo-1_part-phase_bold.json
      │   ├── sub-001_ses-pilot016_task-rest_echo-1_part-phase_bold.nii.gz
      │   ├── sub-001_ses-pilot016_task-rest_echo-2_part-mag_bold.json
      │   ├── sub-001_ses-pilot016_task-rest_echo-2_part-mag_bold.nii.gz
      │   ├── sub-001_ses-pilot016_task-rest_echo-2_part-phase_bold.json
      │   ├── sub-001_ses-pilot016_task-rest_echo-2_part-phase_bold.nii.gz
      │   ├── sub-001_ses-pilot016_task-rest_echo-3_part-mag_bold.json
      │   ├── sub-001_ses-pilot016_task-rest_echo-3_part-mag_bold.nii.gz
      │   ├── sub-001_ses-pilot016_task-rest_echo-3_part-phase_bold.json
      │   ├── sub-001_ses-pilot016_task-rest_echo-3_part-phase_bold.nii.gz
      │   ├── sub-001_ses-pilot016_task-rest_echo-4_part-mag_bold.json
      │   ├── sub-001_ses-pilot016_task-rest_echo-4_part-mag_bold.nii.gz
      │   ├── sub-001_ses-pilot016_task-rest_echo-4_part-phase_bold.json
      │   ├── sub-001_ses-pilot016_task-rest_echo-4_part-phase_bold.nii.gz
      │   ├── sub-001_ses-pilot016_task-rest_part-mag_events.tsv
      │   └── sub-001_ses-pilot016_task-rest_part-phase_events.tsv
      └── sub-001_ses-pilot016_scans.tsv
  

  We started to generate phase and magnitude only after session 15

  As a result, the piloting data up to session 14 will look more like:

  ├── ses-pilot014
  │   ├── anat
  │   │   ├── sub-001_ses-pilot014_acq-original_T1w.json
  │   │   ├── sub-001_ses-pilot014_acq-original_T1w.nii.gz
  │   │   ├── sub-001_ses-pilot014_acq-undistorted_T1w.json
  │   │   ├── sub-001_ses-pilot014_acq-undistorted_T1w.nii.gz
  │   │   ├── sub-001_ses-pilot014_T2w.json
  │   │   └── sub-001_ses-pilot014_T2w.nii.gz
  │   ├── dwi
  │   │   ├── sub-001_ses-pilot014_acq-highres_dir-PA_dwi.bval
  │   │   ├── sub-001_ses-pilot014_acq-highres_dir-PA_dwi.bvec
  │   │   ├── sub-001_ses-pilot014_acq-highres_dir-PA_dwi.json
  │   │   └── sub-001_ses-pilot014_acq-highres_dir-PA_dwi.nii.gz
  │   ├── fmap
  │   │   ├── sub-001_ses-pilot014_acq-b0_dir-AP_epi.json
  │   │   ├── sub-001_ses-pilot014_acq-b0_dir-AP_epi.nii.gz
  │   │   ├── sub-001_ses-pilot014_acq-bold_dir-PA_run-1_epi.json
  │   │   ├── sub-001_ses-pilot014_acq-bold_dir-PA_run-1_epi.nii.gz
  │   │   ├── sub-001_ses-pilot014_acq-bold_dir-PA_run-2_epi.json
  │   │   ├── sub-001_ses-pilot014_acq-bold_dir-PA_run-2_epi.nii.gz
  │   │   ├── sub-001_ses-pilot014_magnitude1.json
  │   │   ├── sub-001_ses-pilot014_magnitude1.nii.gz
  │   │   ├── sub-001_ses-pilot014_magnitude2.json
  │   │   ├── sub-001_ses-pilot014_magnitude2.nii.gz
  │   │   ├── sub-001_ses-pilot014_phasediff.json
  │   │   └── sub-001_ses-pilot014_phasediff.nii.gz
  │   ├── func
  │   │   ├── sub-001_ses-pilot014_task-bht_echo-1_bold.json
  │   │   ├── sub-001_ses-pilot014_task-bht_echo-1_bold.nii.gz
  │   │   ├── sub-001_ses-pilot014_task-bht_echo-2_bold.json
  │   │   ├── sub-001_ses-pilot014_task-bht_echo-2_bold.nii.gz
  │   │   ├── sub-001_ses-pilot014_task-bht_echo-3_bold.json
  │   │   ├── sub-001_ses-pilot014_task-bht_echo-3_bold.nii.gz
  │   │   ├── sub-001_ses-pilot014_task-bht_echo-4_bold.json
  │   │   ├── sub-001_ses-pilot014_task-bht_echo-4_bold.nii.gz
  │   │   ├── sub-001_ses-pilot014_task-bht_events.tsv
  │   │   ├── sub-001_ses-pilot014_task-bht_run-1_echo-1_bold.json
  │   │   ├── sub-001_ses-pilot014_task-bht_run-1_echo-1_bold.nii.gz
  │   │   ├── sub-001_ses-pilot014_task-bht_run-1_echo-2_bold.json
  │   │   ├── sub-001_ses-pilot014_task-bht_run-1_echo-2_bold.nii.gz
  │   │   ├── sub-001_ses-pilot014_task-bht_run-1_echo-3_bold.json
  │   │   ├── sub-001_ses-pilot014_task-bht_run-1_echo-3_bold.nii.gz
  │   │   ├── sub-001_ses-pilot014_task-bht_run-1_echo-4_bold.json
  │   │   ├── sub-001_ses-pilot014_task-bht_run-1_echo-4_bold.nii.gz
  │   │   ├── sub-001_ses-pilot014_task-bht_run-1_events.tsv
  │   │   ├── sub-001_ses-pilot014_task-bht_run-2_echo-1_bold.json
  │   │   ├── sub-001_ses-pilot014_task-bht_run-2_echo-1_bold.nii.gz
  │   │   ├── sub-001_ses-pilot014_task-bht_run-2_echo-2_bold.json
  │   │   ├── sub-001_ses-pilot014_task-bht_run-2_echo-2_bold.nii.gz
  │   │   ├── sub-001_ses-pilot014_task-bht_run-2_echo-3_bold.json
  │   │   ├── sub-001_ses-pilot014_task-bht_run-2_echo-3_bold.nii.gz
  │   │   ├── sub-001_ses-pilot014_task-bht_run-2_echo-4_bold.json
  │   │   ├── sub-001_ses-pilot014_task-bht_run-2_echo-4_bold.nii.gz
  │   │   ├── sub-001_ses-pilot014_task-bht_run-2_events.tsv
  │   │   ├── sub-001_ses-pilot014_task-qct_echo-1_bold.json
  │   │   ├── sub-001_ses-pilot014_task-qct_echo-1_bold.nii.gz
  │   │   ├── sub-001_ses-pilot014_task-qct_echo-2_bold.json
  │   │   ├── sub-001_ses-pilot014_task-qct_echo-2_bold.nii.gz
  │   │   ├── sub-001_ses-pilot014_task-qct_echo-3_bold.json
  │   │   ├── sub-001_ses-pilot014_task-qct_echo-3_bold.nii.gz
  │   │   ├── sub-001_ses-pilot014_task-qct_echo-4_bold.json
  │   │   ├── sub-001_ses-pilot014_task-qct_echo-4_bold.nii.gz
  │   │   ├── sub-001_ses-pilot014_task-qct_events.tsv
  │   │   ├── sub-001_ses-pilot014_task-rest_echo-1_bold.json
  │   │   ├── sub-001_ses-pilot014_task-rest_echo-1_bold.nii.gz
  │   │   ├── sub-001_ses-pilot014_task-rest_echo-2_bold.json
  │   │   ├── sub-001_ses-pilot014_task-rest_echo-2_bold.nii.gz
  │   │   ├── sub-001_ses-pilot014_task-rest_echo-3_bold.json
  │   │   ├── sub-001_ses-pilot014_task-rest_echo-3_bold.nii.gz
  │   │   ├── sub-001_ses-pilot014_task-rest_echo-4_bold.json
  │   │   ├── sub-001_ses-pilot014_task-rest_echo-4_bold.nii.gz
  │   │   └── sub-001_ses-pilot014_task-rest_events.tsv
  │   └── sub-001_ses-pilot014_scans.tsv
  

Clean-up after BIDS conversion and preparation for archival

• Delete incorrect files generated by HeuDiConv:

  find sub-001/ -name "*_part-mag_events.tsv" -or -name "*_part-phase_events.tsv" | xargs rm
  

• Compact DICOM session folder and remove it if successful

  tar vczf ses-18950702.tar.gz \
           /data/datasets/hcph-sourcedata/\
           sub-01/\
           ses-18950702 \
  && \
  rm -rf /data/datasets/hcph-sourcedata/\
         sub-01/\
         ses-18950702
  

• Remove write permissions on the newly downloaded data:

  chmod -R a-w /data/datasets/hcph-sourcedata/sub-01/
  

Generate BIDS' events files

• Execute conversion with code/events/psychopy2events:

  python psychopy2events.py --path ./outputdir/sub-001/ses-pilot016/func/
  

Example of a session with events files

The corresponding events files are highlighted below:

ses-024
├── anat
├── dwi
│   ├── sub-001_ses-024_acq-highres_dir-AP_dwi.bval
│   ├── sub-001_ses-024_acq-highres_dir-AP_dwi.bvec
│   ├── sub-001_ses-024_acq-highres_dir-AP_dwi.json
│   ├── sub-001_ses-024_acq-highres_dir-AP_dwi.nii.gz
│   ├── sub-001_ses-024_acq-highres_dir-AP_recording-cardiac_physio.json
│   ├── sub-001_ses-024_acq-highres_dir-AP_recording-cardiac_physio.tsv.gz
│   ├── sub-001_ses-024_acq-highres_dir-AP_recording-respiratory_physio.json
│   ├── sub-001_ses-024_acq-highres_dir-AP_recording-respiratory_physio.tsv.gz
│   ├── sub-001_ses-024_acq-highres_dir-AP_stim.json
│   └── sub-001_ses-024_acq-highres_dir-AP_stim.tsv.gz
├── fmap
├── func
│   ├── sub-001_ses-024_task-bht_dir-AP_echo-1_part-mag_bold.json
│   ├── sub-001_ses-024_task-bht_dir-AP_echo-1_part-mag_bold.nii.gz
│   ├── sub-001_ses-024_task-bht_dir-AP_echo-1_part-phase_bold.json
│   ├── sub-001_ses-024_task-bht_dir-AP_echo-1_part-phase_bold.nii.gz
│   ├── sub-001_ses-024_task-bht_dir-AP_echo-2_part-mag_bold.json
│   ├── sub-001_ses-024_task-bht_dir-AP_echo-2_part-mag_bold.nii.gz
│   ├── sub-001_ses-024_task-bht_dir-AP_echo-2_part-phase_bold.json
│   ├── sub-001_ses-024_task-bht_dir-AP_echo-2_part-phase_bold.nii.gz
│   ├── sub-001_ses-024_task-bht_dir-AP_echo-3_part-mag_bold.json
│   ├── sub-001_ses-024_task-bht_dir-AP_echo-3_part-mag_bold.nii.gz
│   ├── sub-001_ses-024_task-bht_dir-AP_echo-3_part-phase_bold.json
│   ├── sub-001_ses-024_task-bht_dir-AP_echo-3_part-phase_bold.nii.gz
│   ├── sub-001_ses-024_task-bht_dir-AP_echo-4_part-mag_bold.json
│   ├── sub-001_ses-024_task-bht_dir-AP_echo-4_part-mag_bold.nii.gz
│   ├── sub-001_ses-024_task-bht_dir-AP_echo-4_part-phase_bold.json
│   ├── sub-001_ses-024_task-bht_dir-AP_echo-4_part-phase_bold.nii.gz
│   ├── sub-001_ses-024_acq-highres_dir-AP_dwi.bval
│   ├── sub-001_ses-024_acq-highres_dir-AP_dwi.bvec
│   ├── sub-001_ses-024_acq-highres_dir-AP_dwi.json
│   ├── sub-001_ses-024_acq-highres_dir-AP_dwi.nii.gz
│   ├── sub-001_ses-024_acq-highres_dir-AP_recording-cardiac_physio.json
│   ├── sub-001_ses-024_acq-highres_dir-AP_recording-cardiac_physio.tsv.gz
│   ├── sub-001_ses-024_acq-highres_dir-AP_recording-respiratory_physio.json
│   ├── sub-001_ses-024_acq-highres_dir-AP_recording-respiratory_physio.tsv.gz
│   ├── sub-001_ses-024_acq-highres_dir-AP_stim.json
│   └── sub-001_ses-024_acq-highres_dir-AP_stim.tsv.gz
│   ├── sub-001_ses-024_task-bht_dir-AP_events.tsv
│   ├── sub-001_ses-024_task-qct_dir-AP_echo-1_part-mag_bold.json
│   ├── sub-001_ses-024_task-qct_dir-AP_echo-1_part-mag_bold.nii.gz
│   ├── sub-001_ses-024_task-qct_dir-AP_echo-1_part-phase_bold.json
│   ├── sub-001_ses-024_task-qct_dir-AP_echo-1_part-phase_bold.nii.gz
│   ├── sub-001_ses-024_task-qct_dir-AP_echo-2_part-mag_bold.json
│   ├── sub-001_ses-024_task-qct_dir-AP_echo-2_part-mag_bold.nii.gz
│   ├── sub-001_ses-024_task-qct_dir-AP_echo-2_part-phase_bold.json
│   ├── sub-001_ses-024_task-qct_dir-AP_echo-2_part-phase_bold.nii.gz
│   ├── sub-001_ses-024_task-qct_dir-AP_echo-3_part-mag_bold.json
│   ├── sub-001_ses-024_task-qct_dir-AP_echo-3_part-mag_bold.nii.gz
│   ├── sub-001_ses-024_task-qct_dir-AP_echo-3_part-phase_bold.json
│   ├── sub-001_ses-024_task-qct_dir-AP_echo-3_part-phase_bold.nii.gz
│   ├── sub-001_ses-024_task-qct_dir-AP_echo-4_part-mag_bold.json
│   ├── sub-001_ses-024_task-qct_dir-AP_echo-4_part-mag_bold.nii.gz
│   ├── sub-001_ses-024_task-qct_dir-AP_echo-4_part-phase_bold.json
│   ├── sub-001_ses-024_task-qct_dir-AP_echo-4_part-phase_bold.nii.gz
│   ├── sub-001_ses-024_task-qct_dir-AP_recording-cardiac_physio.json
│   ├── sub-001_ses-024_task-qct_dir-AP_recording-cardiac_physio.tsv.gz
│   ├── sub-001_ses-024_task-qct_dir-AP_recording-respiratory_physio.json
│   ├── sub-001_ses-024_task-qct_dir-AP_recording-respiratory_physio.tsv.gz
│   ├── sub-001_ses-024_task-qct_dir-AP_stim.json
│   ├── sub-001_ses-024_task-qct_dir-AP_stim.tsv.gz
│   ├── sub-001_ses-024_task-qct_dir-AP_events.tsv
│   ├── sub-001_ses-024_task-rest_dir-AP_echo-1_part-mag_bold.json
│   ├── sub-001_ses-024_task-rest_dir-AP_echo-1_part-mag_bold.nii.gz
│   ├── sub-001_ses-024_task-rest_dir-AP_echo-1_part-phase_bold.json
│   ├── sub-001_ses-024_task-rest_dir-AP_echo-1_part-phase_bold.nii.gz
│   ├── sub-001_ses-024_task-rest_dir-AP_echo-2_part-mag_bold.json
│   ├── sub-001_ses-024_task-rest_dir-AP_echo-2_part-mag_bold.nii.gz
│   ├── sub-001_ses-024_task-rest_dir-AP_echo-2_part-phase_bold.json
│   ├── sub-001_ses-024_task-rest_dir-AP_echo-2_part-phase_bold.nii.gz
│   ├── sub-001_ses-024_task-rest_dir-AP_echo-3_part-mag_bold.json
│   ├── sub-001_ses-024_task-rest_dir-AP_echo-3_part-mag_bold.nii.gz
│   ├── sub-001_ses-024_task-rest_dir-AP_echo-3_part-phase_bold.json
│   ├── sub-001_ses-024_task-rest_dir-AP_echo-3_part-phase_bold.nii.gz
│   ├── sub-001_ses-024_task-rest_dir-AP_echo-4_part-mag_bold.json
│   ├── sub-001_ses-024_task-rest_dir-AP_echo-4_part-mag_bold.nii.gz
│   ├── sub-001_ses-024_task-rest_dir-AP_echo-4_part-phase_bold.json
│   ├── sub-001_ses-024_task-rest_dir-AP_echo-4_part-phase_bold.nii.gz
│   ├── sub-001_ses-024_task-rest_dir-AP_recording-cardiac_physio.json
│   ├── sub-001_ses-024_task-rest_dir-AP_recording-cardiac_physio.tsv.gz
│   ├── sub-001_ses-024_task-rest_dir-AP_recording-respiratory_physio.json
│   ├── sub-001_ses-024_task-rest_dir-AP_recording-respiratory_physio.tsv.gz
│   ├── sub-001_ses-024_task-rest_dir-AP_stim.json
│   └── sub-001_ses-024_task-rest_dir-AP_stim.tsv.gz
│   └── sub-001_ses-024_task-rest_dir-AP_events.tsv
└── sub-001_ses-024_scans.tsv

Convert physiological recordings into BIDS (in-house)

• Install the necessary packages.

  python -m pip install bioread pandas matplotlib numpy pathlib scipy
  

• Update the appropriate session number within cell 3 in the conversion Jupyter notebook.

• Execute the notebook.

Example of a session with physiological recordings

The execution of the notebook on session 24 yields the following new outputs (highlighted):

ses-024
├── anat
├── dwi
│   ├── sub-001_ses-024_acq-highres_dir-AP_dwi.bval
│   ├── sub-001_ses-024_acq-highres_dir-AP_dwi.bvec
│   ├── sub-001_ses-024_acq-highres_dir-AP_dwi.json
│   ├── sub-001_ses-024_acq-highres_dir-AP_dwi.nii.gz
│   ├── sub-001_ses-024_acq-highres_dir-AP_recording-cardiac_physio.json
│   ├── sub-001_ses-024_acq-highres_dir-AP_recording-cardiac_physio.tsv.gz
│   ├── sub-001_ses-024_acq-highres_dir-AP_recording-respiratory_physio.json
│   ├── sub-001_ses-024_acq-highres_dir-AP_recording-respiratory_physio.tsv.gz
│   ├── sub-001_ses-024_acq-highres_dir-AP_stim.json
│   └── sub-001_ses-024_acq-highres_dir-AP_stim.tsv.gz
├── fmap
├── func
│   ├── sub-001_ses-024_task-bht_dir-AP_echo-1_part-mag_bold.json
│   ├── sub-001_ses-024_task-bht_dir-AP_echo-1_part-mag_bold.nii.gz
│   ├── sub-001_ses-024_task-bht_dir-AP_echo-1_part-phase_bold.json
│   ├── sub-001_ses-024_task-bht_dir-AP_echo-1_part-phase_bold.nii.gz
│   ├── sub-001_ses-024_task-bht_dir-AP_echo-2_part-mag_bold.json
│   ├── sub-001_ses-024_task-bht_dir-AP_echo-2_part-mag_bold.nii.gz
│   ├── sub-001_ses-024_task-bht_dir-AP_echo-2_part-phase_bold.json
│   ├── sub-001_ses-024_task-bht_dir-AP_echo-2_part-phase_bold.nii.gz
│   ├── sub-001_ses-024_task-bht_dir-AP_echo-3_part-mag_bold.json
│   ├── sub-001_ses-024_task-bht_dir-AP_echo-3_part-mag_bold.nii.gz
│   ├── sub-001_ses-024_task-bht_dir-AP_echo-3_part-phase_bold.json
│   ├── sub-001_ses-024_task-bht_dir-AP_echo-3_part-phase_bold.nii.gz
│   ├── sub-001_ses-024_task-bht_dir-AP_echo-4_part-mag_bold.json
│   ├── sub-001_ses-024_task-bht_dir-AP_echo-4_part-mag_bold.nii.gz
│   ├── sub-001_ses-024_task-bht_dir-AP_echo-4_part-phase_bold.json
│   ├── sub-001_ses-024_task-bht_dir-AP_echo-4_part-phase_bold.nii.gz
│   ├── sub-001_ses-024_task-bht_dir-AP_recording-cardiac_physio.json
│   ├── sub-001_ses-024_task-bht_dir-AP_recording-cardiac_physio.tsv.gz
│   ├── sub-001_ses-024_task-bht_dir-AP_recording-respiratory_physio.json
│   ├── sub-001_ses-024_task-bht_dir-AP_recording-respiratory_physio.tsv.gz
│   ├── sub-001_ses-024_task-bht_dir-AP_stim.json
│   ├── sub-001_ses-024_task-bht_dir-AP_stim.tsv.gz
│   ├── sub-001_ses-024_task-qct_dir-AP_echo-1_part-mag_bold.json
│   ├── sub-001_ses-024_task-qct_dir-AP_echo-1_part-mag_bold.nii.gz
│   ├── sub-001_ses-024_task-qct_dir-AP_echo-1_part-phase_bold.json
│   ├── sub-001_ses-024_task-qct_dir-AP_echo-1_part-phase_bold.nii.gz
│   ├── sub-001_ses-024_task-qct_dir-AP_echo-2_part-mag_bold.json
│   ├── sub-001_ses-024_task-qct_dir-AP_echo-2_part-mag_bold.nii.gz
│   ├── sub-001_ses-024_task-qct_dir-AP_echo-2_part-phase_bold.json
│   ├── sub-001_ses-024_task-qct_dir-AP_echo-2_part-phase_bold.nii.gz
│   ├── sub-001_ses-024_task-qct_dir-AP_echo-3_part-mag_bold.json
│   ├── sub-001_ses-024_task-qct_dir-AP_echo-3_part-mag_bold.nii.gz
│   ├── sub-001_ses-024_task-qct_dir-AP_echo-3_part-phase_bold.json
│   ├── sub-001_ses-024_task-qct_dir-AP_echo-3_part-phase_bold.nii.gz
│   ├── sub-001_ses-024_task-qct_dir-AP_echo-4_part-mag_bold.json
│   ├── sub-001_ses-024_task-qct_dir-AP_echo-4_part-mag_bold.nii.gz
│   ├── sub-001_ses-024_task-qct_dir-AP_echo-4_part-phase_bold.json
│   ├── sub-001_ses-024_task-qct_dir-AP_echo-4_part-phase_bold.nii.gz
│   ├── sub-001_ses-024_task-qct_dir-AP_recording-cardiac_physio.json
│   ├── sub-001_ses-024_task-qct_dir-AP_recording-cardiac_physio.tsv.gz
│   ├── sub-001_ses-024_task-qct_dir-AP_recording-respiratory_physio.json
│   ├── sub-001_ses-024_task-qct_dir-AP_recording-respiratory_physio.tsv.gz
│   ├── sub-001_ses-024_task-qct_dir-AP_stim.json
│   ├── sub-001_ses-024_task-qct_dir-AP_stim.tsv.gz
│   ├── sub-001_ses-024_task-rest_dir-AP_echo-1_part-mag_bold.json
│   ├── sub-001_ses-024_task-rest_dir-AP_echo-1_part-mag_bold.nii.gz
│   ├── sub-001_ses-024_task-rest_dir-AP_echo-1_part-phase_bold.json
│   ├── sub-001_ses-024_task-rest_dir-AP_echo-1_part-phase_bold.nii.gz
│   ├── sub-001_ses-024_task-rest_dir-AP_echo-2_part-mag_bold.json
│   ├── sub-001_ses-024_task-rest_dir-AP_echo-2_part-mag_bold.nii.gz
│   ├── sub-001_ses-024_task-rest_dir-AP_echo-2_part-phase_bold.json
│   ├── sub-001_ses-024_task-rest_dir-AP_echo-2_part-phase_bold.nii.gz
│   ├── sub-001_ses-024_task-rest_dir-AP_echo-3_part-mag_bold.json
│   ├── sub-001_ses-024_task-rest_dir-AP_echo-3_part-mag_bold.nii.gz
│   ├── sub-001_ses-024_task-rest_dir-AP_echo-3_part-phase_bold.json
│   ├── sub-001_ses-024_task-rest_dir-AP_echo-3_part-phase_bold.nii.gz
│   ├── sub-001_ses-024_task-rest_dir-AP_echo-4_part-mag_bold.json
│   ├── sub-001_ses-024_task-rest_dir-AP_echo-4_part-mag_bold.nii.gz
│   ├── sub-001_ses-024_task-rest_dir-AP_echo-4_part-phase_bold.json
│   ├── sub-001_ses-024_task-rest_dir-AP_echo-4_part-phase_bold.nii.gz
│   ├── sub-001_ses-024_task-rest_dir-AP_recording-cardiac_physio.json
│   ├── sub-001_ses-024_task-rest_dir-AP_recording-cardiac_physio.tsv.gz
│   ├── sub-001_ses-024_task-rest_dir-AP_recording-respiratory_physio.json
│   ├── sub-001_ses-024_task-rest_dir-AP_recording-respiratory_physio.tsv.gz
│   ├── sub-001_ses-024_task-rest_dir-AP_stim.json
│   └── sub-001_ses-024_task-rest_dir-AP_stim.tsv.gz
└── sub-001_ses-024_scans.tsv

Convert eye-tracking into BIDS (in-house)

Instead of the current specifications, we are using the following BEP

• Check that PyEDFRead is installed as described in the software annex.
• Employ our in-house conversion code, whose functioning is described in our EDF to BIDS conversion how-to.

Example of a session with eye-tracking recordings

The files corresponding to eye-tracking are highlighted below:

ses-024
├── anat
├── dwi
├── fmap
├── func
│   ├── sub-001_ses-024_task-bht_dir-AP_echo-1_part-mag_bold.json
│   ├── sub-001_ses-024_task-bht_dir-AP_echo-1_part-mag_bold.nii.gz
│   ├── sub-001_ses-024_task-bht_dir-AP_echo-1_part-phase_bold.json
│   ├── sub-001_ses-024_task-bht_dir-AP_echo-1_part-phase_bold.nii.gz
│   ├── sub-001_ses-024_task-bht_dir-AP_echo-2_part-mag_bold.json
│   ├── sub-001_ses-024_task-bht_dir-AP_echo-2_part-mag_bold.nii.gz
│   ├── sub-001_ses-024_task-bht_dir-AP_echo-2_part-phase_bold.json
│   ├── sub-001_ses-024_task-bht_dir-AP_echo-2_part-phase_bold.nii.gz
│   ├── sub-001_ses-024_task-bht_dir-AP_echo-3_part-mag_bold.json
│   ├── sub-001_ses-024_task-bht_dir-AP_echo-3_part-mag_bold.nii.gz
│   ├── sub-001_ses-024_task-bht_dir-AP_echo-3_part-phase_bold.json
│   ├── sub-001_ses-024_task-bht_dir-AP_echo-3_part-phase_bold.nii.gz
│   ├── sub-001_ses-024_task-bht_dir-AP_echo-4_part-mag_bold.json
│   ├── sub-001_ses-024_task-bht_dir-AP_echo-4_part-mag_bold.nii.gz
│   ├── sub-001_ses-024_task-bht_dir-AP_echo-4_part-phase_bold.json
│   ├── sub-001_ses-024_task-bht_dir-AP_echo-4_part-phase_bold.nii.gz
│   ├── sub-001_ses-024_task-bht_dir-AP_eyetrack.json
│   ├── sub-001_ses-024_task-bht_dir-AP_eyetrack.tsv.gz
│   ├── sub-001_ses-024_task-qct_dir-AP_echo-1_part-mag_bold.json
│   ├── sub-001_ses-024_task-qct_dir-AP_echo-1_part-mag_bold.nii.gz
│   ├── sub-001_ses-024_task-qct_dir-AP_echo-1_part-phase_bold.json
│   ├── sub-001_ses-024_task-qct_dir-AP_echo-1_part-phase_bold.nii.gz
│   ├── sub-001_ses-024_task-qct_dir-AP_echo-2_part-mag_bold.json
│   ├── sub-001_ses-024_task-qct_dir-AP_echo-2_part-mag_bold.nii.gz
│   ├── sub-001_ses-024_task-qct_dir-AP_echo-2_part-phase_bold.json
│   ├── sub-001_ses-024_task-qct_dir-AP_echo-2_part-phase_bold.nii.gz
│   ├── sub-001_ses-024_task-qct_dir-AP_echo-3_part-mag_bold.json
│   ├── sub-001_ses-024_task-qct_dir-AP_echo-3_part-mag_bold.nii.gz
│   ├── sub-001_ses-024_task-qct_dir-AP_echo-3_part-phase_bold.json
│   ├── sub-001_ses-024_task-qct_dir-AP_echo-3_part-phase_bold.nii.gz
│   ├── sub-001_ses-024_task-qct_dir-AP_echo-4_part-mag_bold.json
│   ├── sub-001_ses-024_task-qct_dir-AP_echo-4_part-mag_bold.nii.gz
│   ├── sub-001_ses-024_task-qct_dir-AP_echo-4_part-phase_bold.json
│   ├── sub-001_ses-024_task-qct_dir-AP_echo-4_part-phase_bold.nii.gz
│   ├── sub-001_ses-024_task-qct_dir-AP_eyetrack.json
│   ├── sub-001_ses-024_task-qct_dir-AP_eyetrack.tsv.gz
│   ├── sub-001_ses-024_task-rest_dir-AP_echo-1_part-mag_bold.json
│   ├── sub-001_ses-024_task-rest_dir-AP_echo-1_part-mag_bold.nii.gz
│   ├── sub-001_ses-024_task-rest_dir-AP_echo-1_part-phase_bold.json
│   ├── sub-001_ses-024_task-rest_dir-AP_echo-1_part-phase_bold.nii.gz
│   ├── sub-001_ses-024_task-rest_dir-AP_echo-2_part-mag_bold.json
│   ├── sub-001_ses-024_task-rest_dir-AP_echo-2_part-mag_bold.nii.gz
│   ├── sub-001_ses-024_task-rest_dir-AP_echo-2_part-phase_bold.json
│   ├── sub-001_ses-024_task-rest_dir-AP_echo-2_part-phase_bold.nii.gz
│   ├── sub-001_ses-024_task-rest_dir-AP_echo-3_part-mag_bold.json
│   ├── sub-001_ses-024_task-rest_dir-AP_echo-3_part-mag_bold.nii.gz
│   ├── sub-001_ses-024_task-rest_dir-AP_echo-3_part-phase_bold.json
│   ├── sub-001_ses-024_task-rest_dir-AP_echo-3_part-phase_bold.nii.gz
│   ├── sub-001_ses-024_task-rest_dir-AP_echo-4_part-mag_bold.json
│   ├── sub-001_ses-024_task-rest_dir-AP_echo-4_part-mag_bold.nii.gz
│   ├── sub-001_ses-024_task-rest_dir-AP_echo-4_part-phase_bold.json
│   ├── sub-001_ses-024_task-rest_dir-AP_echo-4_part-phase_bold.nii.gz
│   ├── sub-001_ses-024_task-rest_dir-AP_eyetrack.json
│   └── sub-001_ses-024_task-rest_dir-AP_eyetrack.tsv.gz
└── sub-001_ses-024_scans.tsv

Add new data to the DataLad dataset

As new sessions are collected, the corresponding BIDS structures MUST be saved within the DataLad dataset and pushed to remote storage systems:

• Save the files in the dataset history using the command below. Replace <session_id> below with the number of the session (e.g., pilot017):

  datalad save -r -m "add: session <session_id>" sub-001/ses-<session_id>
  

  Always double-check that data are annexed and the metadata committed to git

  Although the creation of a procedure should ensure data and metadata are added to the appropriate version control (Git or Git-Annex), it is possible that some metadata or data formats are not anticipated, or do not follow the general rules.

  A generally good strategy is to avoid recursion (i.e., do not use the -r flag), and leverage Bash's find and xargs tools. For example, the following command line selects metadata files that should be committed to Git within one session (pilot020) and saves them:

  find sub-001/ses-pilot020 -name "*.tsv" -or -name "*.json" -or -name "*.bvec" -or -name "*.bval" | xargs datalad save --to-git -m '"add(pilot020): new session metadata"'
  

  Correspondingly, we can store NIfTI data and physiological information:

  find sub-001/ses-pilot020 -name "*.nii.gz" -or -name "*_eyetrack.tsv.gz" -or -name "*_physio.tsv.gz" -or -name "*_stim.tsv.gz" | xargs datalad save -m '"add(pilot020): new session NIfTI data, eye tracking and physio"'
  

  Please read DataLad's save documentation

  If you overeagerly datalad-saved too many files

  You can revert the datalad save operation without deleting changes with:

  git reset --mixed COMMIT
  

  where COMMIT is the hash of the last commit you want to keep (all the later commits will be dropped).

  To check the commit hash where you can roll history back to, you may want to use:

  git log -50 --oneline
  

  Saving batches of sessions

  It is possible to save several sessions with the following Bash script by enumerating them in the array defined in the first line here:

  SESSIONS=( 001 003 pilot21 ); \
  for SESSION in ${SESSIONS[@]}; do \
      find sub-001/ses-$SESSION -name "*.tsv" -or -name "*.json" -or -name "*.bvec" -or -name "*.bval" | xargs datalad save --to-git -m '"add('"$SESSION"'): new session metadata"'; \
      find sub-001/ses-$SESSION -name "*.nii.gz" -or -name "*_eyetrack.tsv.gz" -or -name "*_physio.tsv.gz" -or -name "*_stim.tsv.gz" | xargs datalad save -m '"add('"$SESSION"'): new session NIfTI data, eye tracking and physio"'; \
  done
  

• Push the new data to the remote storage (if your git containing DataLad and the Git annex is different from origin, e.g., github, replace the name below):

  datalad push --to ria-storage
  datalad push --to origin
  

  Always double-check that data in the annex are uploaded to the RIA store

Formal QC

• Consult the session logs to anticipate session peculiarities (e.g the session was aborted prematurely) and potential quality issues (e.g the participan fell asleep). Those are saved in the issues of our repository with the label scan. Keep note of the peculiar events, associated with their session index, and keep it close to you during quality control.

• Run the BIDS Validator to check the formal quality of the dataset (filenames, homogeneity of modalities and parameters across sessions, etc.)

  docker run -ti --rm -v /path/to/data//hcph/:/data:ro bids/validator /data
  

BIDS non-compliance: WARNINGS and ERRORS

We do not fully comply with current BIDS specifications, so some ERRORS and WARNINGS will emerge. If errors and warnings are not listed here, please reach out to decide on a solution.

1. ERRORS: Because we follow BEP 020 and it is not official yet, ET-related files will source ERRORS:

   [ERR] Files with such naming scheme are not part of BIDS specification.
   

2. WARNINGS: some warnings are expected. Warnings that are not among this list should be addressed and BIDS conversion should be re-run on the affected files:

   • During the piloting phase of the study, we tried out different sequence parameters and sequence type. As such, the following warning is expected:

     [WARN] Not all subjects/sessions/runs have the same scanning parameters. (code: 39 - INCONSISTENT_PARAMETERS)
     

   • The _magnitude{1,2}.nii.gz files corresponding to some pilot sessions are missing:

     [WARN] Each _phasediff.nii[.gz] file should be associated with a _magnitude1.nii[.gz] file. (code: 92 MISSING_MAGNITUDE1_FILE)
     ./sub-001/ses-pilot001/fmap/sub-001_ses-pilot001_phasediff.nii.gz
     ./sub-001/ses-pilot004/fmap/sub-001_ses-pilot004_phasediff.nii.gz
     ./sub-001/ses-pilot006/fmap/sub-001_ses-pilot006_phasediff.nii.gz
     

Visual assessment of unprocessed data with MRIQC

Checking the data quality shortly after they are acquired increases the likelihood of catching systematic artifacts early enough to avert spreading throughout the whole dataset. It also modulates the burden of visual inspection over time, such that we avoid overwhelming raters with outbursts of images to assess. Better pacing in rating throughput also contributes to reducing raters' attrition and fatigue.

• Screen all the unprocessed data and assess them as described in the next section.

Upon updates and bugfixes of the dataset

Update PyBIDS's database index

The PyBIDS index cache dramatically speeds up MRIQC, fMRIPrep and dMRIPrep

To speed up the tear-up time of NiPreps tools (MRIQC, fMRIPrep, and dMRIPrep) and other relevant code using PyBIDS, we have added a database cache under the /data/datasets/hcph-dataset/.bids-index/ folder. This cache can be leverage by adding the --bids-database-dir /data/datasets/hcph-dataset/.bids-index/ to the corresponding command line.

• Unlock the database file to enable update:

  cd /data/datasets/hcph-dataset
  datalad unlock .bids-index/layout_index.sqlite
  

• Reset the database file:

  python -m pip install "pybids>=0.16"
  $( dirname $( which python ) )/pybids layout --reset-db --no-validate --index-metadata . .bids-index/
  

  Successful execution will finalize with a message: Successfully generated database index at /data/datasets/hcph-dataset/.bids-index.

• Save the dataset again:

  datalad save -m "enh: updated PyBIDS' database file" /data/datasets/hcph-dataset/.bids-index
  

• Push the changes back to the repos:

  datalad push --to=github