sct_run_batch

Wrapper to processing scripts, which loops across subjects. Subjects should be organized as folders within a single directory. We recommend following the BIDS convention (https://bids.neuroimaging.io/). The processing script should accept a subject directory as its only argument. Additional information is passed via environment variables and the arguments passed via -script-args. If the script or the input data are located within a git repository, the git commit is displayed. If the script or data have changed since the latest commit, the symbol “*” is added after the git commit number. If no git repository is found, the git commit version displays “?!?”. The script is copied on the output folder (-path-out).

usage: sct_run_batch [-config CONFIG] [-jobs <int>] [-itk-threads <int>]
                     [-path-data PATH_DATA] [-subject-prefix SUBJECT_PREFIX]
                     [-path-output PATH_OUTPUT] [-batch-log BATCH_LOG]
                     [-include INCLUDE]
                     [-include-list INCLUDE_LIST [INCLUDE_LIST ...]]
                     [-include-yml INCLUDE_YML] [-exclude EXCLUDE]
                     [-exclude-list EXCLUDE_LIST [EXCLUDE_LIST ...]]
                     [-exclude-yml EXCLUDE_YML] [-ignore-ses]
                     [-path-segmanual PATH_SEGMANUAL] [-script-args SCRIPT_ARGS]
                     [-email-to EMAIL_TO] [-email-from EMAIL_FROM]
                     [-email-host EMAIL_HOST] [-continue-on-error {0,1}]
                     [-script SCRIPT] [-zip] [-v <int>] [-h]

Named Arguments

-config, -c

A json (.json) or yaml (.yml/.yaml) file with arguments. All arguments to the configuration file are the same as the command line arguments, except all dashes (-) are replaced with underscores (_). Using command line flags can be used to override arguments provided in the configuration file, but this is discouraged. Please note that while quotes are optional for strings in YAML omitting them may cause parse errors.

Note that for the "exclude_list" (or "include_list") argument you can exclude/include entire subjects or individual sessions; see examples below.

Example YAML configuration:

path_data    : "~/sct_data"
path_output  : "~/pipeline_results"
script       : "nature_paper_analysis.sh"
jobs         : -1
exclude_list : ["sub-01/ses-01", "sub-02", "ses-03"]  # this will exclude ses-01 for sub-01, all sessions for sub-02 and ses-03 for all subjects

Example JSON configuration:

{
    "path_data"   : "~/sct_data",
    "path_output" : "~/pipeline_results",
    "script"      : "nature_paper_analysis.sh",
    "jobs"        : -1,
    "exclude_list" : ["sub-01/ses-01", "sub-02", "ses-03"]
}
-jobs

The number of jobs to run in parallel. Either an integer greater than or equal to one specifying the number of cores, 0 or a negative integer specifying number of cores minus that number. For example -jobs -1 will run with all the available cores minus one job in parallel. Set -jobs 0 to use all available cores.

This argument enables process-based parallelism, while -itk-threads enables thread-based parallelism. You may need to tweak both to find a balance that works best for your system.

Default: 1

-itk-threads

Sets the environment variable ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS.

Number of threads to use for ITK based programs including ANTs. Increasing this can provide a performance boost for high-performance (multi-core) computing environments. However, increasing the number of threads may also result in a large increase in memory.

This argument enables thread-based parallelism, while -jobs enables process-based parallelism. You may need to tweak both to find a balance that works best for your system.

Default: 1

-path-data

Setting for environment variable: PATH_DATA

Path containing subject directories in a consistent format)

-subject-prefix

Subject prefix, defaults to “sub-” which is the prefix used for BIDS directories. If the subject directories do not share a common prefix, an empty string can be passed here.

Default: 'sub-'

-path-output

Base directory for environment variables:

  • PATH_DATA_PROCESSED=<path-output>/data_processed

  • PATH_RESULTS=<path-output>/results

  • PATH_QC=<path-output>/qc

  • PATH_LOG=<path-output>/log

Which are respectively output paths for the processed data, results, quality control (QC) and logs

Default: '.'

-batch-log

A log file for all terminal output produced by this script (not necessarily including the individual job outputs. File will be relative to “/log”.

Default: 'sct_run_batch_log.txt'

-include

Optional regex used to filter the list of subject directories. Only process a subject if they match the regex. Inclusions are processed before exclusions. Cannot be used with include-list.

-include-list

Optional space separated list of subjects or sessions to include. Only process subjects or sessions if they are on this list. Inclusions are processed before exclusions. Cannot be used with -include. You can combine subjects and sessions; see examples.

Examples: -include-list sub-001 sub-002 or -include-list sub-001/ses-01 ses-02

-include-yml

Path to a YAML file (e.g. include.yml) containing a bullet list (-) of subjects or files to include.

The list will be treated differently depending on whether subjects or files are listed:

  • Subject/session folders (e.g. sub-01 or sub-01/ses-01): The same as -include-list (subject/session inclusion).

  • Files (ending in .nii or .nii.gz): The script will be run, and an environment variable called INCLUDE_FILES will be created containing the list of files. That way, you can choose to run specific processing steps within your script for only those files.

You can use the following bash syntax to process included files:

# check if FILE_T2 is in the list of included files
FILE_T2="$SUBJECT"_T2w.nii.gz
if [[ " $INCLUDE_FILES " =~ " $FILE_T2 " ]]; then
    # process file only if it is included
    FILE_T2_SEG="$SUBJECT"_T2w_seg.nii.gz
    sct_deepseg spinalcord $FILE_T2 -o $FILE_T2_SEG
fi
-exclude

Optional regex used to filter the list of subject directories. Only process a subject if they do not match the regex. Exclusions are processed after inclusions. Cannot be used with exclude-list

-exclude-list

Optional space separated list of subjects or sessions to exclude. Only process subjects or sessions if they are not on this list. Inclusions are processed before exclusions. Cannot be used with -exclude. You can combine subjects and sessions; see examples.

Examples: -exclude-list sub-003 sub-004 or -exclude-list sub-003/ses-01 ses-02

-exclude-yml

Path to a YAML file (e.g. exclude.yml) containing a bullet list (-) of subjects or files to exclude.

The list will be treated differently depending on whether subjects or files are listed:

  • Subject/session folders (e.g. sub-01 or sub-01/ses-01): The same as -exclude-list (subject/session exclusion).

  • Files (ending in .nii or .nii.gz): The script will still be run, however an environment variable called EXCLUDE_FILES will be created containing the list of files. That way, you can choose to skip specific processing steps within your script.

You can use the following bash syntax to skip files:

# check if FILE_T2 is in the list of excluded files
FILE_T2="$SUBJECT"_T2w.nii.gz
if [[ ! " $EXCLUDE_FILES " =~ " $FILE_T2 " ]]; then
    # process file only if it is not excluded
    FILE_T2_SEG="$SUBJECT"_T2w_seg.nii.gz
    sct_deepseg spinalcord $FILE_T2 -o $FILE_T2_SEG
fi

Note: You can use QC reports to generate a YAML file with excluded files (via the “Save Fails” button).

-ignore-ses

By default, if ‘ses’ subfolders are present, then ‘sct_run_batch’ will run the script within each individual ‘ses’ subfolder. Passing -ignore-ses will change the behavior so that ‘sct_run_batch’ will not go into each ‘ses’ folder. Instead, it will run the script on just the top-level subject folders.

Default: False

-path-segmanual

Setting for environment variable: PATH_SEGMANUAL A path containing manual segmentations to be used by the script program.

Default: '.'

-script-args

A quoted string with extra arguments to pass to the script. For example: sct_run_batch -path-data data/ -script process_data.sh -script-args "ARG1 ARG2".

The arguments are retrieved by a script as ${2}, ${3}, etc.

  • Note: ${1} is reserved for the subject folder name, which is retrieved automatically.

  • Note: Do not use ~ in the path. Use ${HOME} instead.)

Default: ''

-email-to

Optional email address where sct_run_batch can send an alert on completion of the batch processing.

-email-from

Optional alternative email to use to send the email. Defaults to the same address as -email-to

-email-host

Optional smtp server and port to use to send the email. Defaults to gmail’s server. Note that gmail server requires ‘Less secure apps access’ to be turned on, which can be done at https://myaccount.google.com/security

Default: 'smtp.gmail.com:587'

-continue-on-error

Possible choices: 0, 1

Whether the batch processing should continue if a subject fails.

Default: 1

-script

Shell script used to process the data.

-zip

Create zip archive of output folders log/, qc/ and results/.

Default: False

-v

Possible choices: 0, 1, 2

Verbosity. 0: Display only errors/warnings, 1: Errors/warnings + info messages, 2: Debug mode

Default: 1