A simple example of Slurm Job Arrays

What are Slurm Job Arrays?

Using job arrays you can submit collections of similar jobs very easily to your Slurm cluster. Since the official documentation is very hard to understand I will provide an example with explenations on how to use them.

The Example

In my example I will try to process 20 files named input1.txt to input20.txt using my shell script processing.sh. I suggest to rename your input files to a format like <name><ctr>.<file extension>. Using this format the following steps are very easy. Don’t forget to write a note about the renaming.

To use job arrays you need a sbatch script like the following:

#!/path/to/sbatch

# share node
#SBATCH --share

# set max wallclock time
#SBATCH --time=2-00:00:00

# set name of job
#SBATCH --job-name=example_for_job_arrays

# array creation
#SBATCH --array=0-20%3

echo "Input file:" input$((SLURM_ARRAY_TASK_ID)).txt
input_file=input$((SLURM_ARRAY_TASK_ID)).txt

# Run script
bash processing.sh $input_file

Explanation

As you can see in the example sbatch script I set some sbatch options for my jobs. While I only used share, time and job-name ypu can of course use anything else too. These options are used as the options for every job in the job array. The array option 0-20%3 tells slurm to start jobs with indices from 0 to 20. The integer after the ‘%’ seperator tells slurm to run only 3 jobs at the same time. Of course you can use any other number or leave this out to run all jobs at the same time.

Since this script creates the environment variable SLURM_ARRAY_TASK_ID which is the job index from the array option we can use this to set the input file for the job array.

Further Information

This very short example of the job array usage does not help everybody since it does not cover all options. If you want to use all features of job arrays have a look at the official documentation. You might also have a look at some more websites like the following: University of Chicago, New York University.

Please contact me if you have suggestions for helpful websites, so I can add them to this list.