Skip to content

Using the Download Partition

Compute nodes on SuperCloud do not have internet access.

This means that if you need to check out git repositories, download packages or download small data sets, you will need to do this on the login node before you start or launch your job. The login nodes have internet access for pip installs, git repo checkouts, etc.

To accommodate large data downloads, we have added a separate partition named "download" with data transfer nodes. This node is schedulable through this new download partition.

You can request use of this partition by requesting it with your job submission. If you are using sbatch or sbatch flags in your job script you can use the -p or --partition flag to specify the download partition:

  • --partition=download
  • -p download

If you are using LLsub the flag to select a partition is -q:

  • -q download

This enables you to launch a data download job before your job starts or after your job completes in order to sync whatever data you need.

Examples

Interactive Job with LLsub

LLsub -i -q download  

Job Submission Script

my_download_job.sh
#!/bin/bash

#SBATCH -p download
#SBATCH -o my_download.out-%j

wget URL

where URL is the URL to the file you are trying to download. You would launch this job with either:

sbatch my_download_job.sh

or

LLsub my_downlod_job.sh

Monitoring your download job

You can execute the LLstat command at the Linux command line to see if your download job is running. We recommend checking the log file to check for errors and to track the progress of your download.