Storing your Data

Chances are you need to run your HPC job against a dataset, perhaps quite a sizable one. There are a number of places to store data on Spartan while you're working with it, and ways to get data in and out.

Not for Long-Term Storage

While it's often essential to have fast nearby storage while working on your data, take care that you don't use Spartan as your long-term data repository. It's not designed for that, and may not conform to the requirements set by your institution or funding body.

VicNode offers a range of data storage services that may suit your needs. If you're unsure, get in contact with us.

Where to Store Your Data on Spartan

Projects Directory

Your projects directory is the best place to store research data while you're working on it. It's located at /data/projects/myprojectname.

Others in your project can access it, and 500 GB of storage is available per project. If you need more than this, get in touch and we'll find a solution for you.

Home Directory

Your home directory, i.e. /home/yourusername can be used to store small amounts of data, however this is generally discouraged. It's best suited to short-lived and non-critical data, for example while working through our getting started tutorial or testing out new software.

Others in your project won't have access, and you're limited to ??GB of storage.

Scratch Space

You can store temporary working data while your job is running at /scratch/. This is handy if your job generates large files while it's running that you don't need to keep. Scratch space is limited to ??GB.

How to Transfer Data In and Out of Spartan

Secure Copy (scp)

You can use the scp command to move data from your local machine to Spartan. For example, to move mydata.dat from your current working directory to your project directory on Spartan:

# scp local.dat [email protected]:/data/projects/myproject/remote.dat

You can transfer files from Spartan to your local machine by reversing the order of the arguments like so:

# scp [email protected]:/data/projects/myproject/remote.dat local.dat

For Windows users, PuTTY provides an equivalent tool called pscp. If you're data is located on a remote machine, you can SSH into that system first, and then use scp from that machine to transfer your data into Spartan.

If you'd prefer a GUI interface rather than the command line, you can use tools like FileZilla (cross-platform) or CyberDuck (OS X & Windows).

rsync

Repeatedly transferring large files in and out of Spartan via scp can be tedious. A good alternative is rsync, which only transfers the parts that have changed. It can work on single files, or whole directories, and the syntax is much same as for scp.

# rsync local.dat [email protected]:/data/projects/myproject/remote.dat

Note that the first argument is the source, and the second is the destination which will be modified to match the source.

VicNode

VicNode provides data storage that can be mounted on Spartan. It's then very easy to transfer files back and fourth as needed. Contact us with details of the VicNode resources you'd like to mount, and we can arrange this for you.