Skip to content

Data Storage Tutorial

Overview

This data storage tutorial guides you through:

It is intended for people new to requesting data storage volumes. It assumes that you have basic knowledge of:

  • Creating Projects in Research Toolkits

Before you start

Users can request SMB datacommons volumes in either DCC Research Groups or in Self-Service Projects in Research Toolkits. Users can only request NFS datacommons volumes in DCC Research Groups. Faculty can create Self-Service Projects on their own. If you do not know how to do this, just follow our Quickstart tutorial and then come back here! For DCC groups, please contact rescomputing@duke.edu for help.

Part 1: Requesting Storage

Step 1: View Existing Project in Research Toolkits

Visit Research Toolkits in your web browser; log in.

Click on one of your projects in the list to access the project page.

Step 2: Click "Add a Service"

Click on the vertical ellipsis and a dropdown list of options should appear. Select Add a Service > Storage

Step 3: Select the Type of Storage You Need

We offer both active and archival storage for your projects. Active storage includes volumes fit for data sets under analysis. Archival storage is intended for long-term, cold storage for data sets you need to store for long periods of time.

Step 3a: Active Storage

Select "Active" for storage type. Here, you'll have a number of choices to select to determine the right storage for you.

SMB Datacommons storage works well for general lab needs, but cannot be accessed through the DCC. This would best be used with any virtual machines created in Research Toolkits or as a flexible NAS option. Get this storage by selecting Active > Connect to Windows and/or Mac devices.

NFS Datacommons storage is not appropriate for data-under-analysis, but can be accessed through the DCC and works well as a storage location for data that will be analyzed in the near future. It can also function as a place to temporarily hold data received from collaborators. Get this storage by selecting Active > Connect to DCC/HPC > Occassional Compute.

NFS DCC (Standard) storage is mounted to DCC login nodes, DCC compute nodes, as well as to Globus. This storage is fit for data-under-analysis and can be useful to labs that have IRB protocls where data access needs to be controlled. Get this storage by selecting Active > Connect to DCC/HPC > Heavy Compute.

Below is a picture of the decision tree you should see if you were using Active Storage with heavy computational needs:

Note: You can add backups to this storage here by providing a fund code.

Step 3b: Archival Storage

Select "Archival" for storage type and a form should appear.

Step 4: Define your Storage Volume

  • Give your storage a name, adhering to the character restrictions for your chosen storage type
  • Leave the default retention date; you can change it later
  • Contribute some of your allocation to this volume. If you have other admins on your project, you can contribute some of their available allocation as well. Note, the minimum total size is 100GB.

You can add backups and increase the storage size of these volumes beyond that of the no-cost allocations by providing a fund code.

Step 5: Click "Confirm"

You will see a final review screen. Once you've confirmed everything looks correct, click on the "create" button. You should get a message that your storage volume is being created.

Refresh the page occassionally until it is ready.