Purpose

The following tables provide guidelines for data management in OSG. Check the web links for additional details.

Data storage options

Default Limit Purpose Network mounted Backed Up Quota Purge Details
home < 20 GB Meant for quick data access and not for submitting jobs. Yes Yes 20 GB No Data Storage
local-scratch < 25 GB Meant for large temporary storage and I/O for your jobs. Files older than 30 days are automatically removed. No No No Yes, 30 days Data Storage
stash < 200 GB Meant for large storage and I/O for your jobs. Yes No No no Data Storage
public < 10 GB Meant for sharing data and transfer input data via HTTP or staschcp Yes No No No Data Storage

For the data management of more than 200 GB, please email us at user-support@opensciencegrid.org

Transferring input data for a job

Recommended Data Size Command Purpose Details
HTCondor < 100 MB transfer_input_files Input data from home, public or stash HTCondor Transfer
HTTP < 1 GB wget, curl or rsync Input data from ~/public HTTP Access
StachCache > 1 GB, < 50 GB Staschcp Input data from ~/public StachCache
GridFTP > 1 GB, < 50 GB gfal-copy Input data from ~/stash Experts with large work flows. Please contact us if you want to use this.

Transferring output data for a job

Recommended Data Size Command Purpose Details
HTCondor < 100 MB transfer_output_files Transfer data to submit directory HTCondor Transfer
UNIX tools < 1 GB rsync, scp, etc. Transfer data to home, local-scratch, stash, etc. Please contact us if you want to use this.
GridFTP > 1 GB, < 50 GB gfal-copy Output data to ~/stash Experts with large work flows. Please contact us if you want to use this.

External data transfer

Data Size Tools Details
UNIX tools < 1 GB rsync, scp, puty, WinSCP, gFTP, etc. SCP
Globus > 1 GB globus webservice or globus CLI Globus

Getting Help

For assistance or questions, please email the OSG User Support team at user-support@opensciencegrid.org or visit the help desk and community forums.

 

This page was updated on Dec 16, 2018 at 19:00 from data/overview-data-table.md.