Purpose

The following tables provide guidelines for data transfer and access in OSG. Check the web links for additional details for your particular use-case. For datasets larger than 200 GB, please email us at user-support@opensciencegrid.org.

Data storage options

Default Limit Purpose Network mounted Backed Up Quota Purge Details
home < 20 GB Meant for quick data access and not for submitting jobs. Yes Yes 20 GB No Data Storage
local-scratch < 25 GB Meant for large temporary storage and I/O for your jobs. Files older than 30 days are automatically removed. No No No Yes, 30 days Data Storage
stash < 200 GB Meant for large storage and I/O for your jobs. Yes No No no Data Storage
public < 10 GB Meant for sharing data and transfer input data via HTTP or staschcp Yes No No No Data Storage

Transferring input data for a job

Recommended Data Size Command Purpose Details
HTCondor < 100 MB transfer_input_files Input data from home, public or stash HTCondor Transfer
HTTP < 1 GB wget, curl or rsync Input data from ~/public HTTP Access
StashCache > 1 GB, < 50 GB stashcp Input data from ~/public StashCache
GridFTP > 1 GB, < 50 GB gfal-copy Input data from ~/stash Experts with large work flows. Please contact us if you want to use this.

Transferring output data for a job

Recommended Data Size Command Purpose Details
HTCondor < 100 MB transfer_output_files Transfer data to submit directory HTCondor Transfer
UNIX tools < 1 GB rsync, scp, etc. Transfer data to home, local-scratch, stash, etc. Please contact us if you want to use this.
GridFTP > 1 GB, < 50 GB gfal-copy Output data to ~/stash Experts with large work flows. Please contact us if you want to use this.

External data transfer

Data Size Tools Details
UNIX tools < 1 GB rsync, scp, putty, WinSCP, gFTP, etc. SCP
Globus > 1 GB globus webservice or globus CLI Globus

Getting Help

For assistance or questions, please email the OSG User Support team at user-support@opensciencegrid.org or visit the help desk and community forums.

 

This page was updated on Feb 23, 2019 at 01:00 from guide/data/overview-data-table.md.