Archive migration through workflow automation

Norbert Podhorszki, Bertram Ludäscher, Scott Klasky

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

1 Scopus citations

Abstract

The Center for Plasma Edge Simulation project aims to automate the tedious tasks of simulation monitoring, data archival and coupling simulation codes using the Kepler scientific workflow environment. The technology has been successfully applied for migrating a combustion data archive of 10TB from NERSC to ORNL, where there were no other automated solutions for this task. This paper describes the workflow that migrates large files from mass storage systems using external tools and temporary staging to disks, performing different stages in a pipeline-parallel fashion, parallelizing file transfers and doing special checkpointing to make the workflow restartable and also perform operations that failed earlier. The advantage of creating/using such a workflow over specialized data migration services is its independence from specific systems so it can be used by configuring the external tools to be used. The advantage over scripts is the robust exection (handling failures and timeouts) and efficiency (parallelization wherever possible).

Original languageEnglish
Title of host publicationProceedings of the 19th IASTED International Conference on Parallel and Distributed Computing and Systems
Pages282-287
Number of pages6
StatePublished - 2007
Event19th IASTED International Conference on Parallel and Distributed Computing and Systems - Cambridge, MA, United States
Duration: Nov 19 2007Nov 21 2007

Publication series

NameProceedings of the IASTED International Conference on Parallel and Distributed Computing and Systems
ISSN (Print)1027-2658

Conference

Conference19th IASTED International Conference on Parallel and Distributed Computing and Systems
Country/TerritoryUnited States
CityCambridge, MA
Period11/19/0711/21/07

Keywords

  • Data transfer
  • Distributed application
  • Scientific workflow
  • Software tools

Fingerprint

Dive into the research topics of 'Archive migration through workflow automation'. Together they form a unique fingerprint.

Cite this