TY - GEN
T1 - High performance threaded data streaming for large scale simulations
AU - Bhat, Viraj
AU - Klasky, Scott
AU - Atchley, Scott
AU - Beck, Micah
AU - McCune, Doug
AU - Parashar, Manish
PY - 2004
Y1 - 2004
N2 - We have developed a threaded parallel data streaming approach using Logistical Networking (LN) to transfer multi-terabyte simulation data from computers at NERSC to our local analysis/visualization cluster, as the simulation executes, with negligible overhead. Data transfer experiments show that this concurrent data transfer approach is more favorable compared with writing to local disk and later transferring this data to be post-processed. Our algorithms are network aware, and can stream data at up to 97Mbs on a 100Mbs link from CA to NJ during a live simulation, using less than 5% CPU overhead at NERSC. This method is the first step in setting up a pipeline for simulation workflow and data management.
AB - We have developed a threaded parallel data streaming approach using Logistical Networking (LN) to transfer multi-terabyte simulation data from computers at NERSC to our local analysis/visualization cluster, as the simulation executes, with negligible overhead. Data transfer experiments show that this concurrent data transfer approach is more favorable compared with writing to local disk and later transferring this data to be post-processed. Our algorithms are network aware, and can stream data at up to 97Mbs on a 100Mbs link from CA to NJ during a live simulation, using less than 5% CPU overhead at NERSC. This method is the first step in setting up a pipeline for simulation workflow and data management.
UR - http://www.scopus.com/inward/record.url?scp=19944368778&partnerID=8YFLogxK
U2 - 10.1109/GRID.2004.36
DO - 10.1109/GRID.2004.36
M3 - Conference contribution
AN - SCOPUS:19944368778
SN - 0769522564
T3 - Proceedings - IEEE/ACM International Workshop on Grid Computing
SP - 243
EP - 250
BT - Proceedings - Fifth IEEE/ACM International Workshop on Grid Computing
A2 - Buyya, R.
T2 - Proceedings - Fifth IEEE/ACM International Workshop on Grid Computing
Y2 - 8 November 2004 through 8 November 2004
ER -