Abstract
Big data transfer in next-generation scientific applications is now commonly carried out over dedicated channels in high-performance networks (HPNs), where transport protocols play a critical role in maximizing application-level throughput. Optimizing the performance of these protocols is challenging: i) transport protocols perform differently in various network environments, and the protocol choice is not straightforward; ii) even for a given protocol in a given environment, different parameter settings of the protocol may lead to significantly different performance and oftentimes the default setting does not yield the best performance. However, it is prohibitively time-consuming to conduct exhaustive transport profiling due to the large parameter space. In this paper, we propose a PRofiling Optimization Based DAta Transfer Advisor (ProbData) to help end users determine the most effective transport method with the most appropriate parameter settings to achieve satisfactory performance for big data transfer over dedicated connections in HPNs. ProbData employs a fast profiling scheme based on the Simultaneous Perturbation Stochastic Approximation algorithm, namely, FastProf, to accelerate the exploration of the optimal operational zones of various transport methods to improve profiling efficiency. We first present a theoretical background of the optimized profiling approach in ProbData and then detail its design and implementation. The advising procedure and performance benefits of FastProf and ProbData are illustrated and evaluated by both extensive emulations based on real-life performance measurements and experiments over various physical connections in existing production HPNs.
| Original language | English |
|---|---|
| Article number | 8867859 |
| Pages (from-to) | 2280-2293 |
| Number of pages | 14 |
| Journal | IEEE/ACM Transactions on Networking |
| Volume | 27 |
| Issue number | 6 |
| DOIs | |
| State | Published - Dec 2019 |
Funding
Manuscript received September 5, 2018; revised April 29, 2019 and July 8, 2019; accepted September 17, 2019; approved by IEEE/ACM TRANS-ACTIONS ON NETWORKING Editor J. Liu. Date of publication October 14, 2019; date of current version December 17, 2019. This work was supported in part by the Department of Energy’s Office of Science under Grant DESC0015892 and Contract DE-AC05-00OR22725 at Oak Ridge National Laboratory managed by UT-Battelle, LLC, in part by the Harrisburg University’s Presidential Research Grant, and in part by the National Science Foundation with New Jersey Institute of Technology under Grant CNS-1828123. Some preliminary results of this article were presented at IEEE ICCCN 2016 [19] and IEEE LCN 2017 [20]. (Corresponding author: Chase Q. Wu.) D. Yun is with the Computer and Information Sciences Program, Harrisburg University, Harrisburg, PA 17101 USA (e-mail: [email protected]).
Keywords
- Profiling optimization
- big data transfer
- high-performance networks
- stochastic approximation