Performance improvements on SNS and HFIR instrument data reduction workflows using Mantid

William F. Godoy; Peter F. Peterson; Steven E. Hahn; John Hetrick; Mathieu Doucet; Jay J. Billings

doi:10.1007/978-3-030-63393-6_12

Performance improvements on SNS and HFIR instrument data reduction workflows using Mantid

William F. Godoy, Peter F. Peterson, Steven E. Hahn, John Hetrick, Mathieu Doucet, Jay J. Billings

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

3 Scopus citations

Abstract

Performance of data reduction workflows at the High Flux Isotope Reactor (HFIR) and the Spallation Neutron Source (SNS) at Oak Ridge National Laboratory (ORNL) is mainly determined by the time spent loading raw measurement events stored in large and sparse datasets. This paper describes: (1) our long-term view to leverage SNS and HFIR data management needs with our experience at ORNL’s worldclass high performance computing (HPC) facilities, and (2) our shortterm efforts to speed up current workflows using Mantid, a data analysis and reduction community framework used across several neutron scattering facilities. We show that minimally invasive short-term improvements in metadata management have a moderate impact in speeding up current production workflows.We propose a more disruptive domain-specific solution: The No Cost Input Output (NCIO) framework, we provide an overview, the risks and challenges in NCIO’s adoption by HFIR and SNS stakeholders.

Original language	English
Title of host publication	Driving Scientific and Engineering Discoveries Through the Convergence of HPC, Big Data and AI - 17th Smoky Mountains Computational Sciences and Engineering Conference, SMC 2020, Revised Selected Papers
Editors	Jeffrey Nichols, Arthur ‘Barney’ Maccabe, Suzanne Parete-Koon, Becky Verastegui, Oscar Hernandez, Theresa Ahearn
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	175-186
Number of pages	12
ISBN (Print)	9783030633929
DOIs	https://doi.org/10.1007/978-3-030-63393-6_12
State	Published - 2021
Event	17th Smoky Mountains Computational Sciences and Engineering Conference, SMC 2020 - Virtual, Online Duration: Aug 26 2020 → Aug 28 2020

Publication series

Name	Communications in Computer and Information Science
Volume	1315 CCIS
ISSN (Print)	1865-0929
ISSN (Electronic)	1865-0937

Conference

Conference	17th Smoky Mountains Computational Sciences and Engineering Conference, SMC 2020
City	Virtual, Online
Period	08/26/20 → 08/28/20

Funding

A portion of this research used resources at the SNS, a Department of Energy (DOE) Office of Science User Facility operated by ORNL. ORNL is managed by UT-Battelle LLC for DOE under Contract DE-AC05-00OR22725. Acknowledgements. A portion of this research used resources at the SNS, a Department of Energy (DOE) Office of Science User Facility operated by ORNL. ORNL is managed by UT-Battelle LLC for DOE under Contract DE-AC05-00OR22725.

Funders	Funder number
U.S. Department of Energy	DE-AC05-00OR22725
Office of Science
Oak Ridge National Laboratory
UT-Battelle

Keywords

Data indexing
HFIR
Metadata
Neutron scattering
SNS

Access to Document

10.1007/978-3-030-63393-6_12

Cite this

Godoy, W. F., Peterson, P. F., Hahn, S. E., Hetrick, J., Doucet, M., & Billings, J. J. (2021). Performance improvements on SNS and HFIR instrument data reduction workflows using Mantid. In J. Nichols, A. B. Maccabe, S. Parete-Koon, B. Verastegui, O. Hernandez, & T. Ahearn (Eds.), Driving Scientific and Engineering Discoveries Through the Convergence of HPC, Big Data and AI - 17th Smoky Mountains Computational Sciences and Engineering Conference, SMC 2020, Revised Selected Papers (pp. 175-186). (Communications in Computer and Information Science; Vol. 1315 CCIS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-030-63393-6_12

Godoy, William F. ; Peterson, Peter F. ; Hahn, Steven E. et al. / Performance improvements on SNS and HFIR instrument data reduction workflows using Mantid. Driving Scientific and Engineering Discoveries Through the Convergence of HPC, Big Data and AI - 17th Smoky Mountains Computational Sciences and Engineering Conference, SMC 2020, Revised Selected Papers. editor / Jeffrey Nichols ; Arthur ‘Barney’ Maccabe ; Suzanne Parete-Koon ; Becky Verastegui ; Oscar Hernandez ; Theresa Ahearn. Springer Science and Business Media Deutschland GmbH, 2021. pp. 175-186 (Communications in Computer and Information Science).

@inproceedings{784a86d8a1d947c5b4161c2043ab63bd,

title = "Performance improvements on SNS and HFIR instrument data reduction workflows using Mantid",

abstract = "Performance of data reduction workflows at the High Flux Isotope Reactor (HFIR) and the Spallation Neutron Source (SNS) at Oak Ridge National Laboratory (ORNL) is mainly determined by the time spent loading raw measurement events stored in large and sparse datasets. This paper describes: (1) our long-term view to leverage SNS and HFIR data management needs with our experience at ORNL{\textquoteright}s worldclass high performance computing (HPC) facilities, and (2) our shortterm efforts to speed up current workflows using Mantid, a data analysis and reduction community framework used across several neutron scattering facilities. We show that minimally invasive short-term improvements in metadata management have a moderate impact in speeding up current production workflows.We propose a more disruptive domain-specific solution: The No Cost Input Output (NCIO) framework, we provide an overview, the risks and challenges in NCIO{\textquoteright}s adoption by HFIR and SNS stakeholders.",

keywords = "Data indexing, HFIR, Metadata, Neutron scattering, SNS",

author = "Godoy, {William F.} and Peterson, {Peter F.} and Hahn, {Steven E.} and John Hetrick and Mathieu Doucet and Billings, {Jay J.}",

note = "Publisher Copyright: {\textcopyright} Springer Nature Switzerland AG 2020.; 17th Smoky Mountains Computational Sciences and Engineering Conference, SMC 2020 ; Conference date: 26-08-2020 Through 28-08-2020",

year = "2021",

doi = "10.1007/978-3-030-63393-6_12",

language = "English",

isbn = "9783030633929",

series = "Communications in Computer and Information Science",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "175--186",

editor = "Jeffrey Nichols and Maccabe, {Arthur {\textquoteleft}Barney{\textquoteright}} and Suzanne Parete-Koon and Becky Verastegui and Oscar Hernandez and Theresa Ahearn",

booktitle = "Driving Scientific and Engineering Discoveries Through the Convergence of HPC, Big Data and AI - 17th Smoky Mountains Computational Sciences and Engineering Conference, SMC 2020, Revised Selected Papers",

}

Godoy, WF , Peterson, PF , Hahn, SE , Hetrick, J , Doucet, M & Billings, JJ 2021, Performance improvements on SNS and HFIR instrument data reduction workflows using Mantid. in J Nichols, AB Maccabe, S Parete-Koon, B Verastegui, O Hernandez & T Ahearn (eds), Driving Scientific and Engineering Discoveries Through the Convergence of HPC, Big Data and AI - 17th Smoky Mountains Computational Sciences and Engineering Conference, SMC 2020, Revised Selected Papers. Communications in Computer and Information Science, vol. 1315 CCIS, Springer Science and Business Media Deutschland GmbH, pp. 175-186, 17th Smoky Mountains Computational Sciences and Engineering Conference, SMC 2020, Virtual, Online, 08/26/20. https://doi.org/10.1007/978-3-030-63393-6_12

Performance improvements on SNS and HFIR instrument data reduction workflows using Mantid. / Godoy, William F.; Peterson, Peter F.; Hahn, Steven E. et al.
Driving Scientific and Engineering Discoveries Through the Convergence of HPC, Big Data and AI - 17th Smoky Mountains Computational Sciences and Engineering Conference, SMC 2020, Revised Selected Papers. ed. / Jeffrey Nichols; Arthur ‘Barney’ Maccabe; Suzanne Parete-Koon; Becky Verastegui; Oscar Hernandez; Theresa Ahearn. Springer Science and Business Media Deutschland GmbH, 2021. p. 175-186 (Communications in Computer and Information Science; Vol. 1315 CCIS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Performance improvements on SNS and HFIR instrument data reduction workflows using Mantid

AU - Godoy, William F.

AU - Peterson, Peter F.

AU - Hahn, Steven E.

AU - Hetrick, John

AU - Doucet, Mathieu

AU - Billings, Jay J.

N1 - Publisher Copyright: © Springer Nature Switzerland AG 2020.

PY - 2021

Y1 - 2021

N2 - Performance of data reduction workflows at the High Flux Isotope Reactor (HFIR) and the Spallation Neutron Source (SNS) at Oak Ridge National Laboratory (ORNL) is mainly determined by the time spent loading raw measurement events stored in large and sparse datasets. This paper describes: (1) our long-term view to leverage SNS and HFIR data management needs with our experience at ORNL’s worldclass high performance computing (HPC) facilities, and (2) our shortterm efforts to speed up current workflows using Mantid, a data analysis and reduction community framework used across several neutron scattering facilities. We show that minimally invasive short-term improvements in metadata management have a moderate impact in speeding up current production workflows.We propose a more disruptive domain-specific solution: The No Cost Input Output (NCIO) framework, we provide an overview, the risks and challenges in NCIO’s adoption by HFIR and SNS stakeholders.

AB - Performance of data reduction workflows at the High Flux Isotope Reactor (HFIR) and the Spallation Neutron Source (SNS) at Oak Ridge National Laboratory (ORNL) is mainly determined by the time spent loading raw measurement events stored in large and sparse datasets. This paper describes: (1) our long-term view to leverage SNS and HFIR data management needs with our experience at ORNL’s worldclass high performance computing (HPC) facilities, and (2) our shortterm efforts to speed up current workflows using Mantid, a data analysis and reduction community framework used across several neutron scattering facilities. We show that minimally invasive short-term improvements in metadata management have a moderate impact in speeding up current production workflows.We propose a more disruptive domain-specific solution: The No Cost Input Output (NCIO) framework, we provide an overview, the risks and challenges in NCIO’s adoption by HFIR and SNS stakeholders.

KW - Data indexing

KW - HFIR

KW - Metadata

KW - Neutron scattering

KW - SNS

UR - http://www.scopus.com/inward/record.url?scp=85107308644&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-63393-6_12

DO - 10.1007/978-3-030-63393-6_12

M3 - Conference contribution

AN - SCOPUS:85107308644

SN - 9783030633929

T3 - Communications in Computer and Information Science

SP - 175

EP - 186

BT - Driving Scientific and Engineering Discoveries Through the Convergence of HPC, Big Data and AI - 17th Smoky Mountains Computational Sciences and Engineering Conference, SMC 2020, Revised Selected Papers

A2 - Nichols, Jeffrey

A2 - Maccabe, Arthur ‘Barney’

A2 - Parete-Koon, Suzanne

A2 - Verastegui, Becky

A2 - Hernandez, Oscar

A2 - Ahearn, Theresa

PB - Springer Science and Business Media Deutschland GmbH

T2 - 17th Smoky Mountains Computational Sciences and Engineering Conference, SMC 2020

Y2 - 26 August 2020 through 28 August 2020

ER -

Godoy WF , Peterson PF , Hahn SE , Hetrick J , Doucet M, Billings JJ. Performance improvements on SNS and HFIR instrument data reduction workflows using Mantid. In Nichols J, Maccabe AB, Parete-Koon S, Verastegui B, Hernandez O, Ahearn T, editors, Driving Scientific and Engineering Discoveries Through the Convergence of HPC, Big Data and AI - 17th Smoky Mountains Computational Sciences and Engineering Conference, SMC 2020, Revised Selected Papers. Springer Science and Business Media Deutschland GmbH. 2021. p. 175-186. (Communications in Computer and Information Science). doi: 10.1007/978-3-030-63393-6_12

Performance improvements on SNS and HFIR instrument data reduction workflows using Mantid

Abstract

Publication series

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this