Efficient Data Management in Neutron Scattering Data Reduction Workflows at ORNL

William F. Godoy; Peter F. Peterson; Steven E. Hahn; Jay J. Billings

doi:10.1109/BigData50022.2020.9377836

Efficient Data Management in Neutron Scattering Data Reduction Workflows at ORNL

William F. Godoy, Peter F. Peterson, Steven E. Hahn, Jay J. Billings

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

6 Scopus citations

Abstract

Oak Ridge National Laboratory (ORNL) experimental neutron science facilities produce 1.2 TB a day of raw event-based data that is stored using the standard metadata-rich NeXus schema built on top of the HDF5 file format. Performance of several data reduction workflows is largely determined by the amount of time spent on the loading and processing algorithms in Mantid, an open-source data analysis framework used across several neutron sciences facilities around the world. The present work introduces new data management algorithms to address identified input output (I/O) bottlenecks on Mantid. First, we introduce an in-memory binary-tree metadata index that resemble NeXus data access patterns to provide a scalable search and extraction mechanism. Second, data encapsulation in Mantid algorithms is optimally redesigned to reduce the total compute and memory runtime footprint associated with metadata I/O reconstruction tasks. Results from this work show speed ups in wall-clock time on ORNL data reduction workflows, ranging from 11% to 30% depending on the complexity of the targeted instrument-specific data. Nevertheless, we highlight the need for more research to address reduction challenges as experimental data volumes increase.

Original language	English
Title of host publication	Proceedings - 2020 IEEE International Conference on Big Data, Big Data 2020
Editors	Xintao Wu, Chris Jermaine, Li Xiong, Xiaohua Tony Hu, Olivera Kotevska, Siyuan Lu, Weijia Xu, Srinivas Aluru, Chengxiang Zhai, Eyhab Al-Masri, Zhiyuan Chen, Jeff Saltz
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	2674-2680
Number of pages	7
ISBN (Electronic)	9781728162515
DOIs	https://doi.org/10.1109/BigData50022.2020.9377836
State	Published - Dec 10 2020
Event	8th IEEE International Conference on Big Data, Big Data 2020 - Virtual, Atlanta, United States Duration: Dec 10 2020 → Dec 13 2020

Publication series

Name	Proceedings - 2020 IEEE International Conference on Big Data, Big Data 2020

Conference

Conference	8th IEEE International Conference on Big Data, Big Data 2020
Country/Territory	United States
City	Virtual, Atlanta
Period	12/10/20 → 12/13/20

Funding

This manuscript has been authored by UT-Battelle, LLC under Contract No. DEAC05-00OR22725 with the U.S. Department of Energy. The United States Government retains and the publisher, by accepting the article for publication, acknowledges that the United States Government retains a nonexclusive, paid-up, irrevocable, world-wide license to publish or reproduce the published form of this manuscript, or allow others to do so, for United States Government purposes. The Department of Energy will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan (http://energy.gov/downloads/ doe-public-access-plan). ACKNOWLEDGMENT Work at Oak Ridge National Laboratory was sponsored by the Division of Scientific User Facilities, Office of Basic Energy Sciences, US Department of Energy, under Contract no. DE-AC05-00OR22725 with UT-Battelle, LLC. We would like thank Dr. Mathieu Doucet, Dr James Kohl, and Mr. Rich Crompton of the Neutron Sciences Division at Oak Ridge National Laboratory for their helpful input to this work.

Keywords

HDF5
Mantid
NeXus
data management
experimental data
indexing
metadata
neutron scattering
reduction workflows

Access to Document

10.1109/BigData50022.2020.9377836

Cite this

Godoy, W. F., Peterson, P. F., Hahn, S. E., & Billings, J. J. (2020). Efficient Data Management in Neutron Scattering Data Reduction Workflows at ORNL. In X. Wu, C. Jermaine, L. Xiong, X. T. Hu, O. Kotevska, S. Lu, W. Xu, S. Aluru, C. Zhai, E. Al-Masri, Z. Chen, & J. Saltz (Eds.), Proceedings - 2020 IEEE International Conference on Big Data, Big Data 2020 (pp. 2674-2680). Article 9377836 (Proceedings - 2020 IEEE International Conference on Big Data, Big Data 2020). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/BigData50022.2020.9377836

Godoy, William F. ; Peterson, Peter F. ; Hahn, Steven E. et al. / Efficient Data Management in Neutron Scattering Data Reduction Workflows at ORNL. Proceedings - 2020 IEEE International Conference on Big Data, Big Data 2020. editor / Xintao Wu ; Chris Jermaine ; Li Xiong ; Xiaohua Tony Hu ; Olivera Kotevska ; Siyuan Lu ; Weijia Xu ; Srinivas Aluru ; Chengxiang Zhai ; Eyhab Al-Masri ; Zhiyuan Chen ; Jeff Saltz. Institute of Electrical and Electronics Engineers Inc., 2020. pp. 2674-2680 (Proceedings - 2020 IEEE International Conference on Big Data, Big Data 2020).

@inproceedings{e6a4b2457ce74309af049b0e18d9f719,

title = "Efficient Data Management in Neutron Scattering Data Reduction Workflows at ORNL",

abstract = "Oak Ridge National Laboratory (ORNL) experimental neutron science facilities produce 1.2 TB a day of raw event-based data that is stored using the standard metadata-rich NeXus schema built on top of the HDF5 file format. Performance of several data reduction workflows is largely determined by the amount of time spent on the loading and processing algorithms in Mantid, an open-source data analysis framework used across several neutron sciences facilities around the world. The present work introduces new data management algorithms to address identified input output (I/O) bottlenecks on Mantid. First, we introduce an in-memory binary-tree metadata index that resemble NeXus data access patterns to provide a scalable search and extraction mechanism. Second, data encapsulation in Mantid algorithms is optimally redesigned to reduce the total compute and memory runtime footprint associated with metadata I/O reconstruction tasks. Results from this work show speed ups in wall-clock time on ORNL data reduction workflows, ranging from 11% to 30% depending on the complexity of the targeted instrument-specific data. Nevertheless, we highlight the need for more research to address reduction challenges as experimental data volumes increase.",

keywords = "HDF5, Mantid, NeXus, data management, experimental data, indexing, metadata, neutron scattering, reduction workflows",

author = "Godoy, {William F.} and Peterson, {Peter F.} and Hahn, {Steven E.} and Billings, {Jay J.}",

note = "Publisher Copyright: {\textcopyright} 2020 IEEE.; 8th IEEE International Conference on Big Data, Big Data 2020 ; Conference date: 10-12-2020 Through 13-12-2020",

year = "2020",

month = dec,

day = "10",

doi = "10.1109/BigData50022.2020.9377836",

language = "English",

series = "Proceedings - 2020 IEEE International Conference on Big Data, Big Data 2020",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "2674--2680",

editor = "Xintao Wu and Chris Jermaine and Li Xiong and Hu, {Xiaohua Tony} and Olivera Kotevska and Siyuan Lu and Weijia Xu and Srinivas Aluru and Chengxiang Zhai and Eyhab Al-Masri and Zhiyuan Chen and Jeff Saltz",

booktitle = "Proceedings - 2020 IEEE International Conference on Big Data, Big Data 2020",

}

Godoy, WF , Peterson, PF , Hahn, SE & Billings, JJ 2020, Efficient Data Management in Neutron Scattering Data Reduction Workflows at ORNL. in X Wu, C Jermaine, L Xiong, XT Hu, O Kotevska, S Lu, W Xu, S Aluru, C Zhai, E Al-Masri, Z Chen & J Saltz (eds), Proceedings - 2020 IEEE International Conference on Big Data, Big Data 2020., 9377836, Proceedings - 2020 IEEE International Conference on Big Data, Big Data 2020, Institute of Electrical and Electronics Engineers Inc., pp. 2674-2680, 8th IEEE International Conference on Big Data, Big Data 2020, Virtual, Atlanta, United States, 12/10/20. https://doi.org/10.1109/BigData50022.2020.9377836

Efficient Data Management in Neutron Scattering Data Reduction Workflows at ORNL. / Godoy, William F.; Peterson, Peter F.; Hahn, Steven E. et al.
Proceedings - 2020 IEEE International Conference on Big Data, Big Data 2020. ed. / Xintao Wu; Chris Jermaine; Li Xiong; Xiaohua Tony Hu; Olivera Kotevska; Siyuan Lu; Weijia Xu; Srinivas Aluru; Chengxiang Zhai; Eyhab Al-Masri; Zhiyuan Chen; Jeff Saltz. Institute of Electrical and Electronics Engineers Inc., 2020. p. 2674-2680 9377836 (Proceedings - 2020 IEEE International Conference on Big Data, Big Data 2020).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Efficient Data Management in Neutron Scattering Data Reduction Workflows at ORNL

AU - Godoy, William F.

AU - Peterson, Peter F.

AU - Hahn, Steven E.

AU - Billings, Jay J.

PY - 2020/12/10

Y1 - 2020/12/10

N2 - Oak Ridge National Laboratory (ORNL) experimental neutron science facilities produce 1.2 TB a day of raw event-based data that is stored using the standard metadata-rich NeXus schema built on top of the HDF5 file format. Performance of several data reduction workflows is largely determined by the amount of time spent on the loading and processing algorithms in Mantid, an open-source data analysis framework used across several neutron sciences facilities around the world. The present work introduces new data management algorithms to address identified input output (I/O) bottlenecks on Mantid. First, we introduce an in-memory binary-tree metadata index that resemble NeXus data access patterns to provide a scalable search and extraction mechanism. Second, data encapsulation in Mantid algorithms is optimally redesigned to reduce the total compute and memory runtime footprint associated with metadata I/O reconstruction tasks. Results from this work show speed ups in wall-clock time on ORNL data reduction workflows, ranging from 11% to 30% depending on the complexity of the targeted instrument-specific data. Nevertheless, we highlight the need for more research to address reduction challenges as experimental data volumes increase.

AB - Oak Ridge National Laboratory (ORNL) experimental neutron science facilities produce 1.2 TB a day of raw event-based data that is stored using the standard metadata-rich NeXus schema built on top of the HDF5 file format. Performance of several data reduction workflows is largely determined by the amount of time spent on the loading and processing algorithms in Mantid, an open-source data analysis framework used across several neutron sciences facilities around the world. The present work introduces new data management algorithms to address identified input output (I/O) bottlenecks on Mantid. First, we introduce an in-memory binary-tree metadata index that resemble NeXus data access patterns to provide a scalable search and extraction mechanism. Second, data encapsulation in Mantid algorithms is optimally redesigned to reduce the total compute and memory runtime footprint associated with metadata I/O reconstruction tasks. Results from this work show speed ups in wall-clock time on ORNL data reduction workflows, ranging from 11% to 30% depending on the complexity of the targeted instrument-specific data. Nevertheless, we highlight the need for more research to address reduction challenges as experimental data volumes increase.

KW - HDF5

KW - Mantid

KW - NeXus

KW - data management

KW - experimental data

KW - indexing

KW - metadata

KW - neutron scattering

KW - reduction workflows

UR - http://www.scopus.com/inward/record.url?scp=85103826166&partnerID=8YFLogxK

U2 - 10.1109/BigData50022.2020.9377836

DO - 10.1109/BigData50022.2020.9377836

M3 - Conference contribution

AN - SCOPUS:85103826166

T3 - Proceedings - 2020 IEEE International Conference on Big Data, Big Data 2020

SP - 2674

EP - 2680

BT - Proceedings - 2020 IEEE International Conference on Big Data, Big Data 2020

A2 - Wu, Xintao

A2 - Jermaine, Chris

A2 - Xiong, Li

A2 - Hu, Xiaohua Tony

A2 - Kotevska, Olivera

A2 - Lu, Siyuan

A2 - Xu, Weijia

A2 - Aluru, Srinivas

A2 - Zhai, Chengxiang

A2 - Al-Masri, Eyhab

A2 - Chen, Zhiyuan

A2 - Saltz, Jeff

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 8th IEEE International Conference on Big Data, Big Data 2020

Y2 - 10 December 2020 through 13 December 2020

ER -

Godoy WF , Peterson PF , Hahn SE, Billings JJ. Efficient Data Management in Neutron Scattering Data Reduction Workflows at ORNL. In Wu X, Jermaine C, Xiong L, Hu XT, Kotevska O, Lu S, Xu W, Aluru S, Zhai C, Al-Masri E, Chen Z, Saltz J, editors, Proceedings - 2020 IEEE International Conference on Big Data, Big Data 2020. Institute of Electrical and Electronics Engineers Inc. 2020. p. 2674-2680. 9377836. (Proceedings - 2020 IEEE International Conference on Big Data, Big Data 2020). doi: 10.1109/BigData50022.2020.9377836

Efficient Data Management in Neutron Scattering Data Reduction Workflows at ORNL

Abstract

Publication series

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this