Detecting outliers in streaming time series data from ARM distributed sensors

Yuping Lu; Jitendra Kumar; Nathan Collier; Bhargavi Krishna; Michael A. Langston

doi:10.1109/ICDMW.2018.00117

Detecting outliers in streaming time series data from ARM distributed sensors

Yuping Lu, Jitendra Kumar, Nathan Collier, Bhargavi Krishna, Michael A. Langston

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

11 Scopus citations

Abstract

The Atmospheric Radiation Measurement (ARM) Data Center at ORNL collects data from a number of permanent and mobile facilities around the globe. The data is then ingested to create high level scientific products. High frequency streaming measurements from sensors and radar instruments at ARM sites require high degree of accuracy to enable rigorous study of atmospheric processes. Outliers in collected data are common due to instrument failure or extreme weather events. Thus, it is critical to identify and flag them. We employed multiple univariate, multivariate and time series techniques for outlier detection methods and studied their effectiveness. First, we examined Pearson correlation coefficient which is used to measure the pairwise correlations between variables. Singular Spectrum Analysis (SSA) was applied to detect outliers by removing the anticipated annual and seasonal cycles from the signal to accentuate anomalies. K-means was applied for multivariate examination of data from collection of sensors to identify any deviation from expected and known patterns and identify abnormal observation. The Pearson correlation coefficient, SSA and K-means methods were later combined together in a framework to detect outliers through a range of checks. We applied the developed method to data from meteorological sensors at ARM Southern Great Plains site and validated against existing database of known data quality issues.

Original language	English
Title of host publication	Proceedings - 18th IEEE International Conference on Data Mining Workshops, ICDMW 2018
Editors	Hanghang Tong, Zhenhui Li, Feida Zhu, Jeffrey Yu
Publisher	IEEE Computer Society
Pages	779-786
Number of pages	8
ISBN (Electronic)	9781538692882
DOIs	https://doi.org/10.1109/ICDMW.2018.00117
State	Published - Jul 2 2018
Event	18th IEEE International Conference on Data Mining Workshops, ICDMW 2018 - Singapore, Singapore Duration: Nov 17 2018 → Nov 20 2018

Publication series

Name	IEEE International Conference on Data Mining Workshops, ICDMW
Volume	2018-November
ISSN (Print)	2375-9232
ISSN (Electronic)	2375-9259

Conference

Conference	18th IEEE International Conference on Data Mining Workshops, ICDMW 2018
Country/Territory	Singapore
City	Singapore
Period	11/17/18 → 11/20/18

Funding

This research was supported by the Atmospheric Radiation Measurement (ARM) user facility, a U.S. Department of Energy (DOE) Office of Science user facility managed by the Office of Biological and EnvironmentalResearch. Oak Ridge National Laboratory (ORNL) is managed by UT-Battelle,LLC for the U.S. Department of Energy under Contract No. DE-AC05-00OR22725.This manuscript has been authored by UT- Battelle, LLC under Contract No. DE-AC05-00OR22725with the U.S. Department of Energy. The United States Government retains and the publisher, by accepting the article for publication, acknowledges that the United States Government retains a non-exclusive, paid-up, irrevocable, world-wide license to publish or reproduce the published form of this manuscript, or allow others to do so, for United States Government purposes. The Department of Energy will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan (http://energy.gov/downloads/ doe-public-access-plan).

Keywords

atmospheric science
clustering
outlier detection
time series

Access to Document

10.1109/ICDMW.2018.00117

Cite this

Lu, Y., Kumar, J., Collier, N., Krishna, B., & Langston, M. A. (2018). Detecting outliers in streaming time series data from ARM distributed sensors. In H. Tong, Z. Li, F. Zhu, & J. Yu (Eds.), Proceedings - 18th IEEE International Conference on Data Mining Workshops, ICDMW 2018 (pp. 779-786). Article 8637541 (IEEE International Conference on Data Mining Workshops, ICDMW; Vol. 2018-November). IEEE Computer Society. https://doi.org/10.1109/ICDMW.2018.00117

Lu, Yuping ; Kumar, Jitendra ; Collier, Nathan et al. / Detecting outliers in streaming time series data from ARM distributed sensors. Proceedings - 18th IEEE International Conference on Data Mining Workshops, ICDMW 2018. editor / Hanghang Tong ; Zhenhui Li ; Feida Zhu ; Jeffrey Yu. IEEE Computer Society, 2018. pp. 779-786 (IEEE International Conference on Data Mining Workshops, ICDMW).

@inproceedings{bd97518830e44beeb9ae09e33397fc90,

title = "Detecting outliers in streaming time series data from ARM distributed sensors",

abstract = "The Atmospheric Radiation Measurement (ARM) Data Center at ORNL collects data from a number of permanent and mobile facilities around the globe. The data is then ingested to create high level scientific products. High frequency streaming measurements from sensors and radar instruments at ARM sites require high degree of accuracy to enable rigorous study of atmospheric processes. Outliers in collected data are common due to instrument failure or extreme weather events. Thus, it is critical to identify and flag them. We employed multiple univariate, multivariate and time series techniques for outlier detection methods and studied their effectiveness. First, we examined Pearson correlation coefficient which is used to measure the pairwise correlations between variables. Singular Spectrum Analysis (SSA) was applied to detect outliers by removing the anticipated annual and seasonal cycles from the signal to accentuate anomalies. K-means was applied for multivariate examination of data from collection of sensors to identify any deviation from expected and known patterns and identify abnormal observation. The Pearson correlation coefficient, SSA and K-means methods were later combined together in a framework to detect outliers through a range of checks. We applied the developed method to data from meteorological sensors at ARM Southern Great Plains site and validated against existing database of known data quality issues.",

keywords = "atmospheric science, clustering, outlier detection, time series",

author = "Yuping Lu and Jitendra Kumar and Nathan Collier and Bhargavi Krishna and Langston, \{Michael A.\}",

note = "Publisher Copyright: {\textcopyright} 2018 IEEE.; 18th IEEE International Conference on Data Mining Workshops, ICDMW 2018 ; Conference date: 17-11-2018 Through 20-11-2018",

year = "2018",

month = jul,

day = "2",

doi = "10.1109/ICDMW.2018.00117",

language = "English",

series = "IEEE International Conference on Data Mining Workshops, ICDMW",

publisher = "IEEE Computer Society",

pages = "779--786",

editor = "Hanghang Tong and Zhenhui Li and Feida Zhu and Jeffrey Yu",

booktitle = "Proceedings - 18th IEEE International Conference on Data Mining Workshops, ICDMW 2018",

}

Lu, Y, Kumar, J , Collier, N , Krishna, B & Langston, MA 2018, Detecting outliers in streaming time series data from ARM distributed sensors. in H Tong, Z Li, F Zhu & J Yu (eds), Proceedings - 18th IEEE International Conference on Data Mining Workshops, ICDMW 2018., 8637541, IEEE International Conference on Data Mining Workshops, ICDMW, vol. 2018-November, IEEE Computer Society, pp. 779-786, 18th IEEE International Conference on Data Mining Workshops, ICDMW 2018, Singapore, Singapore, 11/17/18. https://doi.org/10.1109/ICDMW.2018.00117

Detecting outliers in streaming time series data from ARM distributed sensors. / Lu, Yuping; Kumar, Jitendra ; Collier, Nathan et al.
Proceedings - 18th IEEE International Conference on Data Mining Workshops, ICDMW 2018. ed. / Hanghang Tong; Zhenhui Li; Feida Zhu; Jeffrey Yu. IEEE Computer Society, 2018. p. 779-786 8637541 (IEEE International Conference on Data Mining Workshops, ICDMW; Vol. 2018-November).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Detecting outliers in streaming time series data from ARM distributed sensors

AU - Lu, Yuping

AU - Kumar, Jitendra

AU - Collier, Nathan

AU - Krishna, Bhargavi

AU - Langston, Michael A.

PY - 2018/7/2

Y1 - 2018/7/2

N2 - The Atmospheric Radiation Measurement (ARM) Data Center at ORNL collects data from a number of permanent and mobile facilities around the globe. The data is then ingested to create high level scientific products. High frequency streaming measurements from sensors and radar instruments at ARM sites require high degree of accuracy to enable rigorous study of atmospheric processes. Outliers in collected data are common due to instrument failure or extreme weather events. Thus, it is critical to identify and flag them. We employed multiple univariate, multivariate and time series techniques for outlier detection methods and studied their effectiveness. First, we examined Pearson correlation coefficient which is used to measure the pairwise correlations between variables. Singular Spectrum Analysis (SSA) was applied to detect outliers by removing the anticipated annual and seasonal cycles from the signal to accentuate anomalies. K-means was applied for multivariate examination of data from collection of sensors to identify any deviation from expected and known patterns and identify abnormal observation. The Pearson correlation coefficient, SSA and K-means methods were later combined together in a framework to detect outliers through a range of checks. We applied the developed method to data from meteorological sensors at ARM Southern Great Plains site and validated against existing database of known data quality issues.

AB - The Atmospheric Radiation Measurement (ARM) Data Center at ORNL collects data from a number of permanent and mobile facilities around the globe. The data is then ingested to create high level scientific products. High frequency streaming measurements from sensors and radar instruments at ARM sites require high degree of accuracy to enable rigorous study of atmospheric processes. Outliers in collected data are common due to instrument failure or extreme weather events. Thus, it is critical to identify and flag them. We employed multiple univariate, multivariate and time series techniques for outlier detection methods and studied their effectiveness. First, we examined Pearson correlation coefficient which is used to measure the pairwise correlations between variables. Singular Spectrum Analysis (SSA) was applied to detect outliers by removing the anticipated annual and seasonal cycles from the signal to accentuate anomalies. K-means was applied for multivariate examination of data from collection of sensors to identify any deviation from expected and known patterns and identify abnormal observation. The Pearson correlation coefficient, SSA and K-means methods were later combined together in a framework to detect outliers through a range of checks. We applied the developed method to data from meteorological sensors at ARM Southern Great Plains site and validated against existing database of known data quality issues.

KW - atmospheric science

KW - clustering

KW - outlier detection

KW - time series

UR - https://www.scopus.com/pages/publications/85062882901

U2 - 10.1109/ICDMW.2018.00117

DO - 10.1109/ICDMW.2018.00117

M3 - Conference contribution

AN - SCOPUS:85062882901

T3 - IEEE International Conference on Data Mining Workshops, ICDMW

SP - 779

EP - 786

BT - Proceedings - 18th IEEE International Conference on Data Mining Workshops, ICDMW 2018

A2 - Tong, Hanghang

A2 - Li, Zhenhui

A2 - Zhu, Feida

A2 - Yu, Jeffrey

PB - IEEE Computer Society

T2 - 18th IEEE International Conference on Data Mining Workshops, ICDMW 2018

Y2 - 17 November 2018 through 20 November 2018

ER -

Lu Y, Kumar J , Collier N , Krishna B, Langston MA. Detecting outliers in streaming time series data from ARM distributed sensors. In Tong H, Li Z, Zhu F, Yu J, editors, Proceedings - 18th IEEE International Conference on Data Mining Workshops, ICDMW 2018. IEEE Computer Society. 2018. p. 779-786. 8637541. (IEEE International Conference on Data Mining Workshops, ICDMW). doi: 10.1109/ICDMW.2018.00117

Detecting outliers in streaming time series data from ARM distributed sensors

Abstract

Publication series

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this