Integrated Clustering and Anomaly Detection (INCAD) for Streaming Data

Sreelekha Guggilam; Syed Mohammed Arshad Zaidi; Varun Chandola; Abani K. Patra

doi:10.1007/978-3-030-22747-0_4

Integrated Clustering and Anomaly Detection (INCAD) for Streaming Data

Sreelekha Guggilam, Syed Mohammed Arshad Zaidi, Varun Chandola, Abani K. Patra

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

3 Scopus citations

Abstract

Most current clustering based anomaly detection methods use scoring schema and thresholds to classify anomalies. These methods are often tailored to target specific data sets with “known” number of clusters. The paper provides a streaming clustering and anomaly detection algorithm that does not require strict arbitrary thresholds on the anomaly scores or knowledge of the number of clusters while performing probabilistic anomaly detection and clustering simultaneously. This ensures that the cluster formation is not impacted by the presence of anomalous data, thereby leading to more reliable definition of “normal vs abnormal” behavior. The motivations behind developing the INCAD model [17] and the path that leads to the streaming model are discussed.

Original language	English
Title of host publication	Computational Science – ICCS 2019 - 19th International Conference, Proceedings
Editors	João M.F. Rodrigues, Pedro J.S. Cardoso, Jânio Monteiro, Roberto Lam, Valeria V. Krzhizhanovskaya, Michael H. Lees, Peter M.A. Sloot, Jack J. Dongarra
Publisher	Springer Verlag
Pages	45-59
Number of pages	15
ISBN (Print)	9783030227463
DOIs	https://doi.org/10.1007/978-3-030-22747-0_4
State	Published - 2019
Externally published	Yes
Event	19th International Conference on Computational Science, ICCS 2019 - Faro, Portugal Duration: Jun 12 2019 → Jun 14 2019

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	11539 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	19th International Conference on Computational Science, ICCS 2019
Country/Territory	Portugal
City	Faro
Period	06/12/19 → 06/14/19

Funding

Acknowledgements. The authors would like to acknowledge University at Buffalo Center for Computational Research (http://www.buffalo.edu/ccr.html) for its computing resources that were made available for conducting the research reported in this paper. Financial support of the National Science Foundation Grant numbers NSF/OAC 1339765 and NSF/DMS 1621853 is acknowledged. The authors would like to acknowledge University at Buffalo Center for Computational Research (http://www.buffalo.edu/ccr.html) for its computing resources that were made available for conducting the research reported in this paper. Financial support of the National Science Foundation Grant numbers NSF/OAC 1339765 and NSF/DMS 1621853 is acknowledged.

Funders	Funder number
NSF/DMS	1621853
NSF/OAC
National Science Foundation
Directorate for Computer and Information Science and Engineering	1339765
University at Buffalo

Keywords

Anomaly detection
Bayesian non-parametric models
Clustering based anomaly detection
Extreme value theory

Access to Document

10.1007/978-3-030-22747-0_4

Cite this

Guggilam, S., Zaidi, S. M. A., Chandola, V., & Patra, A. K. (2019). Integrated Clustering and Anomaly Detection (INCAD) for Streaming Data. In J. M. F. Rodrigues, P. J. S. Cardoso, J. Monteiro, R. Lam, V. V. Krzhizhanovskaya, M. H. Lees, P. M. A. Sloot, & J. J. Dongarra (Eds.), Computational Science – ICCS 2019 - 19th International Conference, Proceedings (pp. 45-59). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11539 LNCS). Springer Verlag. https://doi.org/10.1007/978-3-030-22747-0_4

Guggilam, Sreelekha ; Zaidi, Syed Mohammed Arshad ; Chandola, Varun et al. / Integrated Clustering and Anomaly Detection (INCAD) for Streaming Data. Computational Science – ICCS 2019 - 19th International Conference, Proceedings. editor / João M.F. Rodrigues ; Pedro J.S. Cardoso ; Jânio Monteiro ; Roberto Lam ; Valeria V. Krzhizhanovskaya ; Michael H. Lees ; Peter M.A. Sloot ; Jack J. Dongarra. Springer Verlag, 2019. pp. 45-59 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{1783f0d6e35f47bbb5496a67edea44ac,

title = "Integrated Clustering and Anomaly Detection (INCAD) for Streaming Data",

abstract = "Most current clustering based anomaly detection methods use scoring schema and thresholds to classify anomalies. These methods are often tailored to target specific data sets with “known” number of clusters. The paper provides a streaming clustering and anomaly detection algorithm that does not require strict arbitrary thresholds on the anomaly scores or knowledge of the number of clusters while performing probabilistic anomaly detection and clustering simultaneously. This ensures that the cluster formation is not impacted by the presence of anomalous data, thereby leading to more reliable definition of “normal vs abnormal” behavior. The motivations behind developing the INCAD model [17] and the path that leads to the streaming model are discussed.",

keywords = "Anomaly detection, Bayesian non-parametric models, Clustering based anomaly detection, Extreme value theory",

author = "Sreelekha Guggilam and Zaidi, {Syed Mohammed Arshad} and Varun Chandola and Patra, {Abani K.}",

note = "Publisher Copyright: {\textcopyright} 2019, Springer Nature Switzerland AG.; 19th International Conference on Computational Science, ICCS 2019 ; Conference date: 12-06-2019 Through 14-06-2019",

year = "2019",

doi = "10.1007/978-3-030-22747-0_4",

language = "English",

isbn = "9783030227463",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Verlag",

pages = "45--59",

editor = "Rodrigues, {Jo{\~a}o M.F.} and Cardoso, {Pedro J.S.} and J{\^a}nio Monteiro and Roberto Lam and Krzhizhanovskaya, {Valeria V.} and Lees, {Michael H.} and Sloot, {Peter M.A.} and Dongarra, {Jack J.}",

booktitle = "Computational Science – ICCS 2019 - 19th International Conference, Proceedings",

}

Guggilam, S, Zaidi, SMA, Chandola, V & Patra, AK 2019, Integrated Clustering and Anomaly Detection (INCAD) for Streaming Data. in JMF Rodrigues, PJS Cardoso, J Monteiro, R Lam, VV Krzhizhanovskaya, MH Lees, PMA Sloot & JJ Dongarra (eds), Computational Science – ICCS 2019 - 19th International Conference, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 11539 LNCS, Springer Verlag, pp. 45-59, 19th International Conference on Computational Science, ICCS 2019, Faro, Portugal, 06/12/19. https://doi.org/10.1007/978-3-030-22747-0_4

Integrated Clustering and Anomaly Detection (INCAD) for Streaming Data. / Guggilam, Sreelekha; Zaidi, Syed Mohammed Arshad; Chandola, Varun et al.
Computational Science – ICCS 2019 - 19th International Conference, Proceedings. ed. / João M.F. Rodrigues; Pedro J.S. Cardoso; Jânio Monteiro; Roberto Lam; Valeria V. Krzhizhanovskaya; Michael H. Lees; Peter M.A. Sloot; Jack J. Dongarra. Springer Verlag, 2019. p. 45-59 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11539 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Integrated Clustering and Anomaly Detection (INCAD) for Streaming Data

AU - Guggilam, Sreelekha

AU - Zaidi, Syed Mohammed Arshad

AU - Chandola, Varun

AU - Patra, Abani K.

PY - 2019

Y1 - 2019

N2 - Most current clustering based anomaly detection methods use scoring schema and thresholds to classify anomalies. These methods are often tailored to target specific data sets with “known” number of clusters. The paper provides a streaming clustering and anomaly detection algorithm that does not require strict arbitrary thresholds on the anomaly scores or knowledge of the number of clusters while performing probabilistic anomaly detection and clustering simultaneously. This ensures that the cluster formation is not impacted by the presence of anomalous data, thereby leading to more reliable definition of “normal vs abnormal” behavior. The motivations behind developing the INCAD model [17] and the path that leads to the streaming model are discussed.

AB - Most current clustering based anomaly detection methods use scoring schema and thresholds to classify anomalies. These methods are often tailored to target specific data sets with “known” number of clusters. The paper provides a streaming clustering and anomaly detection algorithm that does not require strict arbitrary thresholds on the anomaly scores or knowledge of the number of clusters while performing probabilistic anomaly detection and clustering simultaneously. This ensures that the cluster formation is not impacted by the presence of anomalous data, thereby leading to more reliable definition of “normal vs abnormal” behavior. The motivations behind developing the INCAD model [17] and the path that leads to the streaming model are discussed.

KW - Anomaly detection

KW - Bayesian non-parametric models

KW - Clustering based anomaly detection

KW - Extreme value theory

UR - http://www.scopus.com/inward/record.url?scp=85067631338&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-22747-0_4

DO - 10.1007/978-3-030-22747-0_4

M3 - Conference contribution

AN - SCOPUS:85067631338

SN - 9783030227463

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 45

EP - 59

BT - Computational Science – ICCS 2019 - 19th International Conference, Proceedings

A2 - Rodrigues, João M.F.

A2 - Cardoso, Pedro J.S.

A2 - Monteiro, Jânio

A2 - Lam, Roberto

A2 - Krzhizhanovskaya, Valeria V.

A2 - Lees, Michael H.

A2 - Sloot, Peter M.A.

A2 - Dongarra, Jack J.

PB - Springer Verlag

T2 - 19th International Conference on Computational Science, ICCS 2019

Y2 - 12 June 2019 through 14 June 2019

ER -

Guggilam S, Zaidi SMA, Chandola V, Patra AK. Integrated Clustering and Anomaly Detection (INCAD) for Streaming Data. In Rodrigues JMF, Cardoso PJS, Monteiro J, Lam R, Krzhizhanovskaya VV, Lees MH, Sloot PMA, Dongarra JJ, editors, Computational Science – ICCS 2019 - 19th International Conference, Proceedings. Springer Verlag. 2019. p. 45-59. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-030-22747-0_4

Integrated Clustering and Anomaly Detection (INCAD) for Streaming Data

Abstract

Publication series

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this