AI ATAC 1: An Evaluation of Prominent Commercial Malware Detectors

Robert Bridges; Brian Weber; Justin M. Beaver; Jared M. Smith; Miki E. Verma; Savannah Norem; Kevin Spakes; Cory Watson; Jeff A. Nichols; Brian Jewell; Michael D. Iannacone; Chelsey Dunivan Stahl; Kelly M.T. Huffer; T. Sean Oesch

doi:10.1109/BigData59044.2023.10386590

AI ATAC 1: An Evaluation of Prominent Commercial Malware Detectors

Robert Bridges, Brian Weber, Justin M. Beaver, Jared M. Smith, Miki E. Verma, Savannah Norem, Kevin Spakes, Cory Watson, Jeff A. Nichols, Brian Jewell, Michael D. Iannacone, Chelsey Dunivan Stahl, Kelly M.T. Huffer, T. Sean Oesch

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

This work presents an evaluation of six prominent commercial endpoint malware detectors, a network malware detector, and a file-conviction algorithm from a cyber technology vendor. The evaluation was administered as the first of the Artificial I ntelligence Applications t o Autonomous Cybersecurity (AI ATAC) prize challenges, funded by / completed in service of the US Navy. The experiment employed 100K files (50/50% benign/malicious) with a stratified distribution of file types, including 1K zero-day program executables (increasing experiment size two orders of magnitude over previous work). We present an evaluation process of delivering a file to a fresh virtual machine donning the detection technology, waiting 90s to allow static detection, then executing the file and waiting another period for dynamic detection; this allows greater fidelity in the observational data than previous experiments, in particular, resource and time-to-detection statistics. To execute all 800K trials (100K files × 8 tools), a software framework is designed to choreograph the experiment into an automated, time-synced, and reproducible workflow with substantial parallelization. Software with base classes for this framework are provided. A cost-benefit model was configured to integrate the tools' detection statistics into a comparable quantity by simulating costs of use. This provides a ranking methodology for cyber competitions and a lens for reasoning about the varied statistical results. The results provide insights on state of commercial malware detection.

Original language	English
Title of host publication	Proceedings - 2023 IEEE International Conference on Big Data, BigData 2023
Editors	Jingrui He, Themis Palpanas, Xiaohua Hu, Alfredo Cuzzocrea, Dejing Dou, Dominik Slezak, Wei Wang, Aleksandra Gruca, Jerry Chun-Wei Lin, Rakesh Agrawal
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	1620-1629
Number of pages	10
ISBN (Electronic)	9798350324457
DOIs	https://doi.org/10.1109/BigData59044.2023.10386590
State	Published - 2023
Event	2023 IEEE International Conference on Big Data, BigData 2023 - Sorrento, Italy Duration: Dec 15 2023 → Dec 18 2023

Publication series

Name	Proceedings - 2023 IEEE International Conference on Big Data, BigData 2023

Conference

Conference	2023 IEEE International Conference on Big Data, BigData 2023
Country/Territory	Italy
City	Sorrento
Period	12/15/23 → 12/18/23

Funding

The research is based on work supported by the US Department of Defense (DOD), Naval Information Warfare Systems Command (NAVWAR), via the US Department of Energy (DOE) under contract DE-AC05-00OR22725. The views and conclusions contained herein are those of the authors and should not be interpreted as representing the official p olicies o re ndorsements, e ither e xpressed o r i mplied, o f t he DOD, NAVWAR, or the US Government. The US Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation thereon. MIT licensed code containing base classes for the software described in this paper, as well as an example implementation of the framework is provided [1]. Authors thank Charlie Horak for the editorial review, Mike Karlbom for ongoing support and guidance; Jessica Briesacker for counseling. This manuscript has been co-authored by UT-Battelle, LLC under Contract No. DE-AC05-00OR22725 with the US DOE. The United States Government retains and the publisher, by accepting the article for publication, acknowledges that the United States Government retains a non-exclusive, paid-up, irrevocable, world-wide license to publish or reproduce the published form of this manuscript, or allow others to do so, for United States Government purposes. The DOE will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan (http://energy.gov/downloads/doe-public-access-plan). The research is based on work supported by the US Department of Defense (DOD), Naval Information Warfare Systems Command (NAVWAR), via the US Department of Energy (DOE) under contract DE-AC05-00OR22725. The views and conclusions contained herein are those of the authors and should not be interpreted as representing the official p olicies o r e ndorsements, e ither e xpressed o r i mplied, o f t he DOD, NAVWAR, or the US Government. The US Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation thereon.

Keywords

cost benefit analysis
dynamic analysis
endpoint detection
evaluation
intrusion detection
machine learning
malware detection
network detection
static analysis
test

Access to Document

10.1109/BigData59044.2023.10386590

Cite this

Bridges, R., Weber, B., Beaver, J. M., Smith, J. M., Verma, M. E., Norem, S., Spakes, K., Watson, C., Nichols, J. A., Jewell, B., Iannacone, M. D., Stahl, C. D., Huffer, K. M. T., & Oesch, T. S. (2023). AI ATAC 1: An Evaluation of Prominent Commercial Malware Detectors. In J. He, T. Palpanas, X. Hu, A. Cuzzocrea, D. Dou, D. Slezak, W. Wang, A. Gruca, J. C.-W. Lin, & R. Agrawal (Eds.), Proceedings - 2023 IEEE International Conference on Big Data, BigData 2023 (pp. 1620-1629). (Proceedings - 2023 IEEE International Conference on Big Data, BigData 2023). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/BigData59044.2023.10386590

Bridges, Robert ; Weber, Brian ; Beaver, Justin M. et al. / AI ATAC 1 : An Evaluation of Prominent Commercial Malware Detectors. Proceedings - 2023 IEEE International Conference on Big Data, BigData 2023. editor / Jingrui He ; Themis Palpanas ; Xiaohua Hu ; Alfredo Cuzzocrea ; Dejing Dou ; Dominik Slezak ; Wei Wang ; Aleksandra Gruca ; Jerry Chun-Wei Lin ; Rakesh Agrawal. Institute of Electrical and Electronics Engineers Inc., 2023. pp. 1620-1629 (Proceedings - 2023 IEEE International Conference on Big Data, BigData 2023).

@inproceedings{2677b2d799df4cd0ba723c7c3632a811,

title = "AI ATAC 1: An Evaluation of Prominent Commercial Malware Detectors",

abstract = "This work presents an evaluation of six prominent commercial endpoint malware detectors, a network malware detector, and a file-conviction algorithm from a cyber technology vendor. The evaluation was administered as the first of the Artificial I ntelligence Applications t o Autonomous Cybersecurity (AI ATAC) prize challenges, funded by / completed in service of the US Navy. The experiment employed 100K files (50/50% benign/malicious) with a stratified distribution of file types, including 1K zero-day program executables (increasing experiment size two orders of magnitude over previous work). We present an evaluation process of delivering a file to a fresh virtual machine donning the detection technology, waiting 90s to allow static detection, then executing the file and waiting another period for dynamic detection; this allows greater fidelity in the observational data than previous experiments, in particular, resource and time-to-detection statistics. To execute all 800K trials (100K files × 8 tools), a software framework is designed to choreograph the experiment into an automated, time-synced, and reproducible workflow with substantial parallelization. Software with base classes for this framework are provided. A cost-benefit model was configured to integrate the tools' detection statistics into a comparable quantity by simulating costs of use. This provides a ranking methodology for cyber competitions and a lens for reasoning about the varied statistical results. The results provide insights on state of commercial malware detection.",

keywords = "cost benefit analysis, dynamic analysis, endpoint detection, evaluation, intrusion detection, machine learning, malware detection, network detection, static analysis, test",

author = "Robert Bridges and Brian Weber and Beaver, {Justin M.} and Smith, {Jared M.} and Verma, {Miki E.} and Savannah Norem and Kevin Spakes and Cory Watson and Nichols, {Jeff A.} and Brian Jewell and Iannacone, {Michael D.} and Stahl, {Chelsey Dunivan} and Huffer, {Kelly M.T.} and Oesch, {T. Sean}",

note = "Publisher Copyright: {\textcopyright} 2023 IEEE.; 2023 IEEE International Conference on Big Data, BigData 2023 ; Conference date: 15-12-2023 Through 18-12-2023",

year = "2023",

doi = "10.1109/BigData59044.2023.10386590",

language = "English",

series = "Proceedings - 2023 IEEE International Conference on Big Data, BigData 2023",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "1620--1629",

editor = "Jingrui He and Themis Palpanas and Xiaohua Hu and Alfredo Cuzzocrea and Dejing Dou and Dominik Slezak and Wei Wang and Aleksandra Gruca and Lin, {Jerry Chun-Wei} and Rakesh Agrawal",

booktitle = "Proceedings - 2023 IEEE International Conference on Big Data, BigData 2023",

}

Bridges, R, Weber, B, Beaver, JM, Smith, JM, Verma, ME, Norem, S, Spakes, K, Watson, C, Nichols, JA , Jewell, B, Iannacone, MD, Stahl, CD , Huffer, KMT & Oesch, TS 2023, AI ATAC 1: An Evaluation of Prominent Commercial Malware Detectors. in J He, T Palpanas, X Hu, A Cuzzocrea, D Dou, D Slezak, W Wang, A Gruca, JC-W Lin & R Agrawal (eds), Proceedings - 2023 IEEE International Conference on Big Data, BigData 2023. Proceedings - 2023 IEEE International Conference on Big Data, BigData 2023, Institute of Electrical and Electronics Engineers Inc., pp. 1620-1629, 2023 IEEE International Conference on Big Data, BigData 2023, Sorrento, Italy, 12/15/23. https://doi.org/10.1109/BigData59044.2023.10386590

AI ATAC 1: An Evaluation of Prominent Commercial Malware Detectors. / Bridges, Robert; Weber, Brian; Beaver, Justin M. et al.
Proceedings - 2023 IEEE International Conference on Big Data, BigData 2023. ed. / Jingrui He; Themis Palpanas; Xiaohua Hu; Alfredo Cuzzocrea; Dejing Dou; Dominik Slezak; Wei Wang; Aleksandra Gruca; Jerry Chun-Wei Lin; Rakesh Agrawal. Institute of Electrical and Electronics Engineers Inc., 2023. p. 1620-1629 (Proceedings - 2023 IEEE International Conference on Big Data, BigData 2023).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - AI ATAC 1

T2 - 2023 IEEE International Conference on Big Data, BigData 2023

AU - Bridges, Robert

AU - Weber, Brian

AU - Beaver, Justin M.

AU - Smith, Jared M.

AU - Verma, Miki E.

AU - Norem, Savannah

AU - Spakes, Kevin

AU - Watson, Cory

AU - Nichols, Jeff A.

AU - Jewell, Brian

AU - Iannacone, Michael D.

AU - Stahl, Chelsey Dunivan

AU - Huffer, Kelly M.T.

AU - Oesch, T. Sean

PY - 2023

Y1 - 2023

N2 - This work presents an evaluation of six prominent commercial endpoint malware detectors, a network malware detector, and a file-conviction algorithm from a cyber technology vendor. The evaluation was administered as the first of the Artificial I ntelligence Applications t o Autonomous Cybersecurity (AI ATAC) prize challenges, funded by / completed in service of the US Navy. The experiment employed 100K files (50/50% benign/malicious) with a stratified distribution of file types, including 1K zero-day program executables (increasing experiment size two orders of magnitude over previous work). We present an evaluation process of delivering a file to a fresh virtual machine donning the detection technology, waiting 90s to allow static detection, then executing the file and waiting another period for dynamic detection; this allows greater fidelity in the observational data than previous experiments, in particular, resource and time-to-detection statistics. To execute all 800K trials (100K files × 8 tools), a software framework is designed to choreograph the experiment into an automated, time-synced, and reproducible workflow with substantial parallelization. Software with base classes for this framework are provided. A cost-benefit model was configured to integrate the tools' detection statistics into a comparable quantity by simulating costs of use. This provides a ranking methodology for cyber competitions and a lens for reasoning about the varied statistical results. The results provide insights on state of commercial malware detection.

AB - This work presents an evaluation of six prominent commercial endpoint malware detectors, a network malware detector, and a file-conviction algorithm from a cyber technology vendor. The evaluation was administered as the first of the Artificial I ntelligence Applications t o Autonomous Cybersecurity (AI ATAC) prize challenges, funded by / completed in service of the US Navy. The experiment employed 100K files (50/50% benign/malicious) with a stratified distribution of file types, including 1K zero-day program executables (increasing experiment size two orders of magnitude over previous work). We present an evaluation process of delivering a file to a fresh virtual machine donning the detection technology, waiting 90s to allow static detection, then executing the file and waiting another period for dynamic detection; this allows greater fidelity in the observational data than previous experiments, in particular, resource and time-to-detection statistics. To execute all 800K trials (100K files × 8 tools), a software framework is designed to choreograph the experiment into an automated, time-synced, and reproducible workflow with substantial parallelization. Software with base classes for this framework are provided. A cost-benefit model was configured to integrate the tools' detection statistics into a comparable quantity by simulating costs of use. This provides a ranking methodology for cyber competitions and a lens for reasoning about the varied statistical results. The results provide insights on state of commercial malware detection.

KW - cost benefit analysis

KW - dynamic analysis

KW - endpoint detection

KW - evaluation

KW - intrusion detection

KW - machine learning

KW - malware detection

KW - network detection

KW - static analysis

KW - test

UR - http://www.scopus.com/inward/record.url?scp=85184984629&partnerID=8YFLogxK

U2 - 10.1109/BigData59044.2023.10386590

DO - 10.1109/BigData59044.2023.10386590

M3 - Conference contribution

AN - SCOPUS:85184984629

T3 - Proceedings - 2023 IEEE International Conference on Big Data, BigData 2023

SP - 1620

EP - 1629

BT - Proceedings - 2023 IEEE International Conference on Big Data, BigData 2023

A2 - He, Jingrui

A2 - Palpanas, Themis

A2 - Hu, Xiaohua

A2 - Cuzzocrea, Alfredo

A2 - Dou, Dejing

A2 - Slezak, Dominik

A2 - Wang, Wei

A2 - Gruca, Aleksandra

A2 - Lin, Jerry Chun-Wei

A2 - Agrawal, Rakesh

PB - Institute of Electrical and Electronics Engineers Inc.

Y2 - 15 December 2023 through 18 December 2023

ER -

Bridges R, Weber B, Beaver JM, Smith JM, Verma ME, Norem S et al. AI ATAC 1: An Evaluation of Prominent Commercial Malware Detectors. In He J, Palpanas T, Hu X, Cuzzocrea A, Dou D, Slezak D, Wang W, Gruca A, Lin JCW, Agrawal R, editors, Proceedings - 2023 IEEE International Conference on Big Data, BigData 2023. Institute of Electrical and Electronics Engineers Inc. 2023. p. 1620-1629. (Proceedings - 2023 IEEE International Conference on Big Data, BigData 2023). doi: 10.1109/BigData59044.2023.10386590

AI ATAC 1: An Evaluation of Prominent Commercial Malware Detectors

Abstract

Publication series

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this