A Mathematical Framework for Evaluation of SOAR Tools with Limited Survey Data

Savannah Norem; Ashley E. Rice; Samantha Erwin; Robert A. Bridges; Sean Oesch; Brian Weber

doi:10.1007/978-3-030-95484-0_32

A Mathematical Framework for Evaluation of SOAR Tools with Limited Survey Data

Savannah Norem, Ashley E. Rice, Samantha Erwin, Robert A. Bridges, Sean Oesch, Brian Weber

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

2 Scopus citations

Abstract

Security operation centers (SOCs) all over the world are tasked with reacting to cybersecurity alerts ranging in severity. Security Orchestration, Automation, and Response (SOAR) tools streamline cybersecurity alert responses by SOC operators. SOAR tool adoption is expensive both in effort and finances. Hence, it is crucial to limit adoption to those most worthwhile; yet no research evaluating or comparing SOAR tools exists. The goal of this work is to evaluate several SOAR tools using specific criteria pertaining to their usability. SOC operators were asked to first complete a survey about what SOAR tool aspects are most important. Operators were then assigned a set of SOAR tools for which they viewed demonstration and overview videos, and then operators completed a second survey wherein they were tasked with evaluating each of the tools on the aspects from the first survey. In addition, operators provided an overall rating to each of their assigned tools, and provided a ranking of their tools in order of preference. Due to time constraints on SOC operators for thorough testing, we provide a systematic method of downselecting a large pool of SOAR tools to a select few that merit next-step hands-on evaluation by SOC operators. Furthermore, the analyses conducted in this survey help to inform future development of SOAR tools to ensure that the appropriate functions are available for use in a SOC.

Original language	English
Title of host publication	Computer Security. ESORICS 2021 International Workshops - CyberICPS, SECPRE, ADIoT, SPOSE, CPS4CIP, and CDT and SECOMANE, 2021
Editors	Sokratis Katsikas, Costas Lambrinoudakis, Nora Cuppens, John Mylopoulos, Christos Kalloniatis, Weizhi Meng, Steven Furnell, Frank Pallas, Jörg Pohle, M. Angela Sasse, Habtamu Abie, Silvio Ranise, Luca Verderame, Enrico Cambiaso, Jorge Maestre Vidal, Marco Antonio Sotelo Monge
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	557-575
Number of pages	19
ISBN (Print)	9783030954833
DOIs	https://doi.org/10.1007/978-3-030-95484-0_32
State	Published - 2022
Event	7th Workshop on the Security of Industrial Control Systems and of Cyber-Physical Systems, CyberICPS 2021, 5th International Workshop on Security and Privacy Requirements Engineering, SECPRE 2021, 4th International Workshop on Attacks and Defenses for Internet-of-Things, ADIoT 2021, 3rd Workshop on Security, Privacy, Organizations, and Systems Engineering, SPOSE 2021, 2nd Cyber-Physical Security for Critical Infrastructures Protection, CPS4CIP 2021 and 1st International Workshop on Cyber Defence Technologies and Secure Communications at the Network Edge, CDT and SECOMANE 2021 held in conjunction with 26th European Symposium on Research in Computer Security, ESORICS 2021 - Virtual, Online Duration: Oct 4 2021 → Oct 8 2021

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	13106 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	7th Workshop on the Security of Industrial Control Systems and of Cyber-Physical Systems, CyberICPS 2021, 5th International Workshop on Security and Privacy Requirements Engineering, SECPRE 2021, 4th International Workshop on Attacks and Defenses for Internet-of-Things, ADIoT 2021, 3rd Workshop on Security, Privacy, Organizations, and Systems Engineering, SPOSE 2021, 2nd Cyber-Physical Security for Critical Infrastructures Protection, CPS4CIP 2021 and 1st International Workshop on Cyber Defence Technologies and Secure Communications at the Network Edge, CDT and SECOMANE 2021 held in conjunction with 26th European Symposium on Research in Computer Security, ESORICS 2021
City	Virtual, Online
Period	10/4/21 → 10/8/21

Funding

This manuscript has been co-authored by UT-Battelle, LLC, under contract DE-AC05-00OR22725 with the US Department of Energy (DOE). The US government retains and the publisher, by accepting the article for publication, acknowledges that the US government retains a nonexclusive, paid-up, irrevocable, worldwide license to publish or reproduce the published form of this manuscript, or allow others to do so, for US government purposes. DOE will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan (http://energy. gov/downloads/doe-public-access-plan). Acknowledgements. Special thanks to Jeff Meredith for assisting with the website. The research is based upon work supported by the Department of Defense (DOD), Naval Information Warfare Systems Command (NAVWAR), via the Department of Energy (DOE) under contract DE-AC05-00OR22725. The views and conclusions contained herein are those of the authors and should not be interpreted as representing the official policies or endorsements, either expressed or implied, of the DOD, NAVWAR, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon. This manuscript has been co-authored by UT-Battelle, LLC, under contract DE-AC05-00OR22725 with the US Department of Energy (DOE). The US government retains and the publisher, by accepting the article for publication, acknowledges that the US government retains a nonexclusive, paid-up, irrevocable, worldwide license to publish or reproduce the published form of this manuscript, or allow others to do so, for US government purposes. DOE will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan (http://energy. gov/downloads/doe-public-access-plan).. Acknowledgements. Special thanks to Jeff Meredith for assisting with the website. The research is based upon work supported by the Department of Defense (DOD), Naval Information Warfare Systems Command (NAVWAR), via the Department of Energy (DOE) under contract DE-AC05-00OR22725. The views and conclusions contained herein are those of the authors and should not be interpreted as representing the official policies or endorsements, either expressed or implied, of the DOD, NAVWAR, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon.

Funders	Funder number
DOE Public Access Plan
U.S. Department of Defense
U.S. Department of Energy	DE-AC05-00OR22725
Naval Information Warfare Systems Command

Keywords

Cybersecurity
SOAR tools
User study

Access to Document

10.1007/978-3-030-95484-0_32

Cite this

Norem, S., Rice, A. E., Erwin, S., Bridges, R. A., Oesch, S., & Weber, B. (2022). A Mathematical Framework for Evaluation of SOAR Tools with Limited Survey Data. In S. Katsikas, C. Lambrinoudakis, N. Cuppens, J. Mylopoulos, C. Kalloniatis, W. Meng, S. Furnell, F. Pallas, J. Pohle, M. A. Sasse, H. Abie, S. Ranise, L. Verderame, E. Cambiaso, J. Maestre Vidal, & M. A. Sotelo Monge (Eds.), Computer Security. ESORICS 2021 International Workshops - CyberICPS, SECPRE, ADIoT, SPOSE, CPS4CIP, and CDT and SECOMANE, 2021 (pp. 557-575). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13106 LNCS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-030-95484-0_32

Norem, Savannah ; Rice, Ashley E. ; Erwin, Samantha et al. / A Mathematical Framework for Evaluation of SOAR Tools with Limited Survey Data. Computer Security. ESORICS 2021 International Workshops - CyberICPS, SECPRE, ADIoT, SPOSE, CPS4CIP, and CDT and SECOMANE, 2021. editor / Sokratis Katsikas ; Costas Lambrinoudakis ; Nora Cuppens ; John Mylopoulos ; Christos Kalloniatis ; Weizhi Meng ; Steven Furnell ; Frank Pallas ; Jörg Pohle ; M. Angela Sasse ; Habtamu Abie ; Silvio Ranise ; Luca Verderame ; Enrico Cambiaso ; Jorge Maestre Vidal ; Marco Antonio Sotelo Monge. Springer Science and Business Media Deutschland GmbH, 2022. pp. 557-575 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{7cffae9e7f1244c699cab7b0d4acf1a3,

title = "A Mathematical Framework for Evaluation of SOAR Tools with Limited Survey Data",

abstract = "Security operation centers (SOCs) all over the world are tasked with reacting to cybersecurity alerts ranging in severity. Security Orchestration, Automation, and Response (SOAR) tools streamline cybersecurity alert responses by SOC operators. SOAR tool adoption is expensive both in effort and finances. Hence, it is crucial to limit adoption to those most worthwhile; yet no research evaluating or comparing SOAR tools exists. The goal of this work is to evaluate several SOAR tools using specific criteria pertaining to their usability. SOC operators were asked to first complete a survey about what SOAR tool aspects are most important. Operators were then assigned a set of SOAR tools for which they viewed demonstration and overview videos, and then operators completed a second survey wherein they were tasked with evaluating each of the tools on the aspects from the first survey. In addition, operators provided an overall rating to each of their assigned tools, and provided a ranking of their tools in order of preference. Due to time constraints on SOC operators for thorough testing, we provide a systematic method of downselecting a large pool of SOAR tools to a select few that merit next-step hands-on evaluation by SOC operators. Furthermore, the analyses conducted in this survey help to inform future development of SOAR tools to ensure that the appropriate functions are available for use in a SOC.",

keywords = "Cybersecurity, SOAR tools, User study",

author = "Savannah Norem and Rice, {Ashley E.} and Samantha Erwin and Bridges, {Robert A.} and Sean Oesch and Brian Weber",

note = "Publisher Copyright: {\textcopyright} 2022, Springer Nature Switzerland AG.; 7th Workshop on the Security of Industrial Control Systems and of Cyber-Physical Systems, CyberICPS 2021, 5th International Workshop on Security and Privacy Requirements Engineering, SECPRE 2021, 4th International Workshop on Attacks and Defenses for Internet-of-Things, ADIoT 2021, 3rd Workshop on Security, Privacy, Organizations, and Systems Engineering, SPOSE 2021, 2nd Cyber-Physical Security for Critical Infrastructures Protection, CPS4CIP 2021 and 1st International Workshop on Cyber Defence Technologies and Secure Communications at the Network Edge, CDT and SECOMANE 2021 held in conjunction with 26th European Symposium on Research in Computer Security, ESORICS 2021 ; Conference date: 04-10-2021 Through 08-10-2021",

year = "2022",

doi = "10.1007/978-3-030-95484-0_32",

language = "English",

isbn = "9783030954833",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "557--575",

editor = "Sokratis Katsikas and Costas Lambrinoudakis and Nora Cuppens and John Mylopoulos and Christos Kalloniatis and Weizhi Meng and Steven Furnell and Frank Pallas and J{\"o}rg Pohle and Sasse, {M. Angela} and Habtamu Abie and Silvio Ranise and Luca Verderame and Enrico Cambiaso and {Maestre Vidal}, Jorge and {Sotelo Monge}, {Marco Antonio}",

booktitle = "Computer Security. ESORICS 2021 International Workshops - CyberICPS, SECPRE, ADIoT, SPOSE, CPS4CIP, and CDT and SECOMANE, 2021",

}

Norem, S, Rice, AE, Erwin, S, Bridges, RA, Oesch, S & Weber, B 2022, A Mathematical Framework for Evaluation of SOAR Tools with Limited Survey Data. in S Katsikas, C Lambrinoudakis, N Cuppens, J Mylopoulos, C Kalloniatis, W Meng, S Furnell, F Pallas, J Pohle, MA Sasse, H Abie, S Ranise, L Verderame, E Cambiaso, J Maestre Vidal & MA Sotelo Monge (eds), Computer Security. ESORICS 2021 International Workshops - CyberICPS, SECPRE, ADIoT, SPOSE, CPS4CIP, and CDT and SECOMANE, 2021. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 13106 LNCS, Springer Science and Business Media Deutschland GmbH, pp. 557-575, 7th Workshop on the Security of Industrial Control Systems and of Cyber-Physical Systems, CyberICPS 2021, 5th International Workshop on Security and Privacy Requirements Engineering, SECPRE 2021, 4th International Workshop on Attacks and Defenses for Internet-of-Things, ADIoT 2021, 3rd Workshop on Security, Privacy, Organizations, and Systems Engineering, SPOSE 2021, 2nd Cyber-Physical Security for Critical Infrastructures Protection, CPS4CIP 2021 and 1st International Workshop on Cyber Defence Technologies and Secure Communications at the Network Edge, CDT and SECOMANE 2021 held in conjunction with 26th European Symposium on Research in Computer Security, ESORICS 2021, Virtual, Online, 10/4/21. https://doi.org/10.1007/978-3-030-95484-0_32

A Mathematical Framework for Evaluation of SOAR Tools with Limited Survey Data. / Norem, Savannah; Rice, Ashley E.; Erwin, Samantha et al.
Computer Security. ESORICS 2021 International Workshops - CyberICPS, SECPRE, ADIoT, SPOSE, CPS4CIP, and CDT and SECOMANE, 2021. ed. / Sokratis Katsikas; Costas Lambrinoudakis; Nora Cuppens; John Mylopoulos; Christos Kalloniatis; Weizhi Meng; Steven Furnell; Frank Pallas; Jörg Pohle; M. Angela Sasse; Habtamu Abie; Silvio Ranise; Luca Verderame; Enrico Cambiaso; Jorge Maestre Vidal; Marco Antonio Sotelo Monge. Springer Science and Business Media Deutschland GmbH, 2022. p. 557-575 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13106 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - A Mathematical Framework for Evaluation of SOAR Tools with Limited Survey Data

AU - Norem, Savannah

AU - Rice, Ashley E.

AU - Erwin, Samantha

AU - Bridges, Robert A.

AU - Oesch, Sean

AU - Weber, Brian

PY - 2022

Y1 - 2022

N2 - Security operation centers (SOCs) all over the world are tasked with reacting to cybersecurity alerts ranging in severity. Security Orchestration, Automation, and Response (SOAR) tools streamline cybersecurity alert responses by SOC operators. SOAR tool adoption is expensive both in effort and finances. Hence, it is crucial to limit adoption to those most worthwhile; yet no research evaluating or comparing SOAR tools exists. The goal of this work is to evaluate several SOAR tools using specific criteria pertaining to their usability. SOC operators were asked to first complete a survey about what SOAR tool aspects are most important. Operators were then assigned a set of SOAR tools for which they viewed demonstration and overview videos, and then operators completed a second survey wherein they were tasked with evaluating each of the tools on the aspects from the first survey. In addition, operators provided an overall rating to each of their assigned tools, and provided a ranking of their tools in order of preference. Due to time constraints on SOC operators for thorough testing, we provide a systematic method of downselecting a large pool of SOAR tools to a select few that merit next-step hands-on evaluation by SOC operators. Furthermore, the analyses conducted in this survey help to inform future development of SOAR tools to ensure that the appropriate functions are available for use in a SOC.

AB - Security operation centers (SOCs) all over the world are tasked with reacting to cybersecurity alerts ranging in severity. Security Orchestration, Automation, and Response (SOAR) tools streamline cybersecurity alert responses by SOC operators. SOAR tool adoption is expensive both in effort and finances. Hence, it is crucial to limit adoption to those most worthwhile; yet no research evaluating or comparing SOAR tools exists. The goal of this work is to evaluate several SOAR tools using specific criteria pertaining to their usability. SOC operators were asked to first complete a survey about what SOAR tool aspects are most important. Operators were then assigned a set of SOAR tools for which they viewed demonstration and overview videos, and then operators completed a second survey wherein they were tasked with evaluating each of the tools on the aspects from the first survey. In addition, operators provided an overall rating to each of their assigned tools, and provided a ranking of their tools in order of preference. Due to time constraints on SOC operators for thorough testing, we provide a systematic method of downselecting a large pool of SOAR tools to a select few that merit next-step hands-on evaluation by SOC operators. Furthermore, the analyses conducted in this survey help to inform future development of SOAR tools to ensure that the appropriate functions are available for use in a SOC.

KW - Cybersecurity

KW - SOAR tools

KW - User study

UR - http://www.scopus.com/inward/record.url?scp=85125219510&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-95484-0_32

DO - 10.1007/978-3-030-95484-0_32

M3 - Conference contribution

AN - SCOPUS:85125219510

SN - 9783030954833

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 557

EP - 575

BT - Computer Security. ESORICS 2021 International Workshops - CyberICPS, SECPRE, ADIoT, SPOSE, CPS4CIP, and CDT and SECOMANE, 2021

A2 - Katsikas, Sokratis

A2 - Lambrinoudakis, Costas

A2 - Cuppens, Nora

A2 - Mylopoulos, John

A2 - Kalloniatis, Christos

A2 - Meng, Weizhi

A2 - Furnell, Steven

A2 - Pallas, Frank

A2 - Pohle, Jörg

A2 - Sasse, M. Angela

A2 - Abie, Habtamu

A2 - Ranise, Silvio

A2 - Verderame, Luca

A2 - Cambiaso, Enrico

A2 - Maestre Vidal, Jorge

A2 - Sotelo Monge, Marco Antonio

PB - Springer Science and Business Media Deutschland GmbH

T2 - 7th Workshop on the Security of Industrial Control Systems and of Cyber-Physical Systems, CyberICPS 2021, 5th International Workshop on Security and Privacy Requirements Engineering, SECPRE 2021, 4th International Workshop on Attacks and Defenses for Internet-of-Things, ADIoT 2021, 3rd Workshop on Security, Privacy, Organizations, and Systems Engineering, SPOSE 2021, 2nd Cyber-Physical Security for Critical Infrastructures Protection, CPS4CIP 2021 and 1st International Workshop on Cyber Defence Technologies and Secure Communications at the Network Edge, CDT and SECOMANE 2021 held in conjunction with 26th European Symposium on Research in Computer Security, ESORICS 2021

Y2 - 4 October 2021 through 8 October 2021

ER -

Norem S, Rice AE, Erwin S, Bridges RA, Oesch S, Weber B. A Mathematical Framework for Evaluation of SOAR Tools with Limited Survey Data. In Katsikas S, Lambrinoudakis C, Cuppens N, Mylopoulos J, Kalloniatis C, Meng W, Furnell S, Pallas F, Pohle J, Sasse MA, Abie H, Ranise S, Verderame L, Cambiaso E, Maestre Vidal J, Sotelo Monge MA, editors, Computer Security. ESORICS 2021 International Workshops - CyberICPS, SECPRE, ADIoT, SPOSE, CPS4CIP, and CDT and SECOMANE, 2021. Springer Science and Business Media Deutschland GmbH. 2022. p. 557-575. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-030-95484-0_32