Testing the Generalization of Neural Language Models for COVID-19 Misinformation Detection

Jan Philip Wahle; Nischal Ashok; Terry Ruas; Norman Meuschke; Tirthankar Ghosal; Bela Gipp

doi:10.1007/978-3-030-96957-8_33

Testing the Generalization of Neural Language Models for COVID-19 Misinformation Detection

Jan Philip Wahle, Nischal Ashok, Terry Ruas, Norman Meuschke, Tirthankar Ghosal, Bela Gipp

Analytics and AI Methods at Scale

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

13 Scopus citations

Abstract

A drastic rise in potentially life-threatening misinformation has been a by-product of the COVID-19 pandemic. Computational support to identify false information within the massive body of data on the topic is crucial to prevent harm. Researchers proposed many methods for flagging online misinformation related to COVID-19. However, these methods predominantly target specific content types (e.g., news) or platforms (e.g., Twitter). The methods’ capabilities to generalize were largely unclear so far. We evaluate fifteen Transformer-based models on five COVID-19 misinformation datasets that include social media posts, news articles, and scientific papers to fill this gap. We show tokenizers and models tailored to COVID-19 data do not provide a significant advantage over general-purpose ones. Our study provides a realistic assessment of models for detecting COVID-19 misinformation. We expect that evaluating a broad spectrum of datasets and models will benefit future research in developing misinformation detection systems.

Original language	English
Title of host publication	Information for a Better World
Subtitle of host publication	Shaping the Global Future - 17th International Conference, iConference 2022, Proceedings
Editors	Malte Smits
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	381-392
Number of pages	12
ISBN (Print)	9783030969561
DOIs	https://doi.org/10.1007/978-3-030-96957-8_33
State	Published - 2022
Event	17th International Conference on Information for a Better World: Shaping the Global Future, iConference 2022 - Virtual, Online Duration: Feb 28 2022 → Mar 4 2022

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	13192 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	17th International Conference on Information for a Better World: Shaping the Global Future, iConference 2022
City	Virtual, Online
Period	02/28/22 → 03/4/22

Keywords

COVID-19
Health
Social media
Transformers

Access to Document

10.1007/978-3-030-96957-8_33

Cite this

Wahle, J. P., Ashok, N., Ruas, T., Meuschke, N., Ghosal, T., & Gipp, B. (2022). Testing the Generalization of Neural Language Models for COVID-19 Misinformation Detection. In M. Smits (Ed.), Information for a Better World: Shaping the Global Future - 17th International Conference, iConference 2022, Proceedings (pp. 381-392). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13192 LNCS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-030-96957-8_33

Wahle, Jan Philip ; Ashok, Nischal ; Ruas, Terry et al. / Testing the Generalization of Neural Language Models for COVID-19 Misinformation Detection. Information for a Better World: Shaping the Global Future - 17th International Conference, iConference 2022, Proceedings. editor / Malte Smits. Springer Science and Business Media Deutschland GmbH, 2022. pp. 381-392 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{f6d9157aed024d6cb8543050d81a3259,

title = "Testing the Generalization of Neural Language Models for COVID-19 Misinformation Detection",

abstract = "A drastic rise in potentially life-threatening misinformation has been a by-product of the COVID-19 pandemic. Computational support to identify false information within the massive body of data on the topic is crucial to prevent harm. Researchers proposed many methods for flagging online misinformation related to COVID-19. However, these methods predominantly target specific content types (e.g., news) or platforms (e.g., Twitter). The methods{\textquoteright} capabilities to generalize were largely unclear so far. We evaluate fifteen Transformer-based models on five COVID-19 misinformation datasets that include social media posts, news articles, and scientific papers to fill this gap. We show tokenizers and models tailored to COVID-19 data do not provide a significant advantage over general-purpose ones. Our study provides a realistic assessment of models for detecting COVID-19 misinformation. We expect that evaluating a broad spectrum of datasets and models will benefit future research in developing misinformation detection systems.",

keywords = "COVID-19, Health, Social media, Transformers",

author = "Wahle, \{Jan Philip\} and Nischal Ashok and Terry Ruas and Norman Meuschke and Tirthankar Ghosal and Bela Gipp",

note = "Publisher Copyright: {\textcopyright} 2022, The Author(s), under exclusive license to Springer Nature Switzerland AG.; 17th International Conference on Information for a Better World: Shaping the Global Future, iConference 2022 ; Conference date: 28-02-2022 Through 04-03-2022",

year = "2022",

doi = "10.1007/978-3-030-96957-8\_33",

language = "English",

isbn = "9783030969561",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "381--392",

editor = "Malte Smits",

booktitle = "Information for a Better World",

}

Wahle, JP, Ashok, N, Ruas, T, Meuschke, N, Ghosal, T & Gipp, B 2022, Testing the Generalization of Neural Language Models for COVID-19 Misinformation Detection. in M Smits (ed.), Information for a Better World: Shaping the Global Future - 17th International Conference, iConference 2022, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 13192 LNCS, Springer Science and Business Media Deutschland GmbH, pp. 381-392, 17th International Conference on Information for a Better World: Shaping the Global Future, iConference 2022, Virtual, Online, 02/28/22. https://doi.org/10.1007/978-3-030-96957-8_33

Testing the Generalization of Neural Language Models for COVID-19 Misinformation Detection. / Wahle, Jan Philip; Ashok, Nischal; Ruas, Terry et al.
Information for a Better World: Shaping the Global Future - 17th International Conference, iConference 2022, Proceedings. ed. / Malte Smits. Springer Science and Business Media Deutschland GmbH, 2022. p. 381-392 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13192 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Testing the Generalization of Neural Language Models for COVID-19 Misinformation Detection

AU - Wahle, Jan Philip

AU - Ashok, Nischal

AU - Ruas, Terry

AU - Meuschke, Norman

AU - Ghosal, Tirthankar

AU - Gipp, Bela

PY - 2022

Y1 - 2022

N2 - A drastic rise in potentially life-threatening misinformation has been a by-product of the COVID-19 pandemic. Computational support to identify false information within the massive body of data on the topic is crucial to prevent harm. Researchers proposed many methods for flagging online misinformation related to COVID-19. However, these methods predominantly target specific content types (e.g., news) or platforms (e.g., Twitter). The methods’ capabilities to generalize were largely unclear so far. We evaluate fifteen Transformer-based models on five COVID-19 misinformation datasets that include social media posts, news articles, and scientific papers to fill this gap. We show tokenizers and models tailored to COVID-19 data do not provide a significant advantage over general-purpose ones. Our study provides a realistic assessment of models for detecting COVID-19 misinformation. We expect that evaluating a broad spectrum of datasets and models will benefit future research in developing misinformation detection systems.

AB - A drastic rise in potentially life-threatening misinformation has been a by-product of the COVID-19 pandemic. Computational support to identify false information within the massive body of data on the topic is crucial to prevent harm. Researchers proposed many methods for flagging online misinformation related to COVID-19. However, these methods predominantly target specific content types (e.g., news) or platforms (e.g., Twitter). The methods’ capabilities to generalize were largely unclear so far. We evaluate fifteen Transformer-based models on five COVID-19 misinformation datasets that include social media posts, news articles, and scientific papers to fill this gap. We show tokenizers and models tailored to COVID-19 data do not provide a significant advantage over general-purpose ones. Our study provides a realistic assessment of models for detecting COVID-19 misinformation. We expect that evaluating a broad spectrum of datasets and models will benefit future research in developing misinformation detection systems.

KW - COVID-19

KW - Health

KW - Social media

KW - Transformers

UR - https://www.scopus.com/pages/publications/85126243726

U2 - 10.1007/978-3-030-96957-8_33

DO - 10.1007/978-3-030-96957-8_33

M3 - Conference contribution

AN - SCOPUS:85126243726

SN - 9783030969561

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 381

EP - 392

BT - Information for a Better World

A2 - Smits, Malte

PB - Springer Science and Business Media Deutschland GmbH

T2 - 17th International Conference on Information for a Better World: Shaping the Global Future, iConference 2022

Y2 - 28 February 2022 through 4 March 2022

ER -

Wahle JP, Ashok N, Ruas T, Meuschke N, Ghosal T, Gipp B. Testing the Generalization of Neural Language Models for COVID-19 Misinformation Detection. In Smits M, editor, Information for a Better World: Shaping the Global Future - 17th International Conference, iConference 2022, Proceedings. Springer Science and Business Media Deutschland GmbH. 2022. p. 381-392. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-030-96957-8_33

Testing the Generalization of Neural Language Models for COVID-19 Misinformation Detection

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this