An Agenda for Multimodal Foundation Models for Earth Observation

Philipe Dias; Abhishek Potnis; Sreelekha Guggilam; Lexie Yang; Aristeidis Tsaris; Henry Medeiros; Dalton Lunga

doi:10.1109/IGARSS52108.2023.10282966

An Agenda for Multimodal Foundation Models for Earth Observation

Philipe Dias, Abhishek Potnis, Sreelekha Guggilam, Lexie Yang, Aristeidis Tsaris, Henry Medeiros, Dalton Lunga

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

2 Scopus citations

Abstract

Archives of remote sensing (RS) data are increasing swiftly as new sensing modalities with enhanced spatiotemporal resolution become operational. While promising new breakthroughs, the sheer volume of RS archives stretches the limits of human analysts and existing AI tools, as most models are: i) limited to single data modalities; ii) task-specific; iii) heavily reliant on labeled data. The emerging Foundation Models (FMs) have the potential to address these limitations. Trained on vast unlabeled datasets through self-supervised learning, FMs enable generic feature extraction that facilitate specialization to a wide variety of downstream tasks. This paper describes a vision towards an FM for multimodal Earth Observation data (FM4EO), discussing key building blocks and open challenges. We put particular emphasis on multimodal reasoning, a topic underexplored in EO. Our ultimate goal is a practical path toward FM4EO with capacity to unlock breakthroughs in few-shot learning scenarios, multimodal geographic knowledge integration, synthesis, and hypothesis generation.

Original language	English
Title of host publication	IGARSS 2023 - 2023 IEEE International Geoscience and Remote Sensing Symposium, Proceedings
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	1237-1240
Number of pages	4
ISBN (Electronic)	9798350320107
DOIs	https://doi.org/10.1109/IGARSS52108.2023.10282966
State	Published - 2023
Externally published	Yes
Event	2023 IEEE International Geoscience and Remote Sensing Symposium, IGARSS 2023 - Pasadena, United States Duration: Jul 16 2023 → Jul 21 2023

Publication series

Name	International Geoscience and Remote Sensing Symposium (IGARSS)
Volume	2023-July

Conference

Conference	2023 IEEE International Geoscience and Remote Sensing Symposium, IGARSS 2023
Country/Territory	United States
City	Pasadena
Period	07/16/23 → 07/21/23

Funding

This manuscript has been authored by UT-Battelle, LLC, under contract DE-AC05-00OR22725 with the US Department of Energy (DOE). The US government retains and the publisher, by accepting the article for publication, acknowledges that the US government retains a nonexclusive, paid-up, irrevocable, worldwide license to publish or reproduce the published form of this manuscript, or allow others to do so, for US government purposes. DOE will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan (http://energy.gov/downloads/doe-public-access-plan).

Keywords

Foundation Model
earth observation
multimodal reasoning
remote sensing
self-supervision

Access to Document

10.1109/IGARSS52108.2023.10282966

Cite this

Dias, P., Potnis, A., Guggilam, S., Yang, L., Tsaris, A., Medeiros, H., & Lunga, D. (2023). An Agenda for Multimodal Foundation Models for Earth Observation. In IGARSS 2023 - 2023 IEEE International Geoscience and Remote Sensing Symposium, Proceedings (pp. 1237-1240). (International Geoscience and Remote Sensing Symposium (IGARSS); Vol. 2023-July). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/IGARSS52108.2023.10282966

@inproceedings{b5547ce818ab4c1eaf170a8e53269f27,

title = "An Agenda for Multimodal Foundation Models for Earth Observation",

abstract = "Archives of remote sensing (RS) data are increasing swiftly as new sensing modalities with enhanced spatiotemporal resolution become operational. While promising new breakthroughs, the sheer volume of RS archives stretches the limits of human analysts and existing AI tools, as most models are: i) limited to single data modalities; ii) task-specific; iii) heavily reliant on labeled data. The emerging Foundation Models (FMs) have the potential to address these limitations. Trained on vast unlabeled datasets through self-supervised learning, FMs enable generic feature extraction that facilitate specialization to a wide variety of downstream tasks. This paper describes a vision towards an FM for multimodal Earth Observation data (FM4EO), discussing key building blocks and open challenges. We put particular emphasis on multimodal reasoning, a topic underexplored in EO. Our ultimate goal is a practical path toward FM4EO with capacity to unlock breakthroughs in few-shot learning scenarios, multimodal geographic knowledge integration, synthesis, and hypothesis generation.",

keywords = "Foundation Model, earth observation, multimodal reasoning, remote sensing, self-supervision",

author = "Philipe Dias and Abhishek Potnis and Sreelekha Guggilam and Lexie Yang and Aristeidis Tsaris and Henry Medeiros and Dalton Lunga",

note = "Publisher Copyright: {\textcopyright} 2023 IEEE.; 2023 IEEE International Geoscience and Remote Sensing Symposium, IGARSS 2023 ; Conference date: 16-07-2023 Through 21-07-2023",

year = "2023",

doi = "10.1109/IGARSS52108.2023.10282966",

language = "English",

series = "International Geoscience and Remote Sensing Symposium (IGARSS)",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "1237--1240",

booktitle = "IGARSS 2023 - 2023 IEEE International Geoscience and Remote Sensing Symposium, Proceedings",

}

Dias, P , Potnis, A, Guggilam, S, Yang, L, Tsaris, A, Medeiros, H & Lunga, D 2023, An Agenda for Multimodal Foundation Models for Earth Observation. in IGARSS 2023 - 2023 IEEE International Geoscience and Remote Sensing Symposium, Proceedings. International Geoscience and Remote Sensing Symposium (IGARSS), vol. 2023-July, Institute of Electrical and Electronics Engineers Inc., pp. 1237-1240, 2023 IEEE International Geoscience and Remote Sensing Symposium, IGARSS 2023, Pasadena, United States, 07/16/23. https://doi.org/10.1109/IGARSS52108.2023.10282966

An Agenda for Multimodal Foundation Models for Earth Observation. / Dias, Philipe ; Potnis, Abhishek; Guggilam, Sreelekha et al.
IGARSS 2023 - 2023 IEEE International Geoscience and Remote Sensing Symposium, Proceedings. Institute of Electrical and Electronics Engineers Inc., 2023. p. 1237-1240 (International Geoscience and Remote Sensing Symposium (IGARSS); Vol. 2023-July).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - An Agenda for Multimodal Foundation Models for Earth Observation

AU - Dias, Philipe

AU - Potnis, Abhishek

AU - Guggilam, Sreelekha

AU - Yang, Lexie

AU - Tsaris, Aristeidis

AU - Medeiros, Henry

AU - Lunga, Dalton

PY - 2023

Y1 - 2023

N2 - Archives of remote sensing (RS) data are increasing swiftly as new sensing modalities with enhanced spatiotemporal resolution become operational. While promising new breakthroughs, the sheer volume of RS archives stretches the limits of human analysts and existing AI tools, as most models are: i) limited to single data modalities; ii) task-specific; iii) heavily reliant on labeled data. The emerging Foundation Models (FMs) have the potential to address these limitations. Trained on vast unlabeled datasets through self-supervised learning, FMs enable generic feature extraction that facilitate specialization to a wide variety of downstream tasks. This paper describes a vision towards an FM for multimodal Earth Observation data (FM4EO), discussing key building blocks and open challenges. We put particular emphasis on multimodal reasoning, a topic underexplored in EO. Our ultimate goal is a practical path toward FM4EO with capacity to unlock breakthroughs in few-shot learning scenarios, multimodal geographic knowledge integration, synthesis, and hypothesis generation.

AB - Archives of remote sensing (RS) data are increasing swiftly as new sensing modalities with enhanced spatiotemporal resolution become operational. While promising new breakthroughs, the sheer volume of RS archives stretches the limits of human analysts and existing AI tools, as most models are: i) limited to single data modalities; ii) task-specific; iii) heavily reliant on labeled data. The emerging Foundation Models (FMs) have the potential to address these limitations. Trained on vast unlabeled datasets through self-supervised learning, FMs enable generic feature extraction that facilitate specialization to a wide variety of downstream tasks. This paper describes a vision towards an FM for multimodal Earth Observation data (FM4EO), discussing key building blocks and open challenges. We put particular emphasis on multimodal reasoning, a topic underexplored in EO. Our ultimate goal is a practical path toward FM4EO with capacity to unlock breakthroughs in few-shot learning scenarios, multimodal geographic knowledge integration, synthesis, and hypothesis generation.

KW - Foundation Model

KW - earth observation

KW - multimodal reasoning

KW - remote sensing

KW - self-supervision

UR - http://www.scopus.com/inward/record.url?scp=85178357178&partnerID=8YFLogxK

U2 - 10.1109/IGARSS52108.2023.10282966

DO - 10.1109/IGARSS52108.2023.10282966

M3 - Conference contribution

AN - SCOPUS:85178357178

T3 - International Geoscience and Remote Sensing Symposium (IGARSS)

SP - 1237

EP - 1240

BT - IGARSS 2023 - 2023 IEEE International Geoscience and Remote Sensing Symposium, Proceedings

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 2023 IEEE International Geoscience and Remote Sensing Symposium, IGARSS 2023

Y2 - 16 July 2023 through 21 July 2023

ER -

Dias P , Potnis A, Guggilam S, Yang L, Tsaris A, Medeiros H et al. An Agenda for Multimodal Foundation Models for Earth Observation. In IGARSS 2023 - 2023 IEEE International Geoscience and Remote Sensing Symposium, Proceedings. Institute of Electrical and Electronics Engineers Inc. 2023. p. 1237-1240. (International Geoscience and Remote Sensing Symposium (IGARSS)). doi: 10.1109/IGARSS52108.2023.10282966

An Agenda for Multimodal Foundation Models for Earth Observation

Abstract

Publication series

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this