OReole-FM: successes and challenges toward billion-parameter foundation models for high-resolution satellite imagery

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

While the pretraining of Foundation Models (FMs) for remote sensing (RS) imagery is on the rise, models remain restricted to a few hundred million parameters. Scaling models to billions of parameters has been shown to yield unprecedented benefits including emergent abilities, but requires data scaling and computing resources typically not available outside industry R&D labs. In this work, we pair high-performance computing resources including Frontier supercomputer, America’s first exascale system, and high-resolution optical RS data to pretrain billion-scale FMs. Our study assesses performance of different pretrained variants of vision Transformers across image classification, semantic segmentation and object detection benchmarks, which highlight the importance of data scaling for effective model scaling. Moreover, we discuss construction of a novel TIU pretraining dataset, model initialization, with data and pretrained models intended for public release. By discussing technical challenges and details often lacking in the related literature, this work is intended to offer best practices to the geospatial community toward efficient training and benchmarking of larger FMs.

Original languageEnglish
Title of host publication32nd ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems, ACM SIGSPATIAL 2024
EditorsMario A. Nascimento, Li Xiong, Andreas Zufle, Yao-Yi Chiang, Ahmed Eldawy, Peer Kroger
PublisherAssociation for Computing Machinery, Inc
Pages597-600
Number of pages4
ISBN (Electronic)9798400711077
DOIs
StatePublished - Nov 22 2024
Event32nd ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems, ACM SIGSPATIAL 2024 - Atlanta, United States
Duration: Oct 29 2024Nov 1 2024

Publication series

Name32nd ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems, ACM SIGSPATIAL 2024

Conference

Conference32nd ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems, ACM SIGSPATIAL 2024
Country/TerritoryUnited States
CityAtlanta
Period10/29/2411/1/24

Keywords

  • Earth Observation
  • Foundation Models
  • High-resolution satellite imagery
  • Remote Sensing
  • Self-supervised learning

Fingerprint

Dive into the research topics of 'OReole-FM: successes and challenges toward billion-parameter foundation models for high-resolution satellite imagery'. Together they form a unique fingerprint.

Cite this