ECHOVIT: Vision Transformers Using Fast-And-Slow Time Embeddings

Oluwanisola Ibikunle; Debvrat Varshney; Jilu Li; Maryam Rahnemoonfar; John Paden

doi:10.1109/IGARSS52108.2023.10281822

ECHOVIT: Vision Transformers Using Fast-And-Slow Time Embeddings

Oluwanisola Ibikunle, Debvrat Varshney, Jilu Li, Maryam Rahnemoonfar, John Paden

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

1 Scopus citations

Abstract

This paper details the preliminary efforts of applying the deep learning transformer architecture to automatically track annual layer stratigraphy in echogram images obtained from mapping near-surface ice layers using airborne radars. Following the success of the transformer architecture in the natural language processing and computer vision communities, we explore a variant termed Echogram Vision Transformer (EchoViT) on the radar echogram layer tracking (RELT) problem. The proposed approach divides the echogram images into patches using different schemes inspired by tokenization methods in natural language processing. We then apply a soft-attention mechanism to model interdependencies between the patches, capturing spatiotemporal stratigraphic information. Experiments conducted on the CREED dataset demonstrate the superiority of transformer-based architectures over existing convolutional-based architectures. Furthermore, the EchoViT fast-time and EchoViT slow-time patchifying schemes achieved precise tracking of the layers with submeter MAE of 3.39 and 3.55, respectively, while the use of cropped patches led to suboptimal results.

Original language	English
Title of host publication	IGARSS 2023 - 2023 IEEE International Geoscience and Remote Sensing Symposium, Proceedings
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	8162-8165
Number of pages	4
ISBN (Electronic)	9798350320107
DOIs	https://doi.org/10.1109/IGARSS52108.2023.10281822
State	Published - 2023
Externally published	Yes
Event	2023 IEEE International Geoscience and Remote Sensing Symposium, IGARSS 2023 - Pasadena, United States Duration: Jul 16 2023 → Jul 21 2023

Publication series

Name	International Geoscience and Remote Sensing Symposium (IGARSS)
Volume	2023-July

Conference

Conference	2023 IEEE International Geoscience and Remote Sensing Symposium, IGARSS 2023
Country/Territory	United States
City	Pasadena
Period	07/16/23 → 07/21/23

Funding

Center for Remote Sensing and Integrated Systems (CReSIS) is the new name of the Center for Remote Sensing and Ice Sheets (CReSIS). This work is supported by NSF IIS-1838236 and ACI-1443054.

Keywords

deep learning
EchoViT
soft-attention
transformer
ViT

Access to Document

10.1109/IGARSS52108.2023.10281822

Cite this

Ibikunle, O., Varshney, D., Li, J., Rahnemoonfar, M., & Paden, J. (2023). ECHOVIT: Vision Transformers Using Fast-And-Slow Time Embeddings. In IGARSS 2023 - 2023 IEEE International Geoscience and Remote Sensing Symposium, Proceedings (pp. 8162-8165). (International Geoscience and Remote Sensing Symposium (IGARSS); Vol. 2023-July). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/IGARSS52108.2023.10281822

@inproceedings{72d29d516fe44cb0a1946d3f5ded005d,

title = "ECHOVIT: Vision Transformers Using Fast-And-Slow Time Embeddings",

abstract = "This paper details the preliminary efforts of applying the deep learning transformer architecture to automatically track annual layer stratigraphy in echogram images obtained from mapping near-surface ice layers using airborne radars. Following the success of the transformer architecture in the natural language processing and computer vision communities, we explore a variant termed Echogram Vision Transformer (EchoViT) on the radar echogram layer tracking (RELT) problem. The proposed approach divides the echogram images into patches using different schemes inspired by tokenization methods in natural language processing. We then apply a soft-attention mechanism to model interdependencies between the patches, capturing spatiotemporal stratigraphic information. Experiments conducted on the CREED dataset demonstrate the superiority of transformer-based architectures over existing convolutional-based architectures. Furthermore, the EchoViT fast-time and EchoViT slow-time patchifying schemes achieved precise tracking of the layers with submeter MAE of 3.39 and 3.55, respectively, while the use of cropped patches led to suboptimal results.",

keywords = "deep learning, EchoViT, soft-attention, transformer, ViT",

author = "Oluwanisola Ibikunle and Debvrat Varshney and Jilu Li and Maryam Rahnemoonfar and John Paden",

note = "Publisher Copyright: {\textcopyright} 2023 IEEE.; 2023 IEEE International Geoscience and Remote Sensing Symposium, IGARSS 2023 ; Conference date: 16-07-2023 Through 21-07-2023",

year = "2023",

doi = "10.1109/IGARSS52108.2023.10281822",

language = "English",

series = "International Geoscience and Remote Sensing Symposium (IGARSS)",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "8162--8165",

booktitle = "IGARSS 2023 - 2023 IEEE International Geoscience and Remote Sensing Symposium, Proceedings",

}

Ibikunle, O, Varshney, D, Li, J, Rahnemoonfar, M & Paden, J 2023, ECHOVIT: Vision Transformers Using Fast-And-Slow Time Embeddings. in IGARSS 2023 - 2023 IEEE International Geoscience and Remote Sensing Symposium, Proceedings. International Geoscience and Remote Sensing Symposium (IGARSS), vol. 2023-July, Institute of Electrical and Electronics Engineers Inc., pp. 8162-8165, 2023 IEEE International Geoscience and Remote Sensing Symposium, IGARSS 2023, Pasadena, United States, 07/16/23. https://doi.org/10.1109/IGARSS52108.2023.10281822

ECHOVIT: Vision Transformers Using Fast-And-Slow Time Embeddings. / Ibikunle, Oluwanisola; Varshney, Debvrat; Li, Jilu et al.
IGARSS 2023 - 2023 IEEE International Geoscience and Remote Sensing Symposium, Proceedings. Institute of Electrical and Electronics Engineers Inc., 2023. p. 8162-8165 (International Geoscience and Remote Sensing Symposium (IGARSS); Vol. 2023-July).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - ECHOVIT

T2 - 2023 IEEE International Geoscience and Remote Sensing Symposium, IGARSS 2023

AU - Ibikunle, Oluwanisola

AU - Varshney, Debvrat

AU - Li, Jilu

AU - Rahnemoonfar, Maryam

AU - Paden, John

PY - 2023

Y1 - 2023

N2 - This paper details the preliminary efforts of applying the deep learning transformer architecture to automatically track annual layer stratigraphy in echogram images obtained from mapping near-surface ice layers using airborne radars. Following the success of the transformer architecture in the natural language processing and computer vision communities, we explore a variant termed Echogram Vision Transformer (EchoViT) on the radar echogram layer tracking (RELT) problem. The proposed approach divides the echogram images into patches using different schemes inspired by tokenization methods in natural language processing. We then apply a soft-attention mechanism to model interdependencies between the patches, capturing spatiotemporal stratigraphic information. Experiments conducted on the CREED dataset demonstrate the superiority of transformer-based architectures over existing convolutional-based architectures. Furthermore, the EchoViT fast-time and EchoViT slow-time patchifying schemes achieved precise tracking of the layers with submeter MAE of 3.39 and 3.55, respectively, while the use of cropped patches led to suboptimal results.

AB - This paper details the preliminary efforts of applying the deep learning transformer architecture to automatically track annual layer stratigraphy in echogram images obtained from mapping near-surface ice layers using airborne radars. Following the success of the transformer architecture in the natural language processing and computer vision communities, we explore a variant termed Echogram Vision Transformer (EchoViT) on the radar echogram layer tracking (RELT) problem. The proposed approach divides the echogram images into patches using different schemes inspired by tokenization methods in natural language processing. We then apply a soft-attention mechanism to model interdependencies between the patches, capturing spatiotemporal stratigraphic information. Experiments conducted on the CREED dataset demonstrate the superiority of transformer-based architectures over existing convolutional-based architectures. Furthermore, the EchoViT fast-time and EchoViT slow-time patchifying schemes achieved precise tracking of the layers with submeter MAE of 3.39 and 3.55, respectively, while the use of cropped patches led to suboptimal results.

KW - deep learning

KW - EchoViT

KW - soft-attention

KW - transformer

KW - ViT

UR - http://www.scopus.com/inward/record.url?scp=85178377905&partnerID=8YFLogxK

U2 - 10.1109/IGARSS52108.2023.10281822

DO - 10.1109/IGARSS52108.2023.10281822

M3 - Conference contribution

AN - SCOPUS:85178377905

T3 - International Geoscience and Remote Sensing Symposium (IGARSS)

SP - 8162

EP - 8165

BT - IGARSS 2023 - 2023 IEEE International Geoscience and Remote Sensing Symposium, Proceedings

PB - Institute of Electrical and Electronics Engineers Inc.

Y2 - 16 July 2023 through 21 July 2023

ER -

Ibikunle O, Varshney D, Li J, Rahnemoonfar M, Paden J. ECHOVIT: Vision Transformers Using Fast-And-Slow Time Embeddings. In IGARSS 2023 - 2023 IEEE International Geoscience and Remote Sensing Symposium, Proceedings. Institute of Electrical and Electronics Engineers Inc. 2023. p. 8162-8165. (International Geoscience and Remote Sensing Symposium (IGARSS)). doi: 10.1109/IGARSS52108.2023.10281822

ECHOVIT: Vision Transformers Using Fast-And-Slow Time Embeddings

Abstract

Publication series

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this