Joint Classification of Hyperspectral and LiDAR Data Using a Hierarchical CNN and Transformer
- Authors
- Zhao, G.[Zhao, G.]; Ye, Q.[Ye, Q.]; Sun, L.[Sun, L.]; Wu, Z.[Wu, Z.]; Pan, C.[Pan, C.]; Jeon, B.[Jeon, B.]
- Issue Date
- 2023
- Publisher
- Institute of Electrical and Electronics Engineers Inc.
- Keywords
- Convolutional neural network (CNN); hyperspectral image (HSI); joint classification; light detection and ranging (LiDAR) data; tokenization; transformer
- Citation
- IEEE Transactions on Geoscience and Remote Sensing, v.61
- Indexed
- SCIE
SCOPUS
- Journal Title
- IEEE Transactions on Geoscience and Remote Sensing
- Volume
- 61
- URI
- https://scholarworks.bwise.kr/skku/handle/2021.sw.skku/103530
- DOI
- 10.1109/TGRS.2022.3232498
- ISSN
- 0196-2892
- Abstract
- The joint use of multisource remote-sensing (RS) data for Earth observation missions has drawn much attention. Although the fusion of several data sources can improve the accuracy of land-cover identification, many technical obstacles, such as disparate data structures, irrelevant physical characteristics, and a lack of training data, exist. In this article, a novel dual-branch method, consisting of a hierarchical convolutional neural network (CNN) and a transformer network, is proposed for fusing multisource heterogeneous information and improving joint classification performance. First, by combining the CNN with a transformer, the proposed dual-branch network can significantly capture and learn spectral-spatial features from hyperspectral image (HSI) data and elevation features from light detection and ranging (LiDAR) data. Then, to fuse these two sets of data features, a cross-token attention (CTA) fusion encoder is designed in a specialty. The well-designed deep hierarchical architecture takes full advantage of the powerful spatial context information extraction ability of the CNN and the strong long-range dependency modeling ability of the transformer network based on the self-attention (SA) mechanism. Four standard datasets are used in experiments to verify the effectiveness of the approach. The experimental results reveal that the proposed framework can perform noticeably better than state-of-the-art methods. The source code of the proposed method will be available publicly at https://github.com/zgr6010/Fusion_HCT.git. © 1980-2012 IEEE.
- Files in This Item
- There are no files associated with this item.
- Appears in
Collections - Information and Communication Engineering > School of Electronic and Electrical Engineering > 1. Journal Articles
![qrcode](https://api.qrserver.com/v1/create-qr-code/?size=55x55&data=https://scholarworks.bwise.kr/skku/handle/2021.sw.skku/103530)
Items in ScholarWorks are protected by copyright, with all rights reserved, unless otherwise indicated.