Revisiting Near/Remote Sensing with Geospatial Attention

    Scott Workman,  Muhammad Usman Rafique,   Hunter Blanton,   Nathan Jacobs

    Paper   Supplemental   Talk

Overview

Abstract

This work addresses the task of overhead image segmentation when auxiliary ground-level images are available. Recent work has shown that performing joint inference over these two modalities, often called near/remote sensing, can yield significant accuracy improvements. Extending this line of work, we introduce the concept of geospatial attention, a geometry-aware attention mechanism that explicitly considers the geospatial relationship between the pixels in a ground-level image and a geographic location. We propose an approach for computing geospatial attention that incorporates geometric features and the appearance of the overhead and ground-level imagery. We introduce a novel architecture for near/remote sensing that is based on geospatial attention and demonstrate its use for five segmentation tasks. The results demonstrate that our method significantly outperforms the previous state-of-the-art methods.

Approach

We propose a high-level neural network architecture for the task of near/remote sensing. Our architecture, visualized in Figure 3, has three primary components. First, we extract features from each image modality. Next, we use geospatial attention to generate a spatially consistent, dense grid of geo-informative features from the set of nearby ground-level images. Finally, we fuse the dense ground-level feature map with the overhead image feature map and use that as input to a decoder that generates the segmentation output. All components are differentiable, enabling end-to-end optimization of the low-level feature extraction networks and the attention model for the given segmentation task

Please refer to the paper for more details.

Results

Recommended citation

@inproceedings{workman2022revisiting,
  title={Revisiting Near/Remote Sensing with Geospatial Attention},
  author={Workman, Scott and Rafique, M. Usman and Blanton, Hunter and Jacobs, Nathan},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year={2022}
}