GlueStick: Robust Image Matching by Sticking Points and Lines Together

*Equal Contribution
1ETZ Zürich, 2Qualcomm XR Labs Europe, 3Microsoft Mixed Reality and AI Zurich Lab, 4Lund University
ICCV 2023


Line segments are powerful features complementary to points. They offer structural cues, robust to drastic viewpoint and illumination changes, and can be present even in texture-less areas. However, describing and matching them is more challenging compared to points due to partial occlusions, lack of texture, or repetitiveness. This paper introduces a new matching paradigm, where points, lines, and their descriptors are unified into a single wireframe structure. We propose GlueStick, a deep matching Graph Neural Network (GNN) that takes two wireframes from different images and leverages the connectivity information between nodes to better glue them together. In addition to the increased efficiency compared to independently matching points and lines, we also demonstrate a large boost of performance when leveraging the complementary nature of these two features in a single architecture. We show that our matching strategy outperforms the state-of-the-art approaches that independently match line segments and points for a wide variety of datasets and tasks.

Method Overview

My Happy SVG

Keypoints, dense descriptors, and lines are extracted from two images, and unified into a wireframe for each image (front-end). We then take the two corresponding wireframes, and enrich the features of their nodes via self, line, and cross-attention inside a Graph Neural Network (GNN). Finally, points and lines are matched separately via two dual-softmax modules.

Wireframe construction diagram

1. From Points and Lines to Wireframes

The first step in our pipeline is to build a wireframe using poins and lines:

  1. We use SuperPoint (SP) to predict keypoints and a dense descriptor map.
  2. We detect segments with the general-purpose LSD detector.
  3. Keypoints located close to line endpoints are redundant, so we remove SP keypoints that are within a small distance to existing line endpoints.
  4. We merge close-by endpoints, again with a distance threshold.

This process lifts the unstructured line cloud into an interconnected wireframe. After this step, each keypoint and line endpoint is represented as a node in the wireframe.

2. Attention-based Graph Neural Network

Detailed GNN diagram Graph Neural Network (GNN) architecture. Node features of the wireframe are enriched via several communication layers. Our proposed Line Message Passing exchanges information between neighboring nodes that are connected together.

3. Dual-Softmax for Points and Lines

Instead of frequent optimal transport assignation, we use a dual-softmax approach, which bring us higher efficiency with similar or better matching results.

GlueStick matches both point and line in a single forward pass. We propose to match nodes and lines separately through two independent dual-softmax assignments. On the one hand, all nodes (keypoints and line endpoints) are matched against each other using the final features output by the GNN. On the other hand, lines are matched in a similar way, except that each line is represented by its two endpoints features. To make the matching agnostic of the endpoint ordering, we take the maximum of the two configurations in the line assignation matrix.

Structura Similarity Diagram Line matching with order-agnostic endpoints. We consider the maximum score assignment between the two possible configurations of endpoint matching.

4. Ground Truth Generation

GT generation process

We use 3D data to train and evaluate our point and line matches. Obtaining matches between lines, even with 3D data is a tricky process. We determine if two lines are a correct match by sampling points along each line (cyan dots in the left image). We compute the 3D point locations in the world and re-project them back to the other image (green points). If a reasonable amount of green points fall close to a 2D segment in the second image, we generate a GT match!

GT Image 1
GT Image 2

Ground truth (GT) line assignations. Line segments with the same color are labeled as matches in our GT.

Some Cool Results 🚀

Line Matching Comparison


LBD matching results


SOLD2 matching results


LineTR matching results


L2D2 matching results


GlueStick matching results

Image Stitching

Examples of GlueStick matches on image pairs of SUN360. We provide the point and line matches, as well as the stitching of the two images using the resulting matches.

image Stitching results

Cite Us! 📝

                title={{GlueStick}: Robust Image Matching by Sticking Points and Lines Together},
                author={Pautrat, R{\'e}mi and Su{\'a}rez, Iago and Yu, Yifan and Pollefeys, Marc and Larsson, Viktor},
                booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},