Automated Building Footprint Extraction using Segment Anything Model 2 (SAM 2) in QGIS and Python

Automated Building Footprint Extraction using Segment Anything Model 2 (SAM 2) in QGIS and Python

Introduction

Implementing SAM 2 Building Footprint Extraction QGIS Python pipelines allows geospatial analysts and data scientists to achieve state-of-the-art vector extraction without training custom neural networks from scratch. In the rapidly evolving domain of Geographic Information Systems (GIS) and Remote Sensing, the extraction of vector building footprints from high-resolution aerial and satellite imagery has historically relied on task-specific deep learning architectures, such as U-Net, Mask R-CNN, or customized YOLO variants. While effective within specific training domains, these models demand extensive manual annotation datasets, rigorous hyperparameter tuning, and often suffer from severe domain shift when applied to unseen urban or rural topographies.
The introduction of Meta’s Segment Anything Model 2 (SAM 2) represents a monumental paradigm shift. As a promptable vision foundation model capable of zero-shot object segmentation across images and video streams, SAM 2 encodes rich spatial priors and hierarchical feature representations. Implementing SAM 2 Building Footprint Extraction QGIS Python pipelines allows geospatial analysts and data scientists to achieve state-of-the-art vector extraction without training custom neural networks from scratch. This comprehensive tutorial explores how to leverage SAM 2, Hugging Face transformers, rasterio, and geopandas to programmatically extract, polygonize, smooth, and export rooftop geometries directly into analysis-ready GIS formats.

Environment Setup & Prerequisites

Before diving into the code, you must configure a robust Python environment capable of handling heavy tensor computations and spatial coordinate transformations. It is strongly recommended to use an NVIDIA GPU with at least 12GB of VRAM and CUDA installed.

1. Creating the Conda Environment

Open your terminal and create a dedicated virtual environment:

Bash

conda create -n geoai-sam2 python=3.10 -y
conda activate geoai-sam2

2. Installing Core Dependencies

Install PyTorch (matching your CUDA version) alongside Hugging Face transformers, rasterio, geopandas, and ancillary spatial libraries:

Bash

pip install torch torchvision --index-url https://download.pytorch.org/whl/cu121
pip install transformers accelerate datasets
pip install rasterio geopandas shapely fiona scikit-image matplotlib
Verify your installation by running a quick python check to ensure CUDA is accessible by PyTorch:

Python

import torch
print(f"CUDA Available: {torch.cuda.is_available()}")
print(f"Device Name: {torch.cuda.get_device_name(0) if torch.cuda.is_available() else 'CPU'}")

Step-by-Step Coding Workflow

Step 1: Loading High-Resolution Orthophotos and Generating Image Embeddings

System architecture diagram illustrating Automated Building Footprint Extraction using Segment Anything Model 2 (SAM 2) in QGIS and Python

Geospatial rasters contain critical spatial referencing metadata—specifically the Affine transform and Coordinate Reference System (CRS)—that standard computer vision libraries ignore, which is especially vital when Optimizing Enterprise Web Maps with Cloud-Optimized GeoTIFF (COG) Architecture. We use rasterio to read the orthophoto while retaining its geospatial anchor.

Python

import rasterio
from rasterio.windows import Window
import numpy as np
import torch
from PIL import Image
from transformers import Sam2Model, Sam2Processor

# Load the georeferenced aerial orthophoto
raster_path = "data/nairobi_ortho_0.2m.tif"

with rasterio.open(raster_path) as src:
    meta = src.meta.copy()
    crs = src.crs
    transform = src.transform
    # Read RGB bands (assuming bands 1, 2, 3 correspond to R, G, B)
    img_array = src.read([1, 2, 3])
    # Reshape from (Bands, Height, Width) to (Height, Width, Bands) for PIL/Transformers
    img_array = np.moveaxis(img_array, 0, -1)
    
# Initialize SAM 2 Model and Processor from Hugging Face
model_id = "facebook/sam2.1-hiera-large"
device = "cuda" if torch.cuda.is_available() else "cpu"

print(f"Loading SAM 2 model ({model_id}) onto {device}...")
processor = Sam2Processor.from_pretrained(model_id)
model = Sam2Model.from_pretrained(model_id).to(device)

# Convert numpy array to PIL Image for the processor
pil_image = Image.fromarray(img_array.astype('uint8'))

Step 2: Passing Prompt Points or Bounding Boxes to Isolate Rooftop Structures

Infographic showing bounding box prompts and mask inference during Automated Building Footprint Extraction using Segment Anything Model 2 (SAM 2) in QGIS and Python

To extract building footprints in a zero-shot fashion, we can either feed bounding box priors (generated via coarse object detection) or prompt points located on rooftops. Below is an implementation utilizing bounding box prompts corresponding to detected industrial and residential structures.

Python

# Define bounding boxes [x_min, y_min, x_max, y_max] for target buildings in pixel coordinates
input_boxes = [
    [450, 320, 680, 510],
    [720, 610, 950, 800]
]

# Format inputs for SAM 2 processor
inputs = processor(
    images=pil_image,
    input_boxes=[input_boxes],
    return_tensors="pt"
).to(model.device)

# Perform inference without gradient tracking
with torch.no_grad():
    outputs = model(**inputs)

# Post-process masks back to original image size
masks = processor.post_process_masks(
    outputs.pred_masks.cpu(),
    inputs["original_sizes"].cpu()
)

# Extract the highest scoring binary mask tensor
# masks[0] shape: (num_boxes, num_masks_per_box, H, W)
pred_masks = masks[0][:, 0, :, :].numpy()
binary_mask = (pred_masks.sum(axis=0) > 0).astype(np.uint8)

Step 3: Extracting Binary Mask Arrays and Polygonizing Raster Contours

Once we obtain the aggregated binary mask array representing all segmented building footprints, we convert the pixel-level raster predictions into clean geometric vector features using rasterio.features.shapes.

Python

from rasterio.features import shapes
import geopandas as gpd
from shapely.geometry import shape

# Extract shapes (polygons and values) from the binary mask array
# We apply the raster's affine transform to map pixels to geospatial coordinates
shape_generator = shapes(
    binary_mask,
    transform=transform
)

# Convert generator output into a list of Shapely geometries and attributes
records = []
for geom, value in shape_generator:
    if value == 1:  # Keep foreground building features
        polygon = shape(geom)
        records.append({"geometry": polygon})

# Construct initial GeoDataFrame in native CRS
gdf = gpd.GeoDataFrame.from_features(records, crs=crs)
print(f"Extracted {len(gdf)} raw building polygons.")

Step 4: Smoothing and Converting Pixel Polygons into Clean GIS Shapefiles

Workflow diagram showing polygonization and smoothing stages for Automated Building Footprint Extraction using Segment Anything Model 2 (SAM 2) in QGIS and Python

Raw raster-derived polygons often exhibit “stair-stepping” artifacts due to pixel grid discretization. To ensure cartographic and topological integrity, we apply buffer operations and Douglas-Peucker simplification before exporting to a GeoPackage (.gpkg).

Python

from shapely.geometry import Polygon

def smooth_and_clean_polygon(poly, tolerance=0.5, buffer_dist=0.6):
    """
    Applies negative/positive buffering (morphological closing/opening equivalent)
    to fill small gaps, followed by Douglas-Peucker simplification.
    """
    if not poly.is_valid:
        poly = poly.buffer(0)
    # Morphological smoothing via buffering
    smoothed = poly.buffer(buffer_dist).buffer(-buffer_dist)
    # Simplify geometry
    simplified = smoothed.simplify(tolerance, preserve_topology=True)
    return simplified

# Apply cleaning function to all geometries
gdf["geometry"] = gdf["geometry"].apply(lambda p: smooth_and_clean_polygon(p, tolerance=0.4, buffer_dist=0.5))

# Filter out sliver polygons falling below minimum area threshold (e.g., 5 square meters)
gdf = gdf[gdf.geometry.area >= 5.0].reset_index(drop=True)

# Export to production GeoPackage layer for QGIS consumption
output_gpkg = "outputs/extracted_buildings_sam2.gpkg"
gdf.to_file(output_gpkg, layer="sam2_buildings", driver="GPKG")
print(f"Successfully exported cleaned vector footprints to {output_gpkg}")

Visualizing Results

SAM 2 Building Footprint Extraction QGIS Python

Integration and Advanced GeoAI Strategies

Integrating Python-based GeoAI workflows directly into QGIS streamlines the transition from raw raster data to vector management:
  1. PyQGIS Integration: You can wrap the script above into a custom QGIS Processing Algorithm script, allowing desktop users to run SAM 2 inference via the graphical Processing Toolbox interface.
  2. Anchor Detection Pipelines: To automate bounding box generation across city-scale datasets without manual annotation, pair SAM 2 with object detection models. For instance, initial anchor coordinates can be generated using YOLOv8 satellite object detection models trained specifically on overhead imagery.
  3. Large Raster Handling: For regional mosaics spanning gigabytes, implement windowed reading strategies as outlined in our Python rasterio tutorials to process imagery in manageable spatial tiles with overlapping buffers to eliminate edge artifacts.
  4. Cartographic Refinement: Standardize rooftop angles and enforce orthogonal right angles using specialized cartographic generalization guides and Python libraries like momepy or shapely envelope fitting.

Summary

The integration of Meta’s Segment Anything Model 2 (SAM 2) into Python and QGIS workflows bridges the gap between state-of-the-art computer vision and operational GIS analytics. By combining Hugging Face transformer pipelines with rasterio spatial referencing and geopandas vector processing, GIS professionals can rapidly execute zero-shot building footprint extraction, bypass the bottleneck of manual training dataset creation, and produce high-fidelity spatial data layers for urban planning, infrastructure management, and disaster risk assessment.