Multitask Learning for Roof Type and Material Segmentation toward Digital Twinning

Abstract

Accurate roof type and material segmentation is essential for urban planning, disaster management, and infrastructure maintenance. Current roof segmentation methods typically focus only on rooftop segmentation, roof type segmentation, or material segmentation separately. This study introduces a novel approach utilizing Unmanned Aerial Vehicle (UAV)based data for the simultaneous segmentation of roof types and materials through a multitask learning neural network model. The model follows an encoder-decoder architecture. The encoder serves as the feature extractor, while the two decoders handle feature decomposition, projection, and prediction. The backbone architecture is a 32-layer High-Resolution Network, pretrained on the PASCAL VOC 2012 and SBD datasets which is performing as the share feature extractor for both heads. Parallel heads for roof type and material segmentation were designed. Dependency connections were implemented to share information between heads. Drone operations were conducted over Missouri University of Science and Technology campus buildings using DJI UAVs. Four roofing materials and two roof types were defined based on campus buildings. Experts with civil engineering knowledge annotated the images at the pixel level. An ablation study was conducted with changes in the dependency connections. All the models performed very well while the model with both way connections demonstrated the most robust performance. In contrast to traditional roof data collection, which may take hours for a few roofs, this model can process multiple roof images in just a second. This research was part of a broader project focused on developing multifunctional models within the digital twin framework for the built environment.

Department(s)

Computer Science

Second Department

Civil, Architectural and Environmental Engineering

Comments

U.S. Department of Transportation, Grant 69A3551747126

Keywords and Phrases

automatic rooftop segmentation; convolutional neural network; digital twin; Multitask learning; roof type segmentation; roofing material segmentation

International Standard Book Number (ISBN)

978-151068656-4

International Standard Serial Number (ISSN)

1996-756X; 0277-786X

Document Type

Article - Conference proceedings

Document Version

Citation

File Type

text

Language(s)

English

Rights

© 2025 Society of Photo-Optical Instrumentation Engineers (SPIE), All rights reserved.

Publication Date

01 Jan 2025

Link to Full Text

Share

 
COinS