Real-time semantic segmentation via sequential knowledge distillation

Deep model-based semantic segmentation has received ever increasing research focus in recent years. However, due to the complex model architectures, existing works are still unable to achieve high accuracy in real-time applications. In this paper, we propose a novel Sequential Prediction Network (termed SPNet) to seek a better trade-off between accuracy and efficiency. SPNet is also an end-to-end encoder-decoder architecture, which introduces a sequential prediction method to spread the contextual information from the low-level layers to the high-level layers. Besides, the proposed method is equipped with a stream Spatial Semantic and Edge Loss (termed SEL) and an adversarial network at multiple resolutions, which greatly improves the segmentation accuracy with a negligible increase in computation cost. To further uti-lize the extra unlabeled data, we present a knowledge distillation scheme to distill the structured knowl-edge from cumbersome to compact networks. Without using any pre-trained model, our method achieves state-of-the-art performance among exiting real-time segmentation models on several challenging data-sets. Impressively, on the Cityscapes test dataset, it obtains 75.8% mIoU at a speed of 61.2 FPS. (c) 2021 Published by Elsevier B.V.

Automated summary

This paper introduces a novel Sequential Prediction Network (SPNet) with Spatial Semantic and Edge Loss (SEL) and an adversarial network to achieve high segmentation accuracy in real-time applications. By utilizing a knowledge distillation scheme, the method effectively compresses structured knowledge from cumbersome networks, achieving promising results.