Semantic Background Subtraction

Summary

We introduce the notion of semantic background subtraction, a novel framework for motion detection in video sequences. The key innovation consists to leverage object-level semantics to address the variety of challenging scenarios for background subtraction. Our framework combines the information of a semantic segmentation algorithm, expressed by a probability for each pixel, with the output of any background subtraction algorithm to reduce false positive detections produced by illumination changes, dynamic backgrounds, strong shadows, and ghosts. In addition, it maintains a fully semantic background model to improve the detection of camouflaged foreground objects. Experiments led on the CDNet dataset show that we managed to improve, significantly, almost all background subtraction algorithms of the CDNet leaderboard, and reduce the mean overall error rate of all the 34 algorithms (resp. of the best 5 algorithms) by roughly 50% (resp. 20%). Paper in PDF
Please note that our method is protected by a pending patent entitled “Foreground and background detection method” (see [5, 4, 6]).
Note also that the algorithm was also used for background generation (see [1]).
Keywords: background subtraction, change detection, semantic segmentation, scene labeling, classification.
Table of Contents
Subsection: Source code in C/C++
Subsection: Semantic segmentation maps
Subsection: Processing flow-chart of our method
Subsection: Credits

Source code in C/C++

Our code uses OpenCV. We provide the full code at the following address:
https://orbi.uliege.be/bitstream/2268/213419/3/SemanticBGS-Code.zip

Semantic segmentation maps

The semantic segmentation maps were obtained with PSPNet [2]. There are downloadable for each video of the CDNet dataset [7] from the table below.
Category Videos
baseline highway office pedestrians PETS2006
cameraJitter badminton boulevard sidewalk traffic
dynamicBackground boats canoe fall fountain01 fountain02 overpass
intermittentObjectMotion abandonedBox parking sofa streetLight tramstop winterDriveway
shadow backdoor bungalows busStation copyMachine cubicle peopleInShade
thermal corridor diningRoom lakeSide library park
badWeather blizzard skating snowFall wetSnow
lowFramerate port_0_17fps tramCrossroad_1fps tunnelExit_0_35fps turnpike_0_5fps
nightVideos bridgeEntry busyBoulvard fluidHighway streetCornerAtNight tramStation winterStreet
PTZ continuousPan intermittentPan twoPositionPTZCam zoomInZoomOut
turbulence turbulence0 turbulence1 turbulence2 turbulence3

Processing flow-chart of our method

figure images/global-presentation.png

Credits

If you use our code, we would appreciate that you cite [3]: 
@inproceedings{Braham2017Semantic, 	
  title = {Semantic Background Subtraction}, 	
  author = {M. Braham and S. Pi\'erard and M. {Van Droogenbroeck}},	
  booktitle = {IEEE International Conference on Image Processing (ICIP)}, 	
  year = {2017}, 	
  month = {September}, 
  pages = {4552-4556},
  address = {Beijing, China},
  doi = {10.1109/ICIP.2017.8297144},	 	
  url = {http://hdl.handle.net/2268/213419} 
}

References

[1] B. Laugraud, S. Piérard, M. Van Droogenbroeck. LaBGen-P-Semantic: A First Step for Leveraging Semantic Segme ntation in Background Generation. Journal of Imaging, 4(7):86, 2018. URL http://dx.doi.org/10.3390/jimaging4070086.

[2] H. Zhao, J. Shi, X. Qi, X. Wang, J. Jia. Pyramid Scene Parsing Network. IEEE International Conference on Computer Vision and Pattern Recognition (CVPR):6230-6239, 2017. URL http://doi.org/10.1109/CVPR.2017.660.

[3] M. Braham, S. Piérard, M. Van Droogenbroeck. Semantic Background Subtraction. IEEE International Conference on Image Processing (ICIP):4552-4556, 2017. URL http://doi.org/10.1109/ICIP.2017.8297144.

[4] M. Van Droogenbroeck, M. Braham, S. Piérard. Foreground and background detection method, Chinese Patent Office, CN109389618A. 2017. URL https://patents.google.com/patent/CN109389618A/en.

[5] M. Van Droogenbroeck, M. Braham, S. Piérard. Foreground and background detection method, European Patent Office, EP 3438929 A1. 2017. URL https://patents.google.com/patent/EP3438929A1/.

[6] M. Van Droogenbroeck, M. Braham, S. Piérard. Foreground and background detection methodi, United States Patent and Trademark Office, US 2019/0043403 A1. 2018. URL https://patents.google.com/patent/US20190043403A1/.

[7] Y. Wang, P.-M. Jodoin, F. Porikli, J. Konrad, Y. Benezeth, P. Ishwar. CDnet 2014: An Expanded Change Detection Benchmark Dataset. IEEE International Conference on Computer Vision and Pattern Recognition Workshops (CVPRW):393-400, 2014. URL http://dx.doi.org/10.1109/CVPRW.2014.126.