Abstract
Image processing and computer vision have been a focus of researchers for decades in various application domains. This research is continuously rising with the rise of Artificial Intelligence in the fourth industrial revolution. One of the important digital image processing applications is to produce panorama images. The wide range of view a panorama image provides can be used in a variety range of applications which may include surveillance applications and remote robot operations. A panorama image is a combination of several individual natural looking images into a composite one to provide a wide field of view that may reach 360 degrees horizontally without any distortion. Wide-angle lenses provide a wide field of view, but using them alone does not necessarily make a panorama image. In this work the design and implementation of a wide-angle stereo vision system that suites many real-time applications is proposed. The system makes use of two wide-angle fisheye cameras where each camera covers around 170 degrees field of view. The horizontal angle between the cameras is 140 degrees. The cameras acquire the instantaneous overlapping images continuously and transmits them to a base station via a communication link. The base station calibrates, corrects, correlates and stitches the non-overlapping corrected images to a composite one. The resultant final image covers 310 degrees field of view. The system is of low computational complexity compared with previously implemented systems. It is tested on a laptop and on a standalone embedded computing device. The processing speed for the panorama image stitching including the correction of the fisheye barrel distortion on the laptop computer and the embedded computer is 11 fps, and 6 fps, respectively.
Similar content being viewed by others
References
Abughalieh KM, Sababha BH, Rawashdeh NA (2018) A video-based object detection and tracking system for weight sensitive UAVs. Multimed Tools Appl:1–19. https://doi.org/10.1007/s11042-018-6508-1
Al-Harasis R, Al-Zmaily E, Al-Bishawi H, Abu Shash J, Shreim M, Sababha BH (2015) Design and Implementation of an Autonomous UGV for the Twenty Second Intelligent Ground Vehicle Competition. The International Conference on Software Engineering, Mobile Computing and Media Informatics (SEMCMI 2015), Kuala Lumpur
Bechar A, Vigneault C (2017) Agricultural robots for field operations. Part 2: Operations and systems. Biosyst Eng 153:110–128
Bellas N, Chai SM, Dwyer M, Linzmeier D (2009) Real-time fisheye lens distortion correction using automatically generated streaming accelerators. In: Field Programmable Custom Computing Machines, 2009. FCCM'09. 17th IEEE Symposium on, Napa
Ben-Tzvi P, Xu X (2010) An embedded feature-based stereo vision system for autonomous mobile robots. In: Robotic and Sensors Environments (ROSE), 2010 IEEE International Workshop on, Phoenix
Chira IM, Chibulcutean A, Danescu RG (2010) Real-time detection of road markings for driving assistance applications. In: Computer Engineering and Systems (ICCES), 2010 International Conference on, Cairo
Driscoll Jr E, Morrow H, Steinhauer AJ, Lomax WC (2002) Method and apparatus for a panoramic camera to capture a 360 degree image. U.S. Patent Patent 6,459,451
Drulea M, Szakats I, Vatavu A, Nedevschi S (2014) Omnidirectional stereo vision using fisheye lenses. In: Intelligent Computer Communication and Processing (ICCP), 2014 IEEE International Conference on, Cluj Napoca, Romania
Fan D-P, Cheng M-M, Liu Y, Li T, Borji A (2017) Structure-measure: A new way to evaluate foreground maps. In: Proceedings of the IEEE International Conference on Computer Vision, Venice
Fan D-P, Gong C, Cao Y, Ren B, Cheng M-M, Borji A (2018) Enhanced-alignment Measure for Binary Foreground Map Evaluation. In: Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence (IJCAI-18), Stockholm
Gao L-F, Gai Y-X, Fu S (2007) Simultaneous localization and mapping for autonomous mobile robots using binocular stereo vision system. In: Mechatronics and Automation, 2007. ICMA 2007. International Conference on
Gao W, Shen S (2017) Dual-Fisheye Omnidirectional Stereo. IEEE/ RSJ International Conference on Intelligent Robots and system (IROS), Vancouver
Ho T, Schizas ID, Rao KR, Budagavi M (2017) 360-Degree Video Stitching for Dual-Fisheye Lens Cameras Based on Rigid Moving Least Squares. arXiv:1708.05922v1
Kim H, Chae E, Jo G, Paik J (2015) Fisheye lens-based surveillance camera for wide field-of-view monitoring. In: Consumer Electronics (ICCE), 2015 IEEE International Conference on, Las Vegas
Liu T-M, Ju C-C, Huang Y-H, Chang T-S, Yang K-M, Lin Y-T (2017) A 360-degree 4Kx2K Panorama Video Processing Over Smart-phones. IEEE International Conference on Consumer Electronics (ICCE)
Mathworks (2009) Interpolation-methods. Available: https://www.mathworks.com/help/vision/ug/interpolation-methods.html. [Accessed 6 July 2017]
National Instruments (2015) myRIO Student Embedded Device. Available: http://www.ni.com/en-lb/shop/select/myrio-student-embedded-device
National Instruments. LabVIEW. Available: http://www.ni.com/en-lb/shop/labview.html
T. Nishimoto and J. Yamaguchi (2007) Three dimensional measurement using fisheye stereo vision. In: SICE Annual Conference 2007, Takamatsu
Ohashi A, Tanaka Y, Masuyama G, Umeda K, Fukuda D, Ogata T, Narita T, Kaneko S, Uchida Y, Irie K (2016) Fisheye stereo camera using equirectangular images. In: Mechatronics (MECATRONICS)/17th International Conference on Research and Education in Mechatronics (REM), 2016 11th France-Japan & 9th Europe-Asia Congress on, Compiegne
Polydoros AS, Nalpantidis L (2017) Survey of model-based reinforcement learning: Applications on robotics. J Intell Robot Syst 86(2):153–173
Rawashdeh NA, Rawashdeh OA, Sababha BH (2017) Vision-based sensing of UAV attitude and altitude from downward in-flight images. J Vib Control 23(5):827–841
Rawashdeh O, Sababha B (2011) An Image-Processing-Based Gimbal System Using Fisheye Video. Computer Technology and Application Journal 2(2):85–93
Rawashdeh OA, Yang HC, AbouSleiman RD, Sababha BH (2009) Microraptor: A low-cost autonomous quadrotor system. In: ASME 2009 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
Ray LA, Schauffele CN, VanSprewenburg B (2000) Method and apparatus for capturing panoramic images with range data. U.S. Patent Patent 6,023,588
Sababha BH, Al Zu'bi HM, Rawashdeh OA (2015) A rotor-tilt-free tricopter UAV: design, modelling, and stability control. International Journal of Mechatronics and Automation 5(2–3):107–113
Sawalmeh HK, Bjanthala HE, Al-Lahham MM, Sababha BH (2015) A Surveillance 3D Hand-Tracking-Based Tele-Operated UGV. In: The 6th International Conference on Information and Communication Systems (ICICS2015), Amman
Semwal VB, Bhushan A, Nandi GC (2013) Study of humanoid Push recovery based on experiments. In: Control, Automation, Robotics and Embedded Systems (CARE), 2013 International Conference on
Semwal VB, Chakraborty P, Nandi GC (2015) Less computationally intensive fuzzy logic (type-1)-based controller for humanoid push recovery. Robot Auton Syst 63:122–135, 2015
Semwal VB, Gaud N, Nandi G (2017) Human Gait State Prediction Using Cellular Automata and Classification Using ELM. In: MISP-2017
Semwal VB, Katiyar SA, Chakraborty R, Nandi GC (2015) Biologically-inspired push recovery capable bipedal locomotion modeling through hybrid automata. Robot Auton Syst 70:181–190, 2015
Semwal VB, Mondal K, Nandi GC (2017) Robust and accurate feature selection for humanoid push recovery and classification: deep learning approach. Neural Comput & Applic 28(3):565–574
Semwal VB, Nandi GC (2016) Generation of joint trajectories using hybrid automate-based model: a rocking block-based approach. IEEE Sensors J 16(12):5805–5816
Semwal VB, Raj M, Nandi GC (2015) Biometric gait identification based on a multilayer perceptron. Robot Auton Syst 65:65–75, 2016
Silva LC, Petraglia MR, Petraglia A (2004) A robust method for camera calibration and 3-D reconstruction for stereo vision systems. In: Proceedings of the 12th European Signal Processing Conference, Vienna
Sinclair MJ, Simmons LE (2003) Panoramic Digital Camera System and Method. US Patent Patent 6,545,701 B2
Takano T, Ono S, Matsushita Y, Kawasaki H, Ikeuchi K (2015) Super resolution of fisheye images captured by on-vehicle camera for visibility support. In: Vehicular Electronics and Safety (ICVES), 2015 IEEE International Conference on, Yokohama
Wachtel RA, Keable J, Paulson R (2007) Digital imaging system using overlapping images to formulate a seamless composite image and implemented using either a digital imaging sensor array. U.S. Patent Patent 7,215,364
Wang W, Yan J, Xu N, Wang Y, Hsu F-H (2015) Real-time high-quality stereo vision system in FPGA. IEEE Transactions on Circuits and Systems for Video Technology 25(10):1696–1708
Wei J, Li C-F, Hu S-M, Martin RR, Tai C-L (2012) Fisheye video correction. IEEE Trans Vis Comput Graph 18(10):1771–1783
Zhang B, Liu N, Jiao Y, Li Y, Zhu J (2015) Research on image matching technology for the spherical stereo vision. In: Mechatronics and Automation (ICMA), 2015 IEEE International Conference on, Beijing
Zhao S, Zhang B, Li L, Cao Z (2010) Fisheye lens camera system calibration and localization error analysis. In: Computer Design and Applications (ICCDA), 2010 International Conference on, Qinhuangdao
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Al-Harasis, R., Sababha, B.H. On the design and implementation of a dual fisheye camera-based surveillance vision system. Multimed Tools Appl 78, 22667–22689 (2019). https://doi.org/10.1007/s11042-019-7501-z
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-019-7501-z