Open Access
Add to BibSonomy
Abstract
In this work, we demonstrate a prototype of a biomimetic multispectral curved compound eye camera (BMCCEC). In comparison with traditional multispectral imaging systems, the BMCCEC developed in this work has the distinct features of multi-spectral imaging on multiple targets in real time in an ultra-large field of view (FOV), which can be attributed to its biomimetic curved compound eye structure as well as the multispectral cluster network. Specifically, the BMCCEC has a total of 104 multispectral ommatidia and a FOV of 98°×98°, which is able to realize 7-band multispectral imaging with center wavelengths of 500 nm, 560 nm, 600 nm, 650 nm, 700 nm, 750 nm and 800 nm and a spectral resolution of 10 nm. A prototype of BMCCEC was then manufactured and multispectral imaging experiments were performed based on it. As a result, the red edge feature of the spectrum of green plants has been successfully obtained and retrieved with a good accuracy.
© 2021 Optical Society of America under the terms of the OSA Open Access Publishing Agreement
1. Introduction
A Compound eye camera is a kind of bionic optical imaging system based on the compound eye structure inspired by the biological compound eye of insects. The compound eye system consists of multiple ommatidia which are arranged on a free-shaped surface. Each ommatidium has its own photoreceptor so that it can image independently. The feature of the ommatidia structure has advantages of variable shape, large field of view (FOV) and low distortion on imaging. However, the structure of the compound eye of insects determines that the imaging resolution of the system is limited due to its rather small system focal length. Thus, the ommatidia of artificial compound eye (ACE) are usually designed as separate imaging channels, and the optical system is optimized with software for aberration correction. By now, several researches about ACE systems have been reported in order to take advantages of its features [1–7]. In our former researches, an optical relay system was proposed to transmit the image formed by the curved lens array to the planar sensor and improve the imaging quality. [8] The optical relay system corrects the deviations and improves the quality of the image, and thus maintains the advantages of compound eye system effectively.
Since each ommatidium of the compound eye in fact is a separate imaging channel, therefore each ommatidium can be tuned to have a different optical feature. Researchers have discovered that insects have specialized ommatidia in different parts of the compound eye, which aims to achieve different functions. In 1999, T. Labhart et al. discovered some insects use a specialized dorsal rim area of the compound eye to acquire the polarization of the sky for compass orientation. [9] And in 2009, Hiroko Awata et al. discovered that the Eastern Pale Clouded Yellow butterfly has different opsins in different parts of its compound eye, and therefore it is sensitive to light with different colors [10]. Inspired by the specialization of ommatidia appeared in insect compound eyes, the compound eye system can be designed with specialized ommatidia to acquire different kinds of information. For instance, one ommatidium can be modified for spectral imaging and the another ommatidium for polarized imaging.
Based on this theory, various multispectral compound eye cameras have been proposed. In 2004, a so called TOMBO system, i.e. Thin Observation Module by Bound Optics, was proposed by R. Shogenji et al. Multispectral imaging with seven wavebands ranging from 400 nm to 700 nm was demonstrated and the TOMBO system had a focal length of only 1.3 mm [11]. In 2010, the multispectral TOMBO system was improved by Keiichiro Kagawa et al to achieve a multispectral depth map acquisition based on the stereo camera distance measuring method. The focal length of the system was 2.35 mm and the pixel count per unit was 220 × 220, which means the FOV of a single unit was only 16°×16° [12]. In 2013, Jian Jin et al. proposed a planar multispectral ACE system by fabricating a planar micro-lens array on a multispectral filter with thermal reflow method. [13] In 2017, Jianwei Chen et al. fabricated a multispectral compound eye imaging system with hybrid imprinting method on a curved substrate. The system used a micro-lens array of 3 × 4 to achieve imaging in RGB and NIR channels, which means only four wavebands were available. [14] In 2020, Tetsuya Nakanishi et al applied an eight wavebands multispectral TOMBO system on an unmanned aerial vehicle (UAV) and target recognition was successfully demonstrated, the focal length of the airborne system was only 1.5 mm, and its field of view was only 50°×50° [15]. As can be seen, the multispectral TOMBO is based on the planar lens array and thus the FOV is small and the optical resolution is rather low as well. As a result, the ultra-large FOV feature of the natural insect compound eye is not utilized effectively. In 2021, we proposed a new design of the so called Multispectral Curved Compound Eye Camera (MCCEC) [16]. The MCCEC system was designed to be able to achieve a FOV of 120° with 7 wavebands multispectral imaging capability. However, the MCCEC system only achieves a short focal length of 0.4 mm, which means the imaging resolution is rather low and can’t meet the requirement on the high imaging resolution. More recently, we demonstrated a biomimetic curved compound-eye camera (BCCEC) with a rather high resolution with a longer system focal length of 5 mm but a smaller FOV of 98° [17].
In comparison with the traditional multispectral imaging camera design, which largely relies on motion parts like filter wheel, tunable filter or push broom components [18–22]. The multispectral imaging system based on curved ACE system has the advantages of larger FOV and more compact system structure. And the split channel design of the compound eye system can effectively prevent possible light crosstalk among spectral channels. However, most of the curved compound eye multispectral imaging systems are fabricated based on micro structures, which can only achieve a rather low imaging resolution with optical aberrations uncorrected.
Considering the requirement of large FOV and high-resolution in both spacial and spectral imaging, a biomimetic multispectral curved compound eye camera (BMCCEC) is proposed by combining the BCCEC system with narrow band multispectral filters array in our previous work [16]. Furthermore, we demonstrate the prototype of an even advanced BMCCEC system. The prototype advanced BMCCEC system is a modified version of the high resolution BCCEC system with 7 wavebands multispectral imaging capability with a spectral resolution of 10 nm. Comparing with the former MCCEC system, it improves the focal length of the system from 0.4 mm to 5 mm, in order to make the system able to work in long distance with a high special resolution. Due to the improvement of the system focal length, the maximum FOV is traded off to be smaller, i.e. 98°, to fit with the size of the CMOS imaging sensor. The detailed design and the assembly of the prototype BMCCEC are given in the paper. A calibration-free image reconstruction method designed for the compound eye image is proposed. The multispectral imaging experiment was also performed and the red edge feature of the spectrum of the green plants has been successfully obtained and retrieved with a good accuracy.
2. Theory of the multispectral compound eye system
2.1 Multispectral ommatidia arrangement theory
The working theory of the BMCCEC is based on a cluster of multispectral ommatidia shown in Fig. 1(a) which should be able to image the same target at the same time. As is shown, a cluster of multispectral ommatidia consists of 7 ommatidia integrated with 7 different narrow band filters with different central wavelengths of λ1 to λ7. It is arranged in a hexagonal way so that one ommatidium is surrounded by other 6 ommatidia. Since a cluster of multispectral ommatidia should image the same target, there must be overlap on FOV in between any two neighboring ommatidia. For the whole FOV of BMCCEC, there are many clusters of multispectral ommatidia and any ommatidium should be surrounded by other six ommatidia with different wavebands. To meet this requirement, a sphere projection method was employed to determine the distribution of ommatidia.
Fig. 1. The arrangement of the multispectral ommatidia of the BMCCEC: (a). the arrangement of a cluster of multispectral ommatidia and how it is arranged in the whole FOV, (b). the projection of the ommatidia on the spherical surface
The coordinate of the ommatidium in spherical coordinate system can be calculated by
2.2 Multispectral compound eye image reconstruction theory
To acquire the multispectral image of an object, the compound eye image has to be re-projected onto the object plane, which is a reverse of the imaging process. The re-projection method can be described as follows. Firstly, the separated sub-image formed by each ommatidium has to be transformed into the coordinate system of the first image plane of the BMCCEC. The first image plane of the BMCCEC is in fact on a spherical surface and its coordinate system can be defined by
For a vector pointing from the center of the curved compound surface to the object point, it can be represented by
3. Experiments
3.1 Prototype of BMCCEC
A prototype of BMCCEC was designed and fabricated based on above theory and is shown in Fig. 3. As can be seen from Fig. 3(a), the designed BMCCEC consists of three main parts: 1) a curved compound eye array that consists of 127 ommatidia fixed on a spherical shell with a radius of 68 mm, where 104 ommatidia are available for multispectral imaging 2) an optical relay system that transforms the curved focal image plane formed by the curved compound eye into a planar focal plane, 3) a Dalsa C5180M industrial camera with a total pixels of 5120×5120 and a pixel size of 4.5 µm×4.5 µm which is used as an image sensor.
Fig. 3. The prototype of BMCCEC: (a) the diagram of the mechanical structure of BMCCEC, (b) the photograph of the assembled prototype of BMCCEC. (c) the photograph of the multispectral curved compound eye.
In the design we use an optical relay subsystem consists of six glass lenses, the structure is shown in Fig. 3(a). The subsystem has a focal length of 14.4 mm. The FOV of the system is designed to be large enough to bring the image formed by all of the ommatidia to the CMOS focal plane. In the optical design process, the BMCCEC system is designed and optimized as a whole to correct the optical aberrations, which leads to a rather good MTF performance. One can find more information about the related optical system design in Ref. [17].
The main parameters of the prototype of BMCCEC are listed in Table 1.
Table 1. Specifications of the prototype BMCCEC
To achieve multispectral imaging, optical filters with a narrow band are attached to the front of the ommatidia as shown in Fig. 3(c). To fulfill the requirement of the system, interference narrowband filter based on Fabry-Perot interference principle is chosen and bought as a commercial product. The filter comes as a glass substrate with multilayer optical films deposited on both sides of which. To cover the red edge spectrum of the plant, central wavelengths of 500 nm, 560 nm, 600 nm, 650 nm, 700 nm, 750 nm and 800 nm are selected. The thickness of the filters is 2.2 mm. The full width at the half maximum (FWHM) for all 7 optical filters is 10 nm, which means a spectral resolution of 10 nm can be obtained for the multispectral imaging. The deviation of the central wavelength of the filters are less than ±2 nm.
While installing the multispectral filters, the mounts which hold the filter appear to interfere with each other at the edge of the hexagon array. This is because the mount space becomes limited at the edge of the curved metallic shell based on the current filter integration method. As the filters on the edge are preserved in installing procedure, this has no influence on the maximum FOV of the system. As a result, the BMCCEC has 104 effective ommatidia for multispectral imaging and the rest 23 ommatidia play the role of monochromatic imaging, as shown in Table 1.
To characterize the imaging quality of the prototype system, the modulation transfer function (MTF) test was performed. An OPTVIS-IR-600 MTF measuring instrument was employed, which has an indication error of less than 0.08. To measure the imaging quality of the whole FOV, the MTF values of central (0°) and edge (42°) ommatidia were tested. Table 2 lists the experimental MTF results of tangential and sagittal planes. As can be seen, the system has an average MTF of larger than 0.2 at 55 lp/mm, which achieves a fine imaging quality.
Table 2. Experimental MTF results of the prototype BMCCEC
3.2 Spectral calibration of BMCCEC
As described, the multispectral information of the target is reconstructed from the multispectral image formed by a cluster of multispectral ommatidia. However, there is a little bit difference on the FOV angle as well as the target distance for different ommatidia in the same cluster. The difference of the angle of incidence and the target distance will lead to the difference on the reflected light intensity for different ommatidia so that the spectral information becomes non-uniform. To address this issue, a uniform plane is chosen as the whiteboard reference to acquire the relative reflection ratio of the target for each ommatidium. For a working distance of 500 mm, a uniform plane with a size of 575 mm×575 mm was used. Two halogen bulbs were chosen as the light source to illuminate the target surface to achieve enough energy among the whole sampling spectrum range.
The circle detection method based on Hough transformation is used on the white board image to differentiate the images formed by different ommatidia. As is shown in Fig. 4, a total of 104 valid image units are clearly resolved and marked with red circles. These detected image units are then separated to seven spectrum groups. After that, all the pixel values in each circle unit are averaged as the reference reflection intensity of the ommatidium, which will be used to calculate the relative reflection ratio of the target in the following spectral imaging experiment.
3.3 Multispectral imaging experiment
After spectral calibration, the multispectral imaging capability of the prototype BMCCEC was demonstrated. In the experiment, targets in different FOVs are imaged and recognized based on their multi-spectrum. A green leaf plant and a plant model made with plastic materials are chosen as targets. After multispectral imaging, the image in the whole FOV containing all ommatidia units was reconstructed and is shown in Fig. 5(a). Based on the reconstructed image in the whole FOV, it is easy for one to locate the interesting targets. To retrieve sampled spectral image for every spectrum channel, the following steps were taken. Firstly, the reconstructed multispectral image in the whole FOV was differentiated and divided into seven spectral groups. Next, the multispectral images were reconstructed by using the projection method. Figure 5(b) shows the reconstructed images of all spectrum channels. As can be seen from Fig. 5(b), all seven pictures show different grayscales in the targets. Especially for green leaves, they become lighter obviously at infrared wavebands due to the higher reflection. Furthermore, the relative reflection spectrum curve can be retrieved by applying the spectral calibration coefficient.
Fig. 5. Multispectral imaging experiment results: (a) the reconstructed image in the whole FOV, (b) reconstructed image for each spectrum channel, (c) reconstructed multispectral 3D cube.
To verify the validity of the multispectral information, a fine spectrometer (ocean optics HR4000CG) with even higher spectral resolution of 1 nm was used to confirm the multispectral information. The measured reflection spectrum curves for the real plant is shown in Fig. 6. As can be seen, the curve measured with fine spectrometer fits quite well with that measured by the prototype BMCCEC.
Fig. 6. The reflection spectrum curves of the real plant obtained by both a fine spectrometer and the prototype BMCCEC
3.3.1 Multispectral imaging in large FOV
To test the uniformity of multispectral imaging in the entire FOV, the same target was put in the central and edge of the FOV respectively in the imaging experiment. The target chosen in the experiment is still green plant, which has large absorption at around 600 nm and a high reflection at the waveband ranging from 750-800 nm. Figures 7(a)-(b) show the multispectral images obtained in different FOVs. Figure 7(c) shows the retrieved reflection spectrum curve. As can be seen, two curves coincide well with an average relative error of less than 10%. Therefore, the prototype BMCCEC performs quite well in the whole FOV of 98°×98°.
Fig. 7. Multispectral imaging of the prototype BMCCEC in the whole FOV: (a) the reconstructed multispectral image in the central FOV, (b) the reconstructed multispectral image in the edge FOV, (c) the retrieved reflection spectrum curves for multispectral images in different FOVs.
3.3.2 Target recognition based on multispectral imaging
To demonstrate target recognition based on multispectral information acquired by BMCCEC, an experiment with plastic and real plants as targets was performed. The reconstructed images for both targets are shown in Figs. 8(a)-(b). The retrieved reflection spectrum curves are shown in Fig. 8(c). As can be seen clearly, though the fake plant target has a similar absorption band at around 600 nm to 650 nm, the feature of the red edge spectrum in infrared waveband ranges from 750 nm to 800 nm almost disappears. That is to say, the reflectance of the real plant is much higher than that of the fake plant in infrared waveband. Therefore, the prototype BMCCEC works quite well for target recognition.
Fig. 8. Target recognition experiment: (a) the reconstructed multispectral image of the real plant, (b) the reconstructed multispectral image of the fake plant, (c) the retrieved reflection spectrum curves for multispectral images of both targets.
4. Discussion and conclusion
In summary, the multispectral imaging by a prototype BMCCEC was demonstrated in this work. Based on the multispectral ommatidia arrangement theory and re-projection method, a prototype BMCCEC with 104 multispectral ommatidia and a FOV of 98° was designed and fabricated. A method for compound eye multispectral image reconstruction was also proposed. After calibration, the multispectral imaging experiment on real and fake plant targets by the prototype BMCCEC was performed in the whole FOV. The multispectral image was then reconstructed and the multi-spectral information was also retrieved. In comparison with the fine spectrometer, the prototype BMCCEC shows a good accuracy in spectrum measurement. Moreover, it shows a good uniformity in the entire FOV and performs well in multispectral target recognition.
Based on the prototype system and the result of the experiments, the BMCCEC system has great potential in multispectral imaging area. A comparison of the BMCCEC prototype and the state-of-the-art compound eye multispectral systems is listed in Table 3. In comparison with others, the BMCCEC prototype shows apparent advantages especially in terms of the field of view as well as the number of multispectral wavebands. Moreover, it can work at a real-time mode.
Table 3. The comparison of optical parameters of multispectral compound eye system
In comparison with the compound eyes of insects in nature, the BMCCEC system has a longer system focal length and thus a higher imaging resolution can be obtained in a longer working distance. The multispectral filter array with 7 different wavebands gains more spectral information than the natural specialized compound eye system, which normally has a few wavebands with specialized functions. Also, the system maintains the advantage offered by a curved compound eye structure, such as large FOV and specialized multi-aperture imaging without light crosstalk.
Also, the large baseline among the ommatidia presents the potential to measure the depth of the multispectral image. Of course, more work needs to be done on the calibration of the camera parameters of each imaging channel to improve the precision of the image reconstruction. We are planning to build a world coordinate based on the spatial position of the central ommatidium and calibrate the system with a self-identifying calibration board, either with coded tags or feature points. And a curved compound eye shell where filters are integrated onto the back of the ommatidia lenses will be designed to make it more impact. This work will be continuously concentrated on in the future.
Funding
National Natural Science Foundation of China (61975231).
Disclosures
The authors declare that there are no conflicts of interest related to this article.
Data availability
Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.
References
1. J. Duparre, D. Radtke, and A. Tunnermann, “Spherical artificial compound eye captures real images,” in Moems and Miniaturized Systems Vi, D. L. Dickensheets, B. P. Gogoi, and H. Schenk, eds. (Spie-Int Soc Optical Engineering, 2007).
2. L. Li and A. Y. Yi, “Development of a 3D artificial compound eye,” Opt. Express 18(17), 18125–18137 (2010). [CrossRef]
3. D. Floreano, R. Pericet-Camara, S. Viollet, F. Ruffier, A. Bruckner, R. Leitel, W. Buss, M. Menouni, F. Expert, R. Juston, M. K. Dobrzynski, G. L’Eplattenier, F. Recktenwald, H. A. Mallot, and N. Franceschini, “Miniature curved artificial compound eyes,” Proc. Natl. Acad. Sci. U. S. A. 110(23), 9267–9272 (2013). [CrossRef]
4. D. Wu, J. N. Wang, L. G. Niu, X. L. Zhang, S. Z. Wu, Q. D. Chen, L. P. Lee, and H. B. Sun, “Bioinspired Fabrication of High-Quality 3D Artificial Compound Eyes by Voxel-Modulation Femtosecond Laser Writing for Distortion-Free Wide-Field-of-View Imaging,” Adv. Opt. Mater. 2(8), 751–758 (2014). [CrossRef]
5. F. Colonnier, A. Manecy, R. Juston, H. Mallot, R. Leitel, D. Floreano, and S. Viollet, “A small-scale hyperacute compound eye featuring active eye tremor: application to visual stabilization, target tracking, and short-range odometry,” Bioinspir. Biomim. 10(2), 026002 (2015). [CrossRef]
6. W. B. Lee, H. Jang, S. Park, Y. M. Song, and H. N. Lee, “COMPU-EYE: a high resolution computational compound eye,” Opt. Express 24(3), 2013–2026 (2016). [CrossRef]
7. J. Z. He, H. J. Jian, Q. T. Zhu, M. C. Ma, and K. Y. Wang, “Matching between the light spots and lenslets of an artificial compound eye system,” in Aopc 2017: 3d Measurement Technology for Intelligent Manufacturing, A. K. Asundi, H. Zhao, and W. Osten, eds. (Spie-Int Soc Optical Engineering, 2017).
8. C. Y. Shi, Y. Y. Wang, C. Y. Liu, T. S. Wang, H. X. Zhang, W. X. Liao, Z. J. Xu, and W. X. Yu, “SCECam: a spherical compound eye camera for fast location and recognition of objects at a large field of view,” Opt. Express 25(26), 32333–32345 (2017). [CrossRef]
9. T. Labhart and E. P. Meyer, “Detectors for polarized skylight in insects: A survey of ommatidial specializations in the dorsal rim area of the compound eye,” Microsc. Res. Tech. 47(6), 368–379 (1999). [CrossRef]
10. H. Awata, M. Wakakuwa, and K. Arikawa, “Evolution of color vision in pierid butterflies: blue opsin duplication, ommatidial heterogeneity and eye regionalization in Colias erate,” J Comp Physiol A 195(4), 401–408 (2009). [CrossRef]
11. R. Shogenji, Y. Kitamura, K. Yamada, S. Miyatake, and J. Tanida, “Multispectral imaging using compact compound optics,” Opt. Express 12(8), 1643–1655 (2004). [CrossRef]
12. K. Kagawa, N. Fukata, and J. Tanida, “High-speed multispectral three-dimensional imaging with a compound-eye camera TOMBO,” Proc. SPIE 7797, 77970N (2010). [CrossRef]
13. J. Jin, S. Di, Y. Yao, and R. Du, “Design and fabrication of filtering artificial-compound-eye and its application in multispectral imaging,” in ISPDI 2013 - Fifth International Symposium on Photoelectronic Detection and Imaging(SPIE2013).
14. J. Chen, H. H. Lee, D. Wang, S. Di, and S.-C. Chen, “Hybrid imprinting process to fabricate a multi-layer compound eye for multispectral imaging,” Opt. Express 25(4), 4180–4189 (2017). [CrossRef]
15. T. Nakanishi, K. Kagawa, Y. Masaki, and J. Tanida, “Development of a Mobile TOMBO System for Multi-spectral Imaging,” Fourth International Conference on Photonics Solutions11331 (2020).
16. X. D. Yu, C. Y. Liu, Y. J. Zhang, H. R. Xu, Y. Y. Wang, and W. X. Yu, “Multispectral curved compound eye camera,” Opt. Express 28(7), 9216–9231 (2020). [CrossRef]
17. H. Xu, Y. Zhang, D. Wu, G. Zhang, Z. Wang, X. Feng, B. Hu, and W. Yu, “Biomimetic curved compound-eye camera with a high resolution for the detection of distant moving objects,” Opt. Lett. 45(24), 6863–6866 (2020). [CrossRef]
18. J. Klein and T. Aach, “Multispectral filter wheel cameras: modeling aberrations for filters in front of lens,” in Digital Photography Viii, S. Battiato, B. G. Rodricks, N. Sampat, F. H. Imai, and F. Xiao, eds. (Spie-Int Soc Optical Engineering, Bellingham, 2012).
19. C. Goenka, J. L. Semeter, J. Noto, H. Dahlgren, R. Marshall, J. Baumgardner, J. Riccobono, and M. Migliozzi, “Tunable filters for multispectral imaging of aeronomical features,” Adv. Space Res. 52(7), 1366–1377 (2013). [CrossRef]
20. T. Skauli, H. E. Torkildsen, S. Nicolas, T. Opsahl, T. Haavardsholm, I. Kasen, and A. Rognmo, “Compact camera for multispectral and conventional imaging based on patterned filters,” Appl. Opt. 53(13), C64–C71 (2014). [CrossRef]
21. M. Abuleil and I. Abdulhalim, “Narrowband multispectral liquid crystal tunable filter,” Opt. Lett. 41(9), 1957–1960 (2016). [CrossRef]
22. C. Zhang, M. Rosenberger, A. Breitbarth, G. Notni, I. Instrumentat, and S. Measurement, “A Novel 3D Multispectral Vision System based on Filter Wheel Cameras,” in 2016 Ieee International Conference on Imaging Systems and Techniques(Ieee, New York, 2016), pp. 267–272.
