 |
Night vision camera technology has come a long way in the digital age. As sensors continue to improve, so does low-light image quality. However, as good as infrared night vision cameras are, they are monochromatic. Researchers from the University of California Irvine hope to change that by combining infrared camera technology with artificial intelligence to develop a night vision camera that produces full-color images, even in what amounts total darkness for human eyes.
Our eyes see in the visible light spectrum. The wavelength of visible light ranges from 400 to 700 nm. Specialized night vision camera systems ‘see’ in infrared light which is then digitally converted into a monochromatic image in the visible light spectrum. This method of ‘seeing’ in the dark has limitations, including producing monochromatic image. The UC Irvine researchers ‘sought to develop an imaging algorithm powered by optimized deep learning architectures whereby infrared spectral illumination of a scene could be used to predict a visible spectrum rendering of the scene as if it were perceived by a human with visible spectrum light. This would make it possible to digitally render a visible spectrum scene to humans when they are otherwise in complete ‘darkness’ and only illuminated with infrared light.’ Part of ‘as if it were perceived by a human with visible spectrum light’ includes a color image.
 |
| ‘(left) Visible spectrum ground truth image composed of red, green and blue input images. (right) Predicted reconstructions for UNet-GAN, UNet and linear regression using 3 infrared input images.’ |
To achieve this goal of colorful night vision imagery, the team needed to adequately train a deep learning model with a suitable image set. Using printed images shown under multispectral illumination, a convolutional neural network was optimized to predict what different images seen under infrared light would look like in the visible spectrum. ‘This study serves as a first step towards predicting human visible spectrum scenes from imperceptible near-infrared illumination.’ As of now, the team has only tested the approach using printed photos. However, the results should translate to real-world subjects and video applications.
This isn’t the first time researchers have attempted to teach night vision cameras to see color. Prior attempts have included photographing the same scene with a typical camera and an infrared camera to teach a machine learning model to predict color from an infrared image. The scientists at UC Irvine have instead used multiple wavelengths of infrared light to improve a color-prediction algorithm. The results show that the team’s infrared light and deep learning method does an excellent job compared to an actual color image. As the team increased the number of infrared channels, the model made better predictions. Instead of relying upon a single infrared image, imaging across multiple infrared channels results in more accurate full-color predictions.
 |
| FID scores (lower is better) improve as the model is trained more and with images captured with additional infrared wavelengths. As you can see by the black line, the best results are achieved using three infrared wavelengths (718, 777 and 807 nm). |
Further work is required, but so far, the team’s deep learning model has produced consistent RGB reconstructions using only three input infrared images. Further, the proposed U-Net architectures can process three images per second. So, it’s not quite ready to behave as a real-time full-color night vision camera. However, it’s getting there. The team believes that different accelerations could be used to increase speed. Further, the starting point is a great baseline for improvements due to improved architecture, multi-threading or faster hardware.
The ‘proof-of-principle’ study shows promise. Possible real-world applications include full-color night vision cameras that could be used for surveillance, security, animal observation and military operations. The technology could also be useful for handling, processing and studying biological samples that are sensitive to visible light, such as when studying retinal tissue. There are also possible medical applications, such as being able to perform sensitive eye surgery in total darkness. The team concludes, ‘In short, this study suggests that CNNs are capable of producing color reconstructions starting from infrared-illuminated images, taken at different infrared wavelengths invisible to humans. Thus, it supports the impetus to develop infrared visualization systems to aid in a variety of applications where visible light is absent or not suitable.’
The full study is available at Plos One. The study’s authors are Andrew W. Browne, Ekaterina Deyneka, Francesco Ceccarelli, Josiah K. To, Siwei Chen, Jianing Tang, Anderson N. Vu and Pierre F. Baldi.
Author:
This article comes from DP Review and can be read on the original site.
