Systems and Control Engineering News

NEC and Tokyo Institute of Technology use AI to dramatically increase image clarity under severe conditions

Automatically combines visible and non-visible images

June 22, 2017

The research group of Professor Masatoshi Okutomi and Visiting Associate Professor Masayuki Tanaka of Tokyo Institute of Technology (Tokyo Tech) and NEC Corporation (NEC; TSE: 6701) , today announced their joint development of a multi-modal image fusion technology that dramatically improves the clarity of images by using artificial intelligence (AI) to automatically combine visible images taken by standard cameras with non-visible images taken by specialized devices, such as thermal or terahertz cameras.

(a) Visible Image

(b) Far-Infrared image

(d) Proposed method

Figure 1. Example of the application¹

In recent years, technological advancements and cost reductions for specialized devices that photograph non-visible images, such as thermal cameras, which capture heat, and X-ray cameras, have enabled them to be used for an expanding range of purposes, including nighttime monitoring under severe weather conditions, such as dense fog. However, these cameras tend to provide images of poor resolution and quality in comparison to images taken of visible subjects.

Conventionally, in order to analyze non-visible and visible images of the same subject, images had to be viewed and compared separately, or they had to be manually combined by a trained expert. Moreover, the detection of abnormalities or hazards contained in non-visible images could be easily overlooked when combining the images.

This new technology uses AI to achieve greater image visibility by automatically selecting highly visible parts from multiple images and combining them, while enhancing the smallest characteristics contained in non-visible images. Specifically, AI carries out detailed examination of each image in order to assess the degree of visibility of each part, then automatically extracts the best areas from each image, taking environmental characteristics into consideration, such as brightness, the direction of light and obstacles.

"As this technology enables instant visual clarification, even under harsh conditions, it allows users to make well informed evaluations. For example, it can be applied to monitoring systems to assist with nighttime observations, or to infrastructure inspection devices to improve the detection of interior and exterior abnormalities, such as cracking," said Akio Yamada, General Manager, Data Science Research Laboratories, NEC Corporation.

"Until now, a specialist had to manually carry out complex conversion tasks in order to combine images taken by different types of cameras. This technology eliminates the need for such manual work, using AI to effectively and automatically combine images taken by different cameras. This also increases visibility by actively utilizing the strong points of each visible image and non-visible image, even when the images are difficult to visualize," said Professor Masatoshi Okutomi, School of Engineering, Tokyo Institute of Technology.

Moreover, AI analyzes the slightest clue to detecting an abnormality and hazard in a non-visible camera image and automatically generates a multimodal (fusion of visible and non-visible) image with exceptionally high visibility, while properly regulating the degree of enhancement to avoid causing image breakdowns, such as clipped highlights and crushed blacks.

NEC and Tokyo Tech present this technology on June 7, at the 23rd Symposium on Sensing via Image Information to be held at Pacifico Yokohama (Nishi-ku, Yokohama City) from Wednesday, June 7 to Friday, June 9.

1 Example of the application

Obtained permission to use the severe environment simulator in the ImPACT Tough Robotics Challenge program.