Teaching AI to accurately colorize marine plankton images (Image by LI Jianping)

Plankton are fundamental to the marine ecosystem and an indispensable means of modern marine ecological management. Since previous studies have shown that zooplankton are insensitive to long wavelength (i.e., red) light, such light is often used to make images of plankton as part of the ecological management process. Unfortunately, red light only produces grayscale images of plankton, thus losing information about their true color. 

In response to this problem, a research team led by Dr. LI Jianping from the Shenzhen Institute of Advanced Technology (SIAT) of the Chinese Academy of Sciences (CAS) recently developed an automatic plankton colorization algorithm, the IsPlanktonCLR network, based on deep convolutional neural networks. 

The algorithm is trained to convert gray images acquired under red lighting into high-fidelity color images. It can automatically and truthfully "dye" marine plankton grayscale images shot underwater into their natural colors, and the colorization effect is very close to human eye perception. 

The study was released in the European Conference on Computer Vision (ECCV), which was held in Israel from October 23 to 27.

Researchers from Xiamen University and the Harbin Institute of Technology (Shenzhen) were also involved in the research. 

The IsPlanktonCLR network uses a two-path structure with a self-guidance function, a customized color palette and a stepwise focusing loss function to achieve automatic colorization of plankton grayscale images. It shows excellent accuracy and fidelity in color restoration for rare species and key parts of common species. 

In order to train and develop the IsPlanktonCLR algorithm, the team constructed a dataset containing thousands of color-gray plankton image pairs. Using this dataset, the team not only trained the colorization algorithm, but also compared its performance with state-of-the-art colorization algorithms. The experimental results showed that the IsPlanktonCLR algorithm offered the best results not only in terms of human visual perception but also in terms of peak signal to noise ratio (PSNR), structural similarity (SSIM), Fréchet inception distance (FID) and other classical metrics frequently used in machine vision. 

In addition, existing colorization algorithms generally lack objective and quantitative evaluation metrics. Existing evaluation metrics not only have difficulty simulating the color perception ability of human vision but also have difficulty objectively evaluating the color similarity between the colored image and the ground-truth image. 

Therefore, the researchers proposed a color similarity evaluation index, Color Dissimilarity (CDSIM), which combines a color histogram, color aggregation vector, color correlogram and color gradient. Through the tests on marine plankton and natural scene images, the researchers verified that CDSIM is not only effective in evaluating the effect of colorization algorithms but is also more suitable for image colorization evaluation in the field of scientific imaging than other existing metrics. 

"The development of IsPlanktonCLR provides a new artificial intelligence solution for ocean imaging instruments to obtain accurate and real observation results," said Dr. LI. "This provides new technical means for us understanding the ocean."

Fig. 1 Overview of IsPlanktonCLR: (a) palette customization; (b) reference module; (c) colorization module. (Image by LI Jianping )  

Fig. 3 (a) Comparison of visual perception and CDSIM evaluations of marine plankton images produced by various colorization models. (b) Comparison of visual perception and numerical metrics-based evaluations of artificially colorized natural scene images. (Image by LI Jianping )