Face Recognition with visible and thermal IR images

Abstract

This thesis describes how the fusion of visible and thermal infrared (IR) images can be used to improve the performance of face recognition techniques, especially when illumination variations and occlusions are involved. Visible images are sensitive to illumination variations, while thermal IR images are robust to them. However, thermal IR images are degraded by occlusions caused from eyeglasses, but visible images can provide detailed information around the eyes even when eyeglasses are present. Fusion techniques, which combine complementary information from both spectrums, generate information that is robust to both illumination variations and occlusions. Before two images are fused, they must be registered. In this thesis, edge-based mutual information is used to register both visible and thermal IR images taken under different conditions. Following that, eyeglasses (if present) are removed from the thermal IR image, and replaced by eyes that are reconstructed from the visible image. Then, data-level, feature-level, and score-level fusion techniques are applied to the visible and thermal IR images for face recognition. Experimental results using the NIST/Equinox database showed that the fusion of visible and thermal IR images increased the number of first matches by 22% over visible images, and 8% over thermal IR images. Unfortunately, thermal IR sensors may be cost-prohibitive for many applications. In consideration of this, this thesis explores ways to predict a novelty component from the visible image. A novelty component is a thermal-like image that can be obtained from information in the visible image. It is later fused with the visible image for face recognition. Experimental results based upon four face recognition algorithms showed that the fusion of visible images and their novelty components increased the number of first matches over visible images by 21% (using the NIST/Equinox database) and 17% (using the Extended Yale Face Database B).