Video based facial recognition and visual processing: advances in the field of face recognition

As the usage of surveillance and personal video capturing devices is more evident in our lives, enormous amount of video data is being captured every day. Resulting from this, video based facial recognition has received unprecedented attention.


What is face recognition?

Face recognition has been an active yet challenging topic in the field of biometrics for the past two decades. The terms describes a special form of pattern recognition from human faces which is employed to identify individuals and pick up information from their expressions. Although identifying faces feels effortless for humans, it is actually a challenging computational problem (Reddy, et al. 2014). The difficulty lies in that the general impression is that human faces share overall similar configuration, and in that sense, face images can be described by a relatively low dimensional subspace (Patel, et al. ref).Based on this perception, holistic dimensionality reduction subspace methods have been employed for face recognition (Patel, et al. ref).

Further to this characteristic, work in cognitive neuroscience suggests that face recognition depends at least partly on mechanisms that differ from other types of visual recognition. Evidence from the latter occur from brain-damaged patients who no longer have the ability to recognize faces despite their ability to recognize objects (Rezlescu, Pitcher, &Duchaine, 2012). However there is also the case that the opposite happens, patients being able to recognize upright faces without any problem however experience severe problems recognizing objects (Moscovitch, Winocur, &Behrmann, 1997). Research findings from functional magnetic resonance imaging revealed that the occipital and temporal lobes consist of a number of small patches that respond particularly strongly to faces (Kanwisher, McDermott, & Chun, 1997; Duchaine&Yovel, in press). These patches based on findings from single-cell recordings in monkey face patches indicate that consist of neurons that respond almost exclusively to faces (Freiwald&Tsao, 2010).


Video based facial recognition

Video based face recognition is an alternative approach to face recognition that traditionally focused on recognition from still images. There are significant advantages in the use of video based face recognition. For example, you can gain much more information from a video scene than a single image. This could lead to more robust and stable recognition by fusing information of multi frames. Another advantage relates to how temporal information from videos can be used to improve the accuracy of face recognition. Finally, through a video you can access multi poses of faces which allow to explore shape information of face. Nevertheless, video based face recognition is also a very challenging problem, with problems ranging from low quality facial images, illumination changes, pose variations, occlusions and more (Zhang, et al. 2011). The procedure of video based face recognition is shown in Fig. 1(Zhang, et al. 2011).



Video based face recognition methods can be divided into two categories based on their applications: video-image based methods and video-video based methods (Zhang, et al. 2011). The first category can be seen as an extension of still image based face recognition as it uses still face images from multi frame information. The second category involves information in the form of videos, which has its difficulties. This sort of analysis primarily focuses on either feature vector extracted from video input; probability density function or manifold to show the distribution of faces in videos; or generative models to describe dynamic variance of face in images (Zhang, et al. 2011).

Video-image based methods only exploit physiological information of the face while the video-video based methods have more information to be exploited. It is evident that video based face recognition has great potential to make progress and be adopted in real application(Sun et al.2011).

The challenges

Despite the extensive research in the area of face recognition with high performance levels reported from using face recognition systems, there is still a long way to go in comparison to proven recognition methods using fingerprint and iris (Chacon & Rivas, 2009).

And in the case of video based face recognition, it is proving even more difficult to maintain similar levels of performance (Chen, et al. 2010). A key challenge for video-based facial expression analysis in practice is to exploit the extra information available in a video. In addition, making use of dynamic features. There is also difficulty due to the fact that different video sequences of the same subject may contain variations in resolution, illumination, pose, and facial expressions. In this sense it is difficult to design effective video-based face recognition algorithm.


Applications and future prospects

Substantive research in recent years sought to address the problem of video based face recognition. Most of these approaches can be categorizedinto sequence-based ones and set-based methods(Huang, et al. 2015). The former methods exploit the temporal or dynamic information of the faces in the video. The latter ones represent videos as image sets of separated video frames, without using the temporal information (Huang, et al. 2015).

Novel frameworks for video based facial recognitions has been Hybrid Euclidean-and-Riemannian Metric Learning (HERML) method to fuse multiple statistics of image set. Another video-based face recognition method has been proposed that uses video frames of a subject rotating his/her head. These models are based on manifold interpretations of the brain’s visual perception (Hamedani et al. 2015).

Furthermore, another breakthrough approach is generative approach based on dictionary learning methods. One major advantage of Dictionary-based Face Recognition from Video (DFRV) approach is that it is robust to some variations in video sequences.

Video based face recognition is an emerging technology that its practical applications have only started to show. Much research around the world is taking place to achieve maximum accuracy and exploit the capabilities of this approach. The future holds the promise that automated surveillance systems enabled by this type of face recognition will be used vastly.




Chen, S., Mau, S., Harandi, M., Sanderson, C., Bigdeli, A., & LovelL, B.C. (2011)Face Recognition from Still Images to Video Sequences: A Local-Feature-Based Framework. EURASIP Journal on Image and Video Processing.790598, DOI:10.1155/2011/790598

Duchaine, B. &Yovel, G. (in press) The role of face-selective areas in face perception: An updated neural framework. Annual Review of Vision Science.

Freiwald, W.A., &Tsao D.Y. (2010) Functional compartmentalization and viewpoint generalization within the macaque face-processing system.Science, 330, pp.845-851.

Hamedani, K., Seyyedsalehi, S.A, Ahamdi, R. ()Video-based face recognition and image synthesis from rotating head frames using nonlinear manifold learning by neural networks, Neural Computing and Applications, pp.1-9.

Huang, Z., Wang, R., Shan, S., & Chen, X. (2015) Face recognition on large-scale video in the wild with hybrid Euclidean-and-Riemannian metric learning. Pattern recognition, 48(10), pp.3113-3124.

Kanwisher, N., McDermott, J., & Chun, M. (1997) The fusiform face area: A module in human extrastriate cortex specialized for the perception of faces. Journal of Neuroscience, 17, pp.4302-4311.

Moscovitch, M., Winocur, G., &Behrmann, M. (1997) What is special about face recognition? 19 experiments on a person with visual object agnosia and dyslexia but normal face-recognition. Journal of Cognitive Neuroscience. 9, pp.555-604.


Reddy A. S., Tan Z., Zhang S. (2014). Curation and analysis of multitargeting agents for polypharmacological modeling. J. Chem. Inf. Model. 54, pp. 2536–2543. 10.1021/ci500092j


Rezlescu, C., Pitcher, D., &Duchaine, B. (2012) Acquired prosopagnosia with spared within-class object recognition but impaired recognition of basic-level objects. Cognitive Neuropsychology, 29: 325-347.


Patel, V.M., Chen, Y.C., Chellappa,R., & Phillips, P.J. (n.d.) Dictionaries for Image and Video-based Face Recognition. National Institute of Standards and Technology, Gaithersburg.

Zhang, Z., Wang, C., & Wang, Y. (2011) Video-Based Face Recognition: State of the Art. Proceedings of the 6th Chinese Conference, CCBR 2011, Beijing, China, December 3-4.

Video based facial recognition and visual processing: advances in the field of face recognition

Leave a Reply

Your email address will not be published. Required fields are marked *