This is a list of reviewers who served JEI in 2018.

License plate image database is the most significant factor that supports the development of license plate recognition. We first present a general method for building a license plate image database and establish a Chinese license plate image database. There are 29,015 images included in the database, named SYSU. The main work includes: first, based on a quantitative analysis of the attributes of license plate images that affect license plate recognition, relational license plate image database models are established, which consist of function and performance dataset models; second, based on the function dataset models, we present a semiautomatic method that can extract the values of the attributes in the road monitoring image and establish the function datasets, which include type and provincial abbreviation variation images. To open the function datasets, we also propose a preservative deidentification method to balance privacy protection and attribute preservation. Third, based on the performance dataset models, we establish the performance datasets, which are digital processing images from the function image datasets, including resolution, average luminance, nonuniformity of luminance, horizontal rotation angle, vertical shear angle, horizontal parallel perspective angle, vertical parallel perspective angle, out-of-focus blur, and linear uniform motion blur variation images. The recognition performance of five commercial softwares on the SYSU license plate database indicates that the database is a valuable test-bed for the evaluation and analysis of license plate recognition technology.

We present a gaze estimation method for a head-mounted full-view egocentric camera that can capture egocentric video together with users’ gaze cues. While the conventional gaze recording device has two cameras, an eye camera and a scene camera, the proposed gaze recording device has only a single spherical camera with a full field of view. To determine the point of gaze on full-view images, we present an eye-model-based gaze estimation method by means of three-dimensional iris projection. First, an eye model is built according to the precalibration process of an eyeball center and the eyeball biological parameters; then, we express the 3D iris contour in the eye model and project it back to the spherical camera model. Since 2D iris contours can be detected directly on images and can also be expressed under the spherical model, the relationship between the 3D iris contour and the 2D iris contour can be found. By solving this problem, the gaze direction under the camera model is determined; subsequently, the point of gaze on the images can be inferred. Since in the proposed method, a single spherical camera can play the role of the conventional two cameras, not only do the proposed method results have a simpler system structure but also the cumbersome operation of the calibration for the conventional two-camera gaze measurement devices becomes unnecessary. The effectiveness of the proposed method is shown by the experimental results.

Face recognition based-access systems have been used widely in security systems as the recognition accuracy can be quite high. However, these systems suffer from low robustness to spoofing attacks. To achieve a reliable security system, a well-defined face liveness detection technique is crucial. We present an approach for this problem by combining data of the light-field camera (LFC) and the convolutional neural networks in the detection process. The LFC can detect the depth of an object by a single shot, from which we derive meaningful features to distinguish the spoofing attack from the real face, through a single shot. We propose two features for liveness detection: the ray difference images and the microlens images. Experimental results based on a self-built light-field imaging database for three types of the spoofing attacks are presented. The experimental results show that the proposed system gives a lower average classification error (0.028) as compared with the method of using hand-crafted features and conventional imaging systems. In addition, the proposed system can be used to classify the type of the spoofing attack.

In recent years, various synthesis dictionary learning-based techniques have been applied for the task of visual object tracking. However, there are challenges in the representation ability of the target and the background using a synthesis dictionary. Owing to such challenges, the target in a given video sequence is not tracked very efficiently. We propose an online analysis dictionary learning framework for visual tracking. Our main motivation stems from the fact that analysis dictionary is able to capture significantly more variability in the data as compared to a synthesis dictionary of the same dimensions. We make use of single and multiple analysis dictionaries in our work. In order to leverage the advantage of multiple dictionaries, we adaptively fuse them. To the best of our knowledge, this is the first work that uses analysis dictionary for visual tracking. We use the entire OTB-50 dataset to evaluate the performance of proposed trackers. Our experiments show a significant improvement over existing synthesis dictionary learning approaches. This makes our approach much more preferable over the conventional approaches that use synthesis dictionary learning model for visual tracking.

Training autonomous vehicles requires lots of driving sequences in all situations. Collecting and labeling these drive scenes is a very time-consuming and expensive process. Currently, it is not possible to reuse these drive scenes with different optical properties, because there exists no numerically efficient model for the transfer function of the optical system. We present a numerical model for the point spread function (PSF) of an optical system that can efficiently model both experimental measurements and lens design simulations of the PSF. The numerical basis for this model is a nonlinear regression of the PSF with an artificial neural network. The novelty lies in the portability and the parameterization of this model. We present a lens measurement series, yielding a numerical function for the PSF that depends only on the parameters defocus, field, and azimuth. By convolving existing images and videos with this PSF, we generate images as if seen through the measured lens. The methodology applies to any optical scenario, but we focus on the context of autonomous driving, where the quality of the detection algorithms depends directly on the optical quality of the used camera system. With this model, it is possible to reuse existing recordings, with the potential to avoid millions of test drive miles. The parameterization of the optical model allows for a method to validate the functional and safety limits of camera-based advanced driver assistance systems based on the real, measured lens actually used in the product.

Accurate road segmentation is imperative for any effective advanced driver assistance system. Existing state-of-the-art methods are generally based on network architectures; however, high accuracy comes at the expense of abundant annotated data and high-computational cost. The inherently large amount of nonroad areas increases the running time and affects road segmentation accuracy. We propose a method based on the data enhancement of vanishing points (VPs) and deep learning to segment road areas. The VP serves as the location prior to remove redundant areas (e.g., sky and tall buildings). Contour maps reduce noise voters from background interference to mitigate the low VP estimation speed. A voting strategy is deployed to further accelerate the processing speed. After VP presegmentation, the remaining regions are fed into a multilayered extended semantic network (MultiNet), which consists of VGG16 and FCN. By combining VPs and MultiNet, the proposed method reveals discriminative features for road segmentation better than the original MultiNet. The convergent speed of training the MultiNet model is 25% faster than the original network owing to the location priors. Experimental results reveal that the proposed method outperforms several other methods with regard to both efficiency and accuracy.

Direct visual tracking (DVT) for planar objects is a fundamental problem in computer vision. DVT methods often formulate tracking as an image registration problem, where image intensities are directly used to match two images. However, these methods are usually sensitive to illumination changes as they assume intensities are constant. The gradient orientation (GO) was proven to be insensitive to illumination variations in previous reports. We further confirmed that the GO’s robustness can be significantly improved when the pyramid technique is employed. We present a robust DVT method, named gradient orientation pyramid efficient second-order minimization (GOP-ESM), based on the proposed gradient orientation pyramid descriptor. GOP-ESM takes the advantages of the robust feature descriptors and the efficient second-order minimization method as to enhance tracking robustness and accuracy. We also published a tracking dataset for planar objects with illumination changes (POIC). The evaluations on the proposed POIC dataset and the other two public benchmark datasets demonstrated that GOP-ESM outperforms the state-of-the-art tracking methods against various environmental variations, especially illumination changes.

Although pedestrian detection has been largely improved with the emergence of convolutional neural networks (CNN), the performance in autonomous driving still faces various challenges, which mainly include large-scale variation, illumination variation, and occlusion of different levels. A robust pedestrian detector enhanced by semantic segmentation is proposed. Inspired by the benefits of multitask learning, our main idea lies in integrating the task of semantic segmentation into the detection framework with auxiliary supervision, inheriting the merits of the two-stream network. Specifically, anchor boxes with various scales are paved on the feature maps of a base CNN; detection is performed based on bounding box classification and regression. On the other stream, semantic segmentation is also performed based on the same feature maps. Extensive experiments on the recently published large-scale pedestrian detection benchmark, i.e., CityPersons, show that the additional supervision from semantic segmentation can significantly improve the detection accuracy without extra computational burdens during inference, which demonstrates the superiority of the proposed method.

We present a spatiotemporal saliency detection method for videos. In contrast to previous methods that focus on exploiting the different underlying saliency cues or ignore motion information, the proposed method aims to use both appearance information based on spatial edges and spatial color saliency and motion information based on temporal motion boundaries as indicators of foreground object locations. Spatial color saliency is obtained from the fusion of three color features: color edge connectivity, color rarity, and color compactness. Subsequently, we further smooth the color saliency to eliminate background noises and to further boost the detection accuracy. Then, we propose a strategy-based low-level saliency fusion that guarantees to complementarily employ the smoothed color saliency, spatial edges, and temporal motion boundaries clues toward producing high-accuracy low-level saliency. Subsequently, we generate framewise spatiotemporal saliency maps using a geodesic distance from the low-level saliency. Subsequently, high-quality results are obtained through the geodesic distance to the background area in the subsequent frames. Extensive quantitative and qualitative experiments on three public video datasets demonstrate the superiority of the proposed method over the state-of-the-art algorithms.

To deal with the three-dimensional (3-D) point cloud loss caused by object reflection in the active fringe projection 3-D measurement, an active reflection suppression method for 3-D measurement is proposed. The method employs high-dynamic range images obtained by multiple exposure image fusion and a three-wavelength phase-shift profilometry method to achieve high-precision 3-D measurement of reflective objects. Experimental results show that, compared to traditional 3-D measurement methods, the proposed one can more effectively handle reflections thereby avoiding 3-D point cloud loss in the measurement of reflective objects.

Accurate and fast extraction of foreground objects is a key prerequisite for a wide range of computer vision applications, such as object tracking and recognition. Thus, many background subtraction (BGS) methods for foreground object detection have been proposed in recent decades. However, this is still regarded as a tough problem due to a variety of challenges, such as illumination variations, camera jitter, dynamic backgrounds, and shadows. Currently, there is no single method that can handle all the challenges in a robust way. We try to solve this problem from a perspective of combining different state-of-the-art BGS algorithms to create a more robust and more advanced foreground detection algorithm. More specifically, an encoder–decoder fully convolutional neural network architecture is adapted and trained to automatically learn how to leverage the characteristics of different algorithms to fuse the results produced by different BGS algorithms and produce a more precise result. Comprehensive experiments evaluated on the CDnet 2014 dataset demonstrate that the proposed method outperforms all the considered single BGS algorithms. We show that our solution is more efficient than other BGS combination strategies.

We develop a gender classification method using convolutional neural networks. We train Alexnet Architecture using the luminance (Y) component of the facial image (YCbCr) for the SoF, groups, and face recognition technology datasets. The Y component is reduced to a size of 32  ×  32 via discrete wavelet transform (DWT). The use of the Y plane and a low-resolution subband image of the DWT significantly reduce the amount of processed data. We are able to achieve better results than other machine learning, rule-based approaches and the traditional convolutional neural net structure that are trained with three-dimensional RGB images. We are able to maintain comparably high recognition accuracy, even with the reduction of the number of network layers. We have also compared our structure with the state-of-the-art methods and provided the recognition rates.

With the aim of reducing the distortion of the marked image to realize high-fidelity reversible data hiding (RDH), we propose an RDH scheme based on dynamic prediction and expansion (DPE). By introducing a dynamic pixels-value-ordering (D-PVO) method into DPE, a full reversibility can be achieved. The pixels of the cover image are predicted by sorting their cross-over neighbors and using the two end pixels or the median pixels to conduct the prediction. The covert data are embedded by using prediction-error-expansion. We have theoretically proved higher prediction accuracy for the proposed method compared with the two existing classical prediction methods. In our experiments, we first verify the prediction accuracy of D-PVO by applying it to Gaussian distributed sampled data. Next, we perform experiments on the standard pictures from USC-SIPI and Kodak image datasets. The experimental results demonstrate that the proposed scheme can ensure a full recovery of the original image and a higher fidelity of the marked cover images, and the peak signal-to-noise ratio can exceed 60.00 dB when embedding capacity reaches 10,000 bits.

The loop closure detection (LCD) problem in simultaneous localization and mapping has been an important research topic to reconstruct three-dimensional environments and estimate a camera trajectory accurately. Although the bag-of-visual-word (BoVW) scheme has been used widely and has shown good results, it has several problems in some aspects, such as occlusion, deformation, illumination change, and viewpoint change. In order to tackle the challenges, we propose an LCD method using the BoVW method with a local patch descriptor obtained from the learning-based approach. We have trained a neural network model with a place-oriented dataset and extract the descriptors for the local patches from the trained neural network model. In addition, we have constructed the ground-truth label for the evaluation. Our experiment shows promising results, compared to the state-of-the-art LCD method. The implementation of the proposed deep convolutional generative adversarial network descriptor, as well as the evaluation toolbox, can be found online at https://github.com/JustWon/DCGAN_Descriptor.

We have collected a large dataset of subjective image quality “*nesses,” such as sharpness or colorfulness. The dataset comes from seven studies and contains 39,415 quotations from 146 observers who have evaluated 62 scenes either in print images or on display. We analyzed the subjective evaluations and formed a hierarchical image quality attribute lexicon for *nesses, which is visualized as image quality wheel (IQ-Wheel). Similar wheel diagrams for attributes have become industry standards in other sensory experience fields such as flavor and fragrance sciences. The IQ-Wheel contains the frequency information of 68 attributes relating to image quality. Only 20% of the attributes were positive, which agrees with previous findings showing a preference for negative attributes in image quality evaluation. Our results also show that excluding physical attributes of paper gloss, observers then use similar terminology when evaluating images with printed images or images viewed on a display. IQ-Wheel can be used to guide the selection of scenes and distortions when designing subjective experimental setups and creating image databases.

Performance of an automatic fingerprint identification system (AFIS) depends on the quality of fingerprint images. Therefore, quality estimation of fingerprint images can lead to performance enhancement of AFIS by eliminating the poor quality fingerprint images. A fingerprint quality estimation algorithm is proposed, which computes the fingerprint image quality at local level (blockwise). The proposed quality estimation algorithm analyzes blocks of fingerprint images in terms of their quality nature (dry, normal dry, good, normal wet, and wet). The features used for quality nature assessment are moisture, mean, variance, ridge valley area uniformity, and ridge line count. The performance of the block quality nature is assessed on fingerprint verification competition (FVC) 2004 datasets using a decision tree classifier. The proposed approach achieves classification accuracy of 95.90%. Further, the overall quality score (QS) for a fingerprint image is obtained by combining QSs assigned to all minutiae centered local patches of the fingerprint image using quality nature assessment, orientation analysis, and clarity analysis. The overall QSs for fingerprint images of FVC 2004 database (DB1, DB2, DB3, and DB4 datasets) are computed. These scores are used to evaluate the quality ranked recognition performance on each dataset of FVC 2004 database. Experimental evaluations reveal that rejecting low quality fingerprint images improves the performance of the recognition system. A comparative study with state-of-the-art quality estimation algorithms indicates that the QSs assigned using the proposed method are accurate and precise. Therefore, the proposed method can be used as a quality control unit during fingerprint acquisition, which helps in improving the performance of a recognition algorithm.

High-efficiency video coding (HEVC) is a very common application in today’s electronics field. Lossless intraframe compression is preferred for many specific applications. Compared to lossy intracompression or interframe encoding, intraframe encoding has lower computation complexity but an unsatisfactory compressed bit-rate. Extant optimization algorithms center on improving coded bits in space domains and few include considerations for accelerating encoding time. We focus on video compression in the frequency domain to reduce bit-rate while saving encoding time. The proposed method involves placing discrete wavelet transform in front of the traditional HEVC tools to transform a given frame into frequency coefficients. To accelerate the encoding process, fast prediction mode skip and CU depth decision are used to bypass certain unnecessary computations based on texture detection results. The experimental results indicated that our method reduces bit-rate by 6.466% on average with 48.068% encoding time savings compared to the original HEVC encoding.

Color demosaicing is a key image processing step aiming to reconstruct the missing pixels from a recorded raw image that has a color filter array (CFA) pattern. The color correlation–based guided filters, such as minimized-Laplacian residual interpolation (MLRI), are known as the state-of-the-art demosaicing techniques. However, in the conventional guided filter-based techniques, the artifacts are generated in areas with low color correlation. Furthermore, a large number of line memories are required in the hardware implementation of a conventional guided filter-based technique because of the large effective field size. To overcome these two problems, we propose a color correlation–adaptive demosaicing algorithm that selectively applies a specific intracolor demosaicing to regions with low color correlation. We also propose an algorithm structure that reduces the effective field size in the vertical direction to reduce the number of line memories, while maintaining the image quality performance during the hardware implementation. The experimental results show that the proposed scheme can reduce the line memory to one-third while showing marginal performance degradation compared to the state-of-the-art MLRI weighted framework in terms of the color peak signal-to-noise ratio for the IMAX datasets.

The 360-deg or virtual reality video can capture all sides around the observer at the same time and provides the freedom to select any part of the surroundings for display using a head mounted display in the case of a single user or smart TV, curved screen, and even with video projectors placed in a 360-deg projection room in the case of multiusers. In order to encode these spherical videos with standard codecs, a projection step is necessary to transform the original 3-D space scene into a regular 2-D video sequence. Thus different geometric shapes can be used as equirectangular projection (ERP), cubemap projection (CMP), segmented sphere projection (SSP), etc. The proposed model presents the 360-deg video on eight square faces, six faces for the equatorial part, and two faces for the top and bottom views. This shape can ensure a better projection of the video thanks to the size ratio of the lateral faces, which is very close to the size ratio of televisions and video projectors used in 360-deg rooms. Quality metrics comparison shows average gains when compared with ERP and SSP models and the encoding time is hugely reduced when compared with CMP model.

We address the problem of food items detection and recognizing different food categories in images. Given the variety of food items with low inter- and high intraclass variations and the limited information contained in a single image, the problem is known to be particularly hard. In order to achieve better detection and recognition capabilities, we propose a joint use of multiple classifiers trained on features extracted via multiple deep models using different fusion techniques, including an early and two different late fusion schemes, namely induced order weighted averaging and particle swarm optimization based fusion. Moreover, we assess the performance of different deep models in food items detection and recognition. Experimental evaluations are carried out on two large-scale benchmark datasets, demonstrating better results for the proposed approach.

Image demosaicking is a method of reconstructing an RGB image from a Bayer pattern, which is required when color information is lacking because a single charge-coupled device is used during the image extraction process of a digital camera. There are many restrictions on the reconstruction of a Bayer image to an RGB image. Given that each pixel contains only one-color information, artifacts, such as false color or the zipper effect, may occur at the edges, which can arise as a result of significant differences in brightness and color change. We propose a demosaicking method for adaptively selecting the reference range of color difference to obtain reliable information from texture regions and reconstructing into the RGB image. In particular, we determine the adaptive weight and reference range for four directions, east (E), west (W), south (S), and north (N), to improve the reliability of the color pixel obtained by a color difference estimation using guided filtering applied on residuals. In our experiment, we compare the results of the proposed method for the Kodak and IMAX datasets with those of nine demosaicking methods. The proposed method shows similar or improved results in terms of the color peak signal-to-noise ratio. In addition, compared to other methods, the visual quality improved by reducing residual artifacts.

Multiexposure image fusion is an effective method for depicting a high dynamic range display of target scenes. However, the loss of local details and ghosting artifacts in fusion images has not yet been well resolved in the existing research. To solve these problems, a multiexposure image fusion algorithm with detail enhancement and ghosting removal is proposed. First, the exposure brightness and chromaticity information in YIQ color space are used to measure weight maps. When the captured scene is dynamic with moving objects, the weight maps are refined to remove ghosting artifacts based on the image difference and superpixel segmentation. Finally, an improved Laplacian pyramid fusion framework is proposed to achieve image fusion with detail enhancement. The experimental results demonstrate the effectiveness of the proposed method in terms of both visual quality and objective evaluation and show that the proposed algorithm can preserve more details and can remove ghosting artifacts effectively.

We tackle automatic meter reading (AMR) by leveraging the high capability of convolutional neural networks (CNNs). We design a two-stage approach that employs the Fast-YOLO object detector for counter detection and evaluates three different CNN-based approaches for counter recognition. In the AMR literature, most datasets are not available to the research community since the images belong to a service company. In this sense, we introduce a public dataset, called Federal University of Paraná-AMR dataset, with 2000 fully and manually annotated images. This dataset is, to the best of our knowledge, three times larger than the largest public dataset found in the literature and contains a well-defined evaluation protocol to assist the development and evaluation of AMR methods. Furthermore, we propose the use of a data augmentation technique to generate a balanced training set with many more examples to train the CNN models for counter recognition. In the proposed dataset, impressive results were obtained and a detailed speed/accuracy trade-off evaluation of each model was performed. In a public dataset, state-of-the-art results were achieved using <200 images for training.

Among the most challenging difficulties that hinder the visually impaired is a safe navigation from one place to another. We will provide a comprehensive framework to evaluate our smartphone-based obstacle detection method for safe navigation for the visually impaired people. This method aims to provide a safe navigation by detecting static and dynamic obstacles in unknown environments while offering maximum flexibility to the user and using the most cost-effective equipment possible. Our obstacle detection method relies on a multiregion analysis technique to examine the results of analyzing different regions of video frames. The user is notified about the existence of an obstacle by receiving alert messages. We (1) introduce a rich dataset, which is available for research purposes and can be used for the future evaluation and development of computer vision-based applications pertinent to static and dynamic obstacle detection in the unknown environment via smartphone. (The dataset is made available by King Abdulaziz University for research purposes only. Please, contact the authors for further information.) In addition, (2) we demonstrate the efficiency of the proposed method through a comprehensive framework of testing and evaluation in various complex and unknown scenes. The usability of the method has been tested by allowing a set of visually impaired people to use our system in their navigation. The experimental results show that our obstacle detection method is easy to use, operates efficiently, can be deployed on an average range smartphone device, and has a high accuracy rate ranging from 85% to 96%.

Image ordinal estimation usually defined as a regression or classification problem is challenging in computer vision. Many recent works simply implement deep convolutional neural networks (CNNs) to get ideal results, while they have not analyzed why it works well as conventional computer vision methods. We revisit the problem of image ordinal estimation and propose two methods on ordinal estimation. First, a double-task convolutional neural network model (DTCNN) is proposed by combining classification and regression objectives. Then we have analyzed why DTCNN works well from three perspectives: (1) the ordinal relationship between two tasks, (2) the coarse/fine category level for two tasks, and (3) neurons’ activation on two tasks. To avoid choosing heuristic combination parameter, we further propose risk-based CNN (R-CNN) model to embed the ordinal constraint into classification task. Based on Bayesian risk rule, the category information and ordinal information are fused by a dynamic weighting loss function. Experimental results on two ordinal datasets show that both DTCNN and R-CNN obtain competitive performance.

Due to its advantages in size and energy consumption, mechanical scanning imaging sonar (MSIS) has been widely used in portable and economic underwater robots to observe the turbid and noisy underwater environment. However, handicapped by the coarseness in spatial and temporal resolution, it is difficult to stitch the scan pieces together into a panoramic map for global understanding. A registration method named symmetrical Kullback–Leibler divergence (SKLD)-distribution-to-distribution (D2D), which models each scan as a Gaussian mixture model (GMM) and evaluates the similarity between two GMMs in a D2D way with the measure defined by SKLD, is proposed to register the scans collected by MSIS. SKLD not only weights the difference between distributions with the prior probability but also increases the numerical stability with the symmetrical constraint in distance measure. Moreover, an approximation strategy is designed to derive a tractable solution for the KLD between two GMMs. Experimental results on the scans that were collected from the realistic underwater environment demonstrate that SKLD-D2D dramatically reduces the computational cost without compromising the estimation precision.

In recent years, deep neural networks have achieved great success in various fields, especially in computer vision. However, recent investigations have shown that current state-of-the-art classification models are highly vulnerable to adversarial perturbations contained in the input examples. Therefore, we propose a defense methodology against the adversarial perturbations. Prior to a targeted network, adversarial perturbations are erased or mitigated via a deep residual generative network (RGN). Through adopting an auxiliary network VGG-19, the RGN is trained toward optimization of a joint loss, including low-level pixel loss, middle-level texture loss, and high-level task loss, thereby the restored examples are highly consistent with the original legitimate examples. We call our proposed defense based on RGN as RGN-Defense. It is an independent defense module that can be flexibly integrated with other defense strategy, for example, adversarial training, to construct a more powerful defense system. In the experiment, we evaluate our approach on ImageNet, and the comprehensive results have demonstrated robustness of RGN-Defense against current representative attacks.

It is very difficult for visually impaired people (VIP) to perceive and avoid obstacles at a distance. To address this problem, we propose a sensor fusion system, which combines the RGB-depth (RGB-D) sensor and millimeter wave (MMW) radar sensor, to perceive the surrounding obstacles. The position and velocity information of the multiple targets are detected by the MMW radar based on the principle of frequency modulated continuous wave. The depth and position information of the obstacles are verified by the RGB-D sensor based on the MeanShift algorithm. The data fusion based on the joint probabilistic data association algorithm and Kalman filter enable the navigation assistance system to obtain more accurate state estimates compared with using only one sensor. The nonsemantic stereophonic interface is utilized to transfer the obstacle detection results to the VIP. The experiment results show that multiple objects with different ranges and angles are detected by the radar and the RGB-D sensor. The effective detection range is expanded up to 80 m compared to using only the RGB-D sensor. Moreover, the measurement results are stable under diverse illumination conditions. As a wearable system, the sensor fusion system has the characteristics of versatility, portability, and cost-effectiveness.

Head pose estimation (HPE) plays a vital role in the field of human–computer interaction and remains a challenging task due to individual differences. To mitigate this issue, we propose a simple yet effective deep multitask learning framework for joint HPE and face verification (FV), where FV acts as an auxiliary task to boost the performance of HPE. The framework comprises a backbone net, data separate module, and two branches for HPE and FV, respectively. Considering other regions beyond those of the face provides useful information for HPE while FV should focus only on face regions, two kinds of regions, head and face, which share common feature representations in the backbone net, are separated by the data separate module and then fed into the appropriate branches. Kullback–Leibler divergence loss and L₂-constrained softmax loss are connected to the end of the HPE branch and FV branch, respectively, to optimize the architecture. The proposed method is validated on three publicly available datasets: Pointing04, CAS-PEAL-R1, and CMU multi-PIE. The experimental results demonstrate that our method surpasses the state of the art, showing up to a 16.38% improvement on the famous benchmark, Pointing04. The best accuracies we report on these datasets are 89.53%, 99.74%, and 99.72%, respectively.

Application of computer images processing technology to analyze materials microstructural images, particularly metallographic images, has received increasing attention. The metallographic images contain the mesoscopic information on structural relation and components of materials. Quantitative analysis of these images can help to correlate the materials structures to their performance and properties at various levels. There are two challengeable issues necessary to be resolved, i.e., automatic segmentation and classification of different microscopic structures in metallographic images. Since the metallographic images often contain complex textures, the segmentation of them is usually inaccurate with present methods. We propose a hybrid algorithm, which combines the Gaussian filter, the mean shift method, the FloodFill, the improved flow-based difference-of-Gaussians, and the clustering to resolve the issues. The experiment results and the comparative results show that our method is effective to segment and classify the microstructural elements in metallographic images with complex textures.

Our paper presents the development of a real-time system based on detection, classification, and position estimation of objects in an outdoor environment to provide the visually impaired individuals with a voice output-based scene perception. The system is low-cost, light weight, simple, and easily wearable. An odroid board integrated with an USB camera and USB laser is utilized for the purpose. To reduce utility problems, a user-centered design approach has been acquired in which feedback from various individuals was obtained to understand their problems and requirements. The valuable insights gained from the feedback were then used to modify the system to best suit the requirements of the user. The object detection framework exploits a multimodal feature fusion-based deep learning architecture using edge, multiscale as well as optical flow information. Fusing edge information with raw data is motivated from the fact that stronger edge regions result in a higher number of activated neurons, hence inducing better feature representations. Learning deep features from multiple scales as well as use of motion dynamics at feature level lead to better semantic and discriminative representations, thus providing robustness to the detection framework. Experimental results carried out using PASCAL VOC 2007 dataset, Caltech dataset as well as captured real-time data are demonstrated.

Sharpness is an important indicator to evaluate image quality or to optimize parameters in computer vision tasks, such as image acquisition, compression, and restoration. We utilize difference quotients to construct an absolute difference quotient and a relative difference quotient to evaluate the sharpness among images containing difference contents and the sharpness among pixels in the same image, respectively. Based on the constructed quotients, we estimate the pixel sharpness index and the image block sharpness index and create a single sharpness index as the overall sharpness of an image by pooling strategy. Our quotient-based methods can assess image sharpness effectively and efficiently. Experimental results on four simulated databases with real blurring and synthetic blurring images show the proposed sharpness metric is consistent with subjective sharpness evaluations and is competitive with existing sharpness metrics. It achieves a balance between running time and high performance.

Considering the relatively poor real-time performance when extracting transform-domain image features and the insufficiency of spatial domain features extraction, a no-reference remote sensing image quality assessment method based on gradient-weighted spatial natural scene statistics is proposed. A 36-dimensional image feature vector is constructed by extracting the local normalized luminance features and the gradient-weighted local binary pattern features of local normalized luminance map in three scales. First, a support vector machine classifier is obtained by learning the relationship between image features and distortion types. Then based on the support vector machine classifier, the support vector regression scorer is obtained by learning the relationship between image features and image quality scores. A series of comparative experiments were carried out in the optics remote sensing image database, the LIVE database, the LIVEMD database, and the TID2013 database, respectively. Experimental results show the high accuracy of distinguishing distortion types, the high consistency with subjective scores, and the high robustness of the method for remote sensing images. In addition, experiments also show the independence for the database and the relatively high operation efficiency of this method.

A shadow-removal algorithm based on cycle generative adversarial network is proposed. There are two networks in the proposed method. To increase the diversity of shadow images for improving the robustness in the shadow-removal process, the first network is used to add shadows in nonshadow images to increase the variation of training data. Then, the second network is trained for the shadow-removal task. Six different losses are calculated and combined in the loss function to increase the performance. Ablation experiments show that the resulting images suffer from some artifacts without any of the six losses in the loss function. The proposed method presents lower value of root-mean-squared error and the superior visual quality compared to the state-of-the-art image removal algorithms.

Face detection from images has been an essential task in the field of computer vision, which is the premise of face recognition. Concerning the problem of low-face detection accuracy for multiscale face images in unconstrained settings, we propose single-stage headless face detector with IRNN, multilayer and soft-NMS face detection method based on single-stage headless face detector in unconstrained settings. First, considering the importance of the context for small-scale face detection, recurrent neural network initialized by the unit matrix module is joined to fully learn the contextual information. Second, in order to further improve the accuracy for multiscale face detection, multilayer fusion strategy is proposed, which learns the facial texture features from the lower layer in more detail. Finally, aiming at the face occlusion problem in unconstrained settings, soft nonmaximum suppression is used to join the predicted boxes together from different scales together to form final detection results. The results of experiments show that IMS-SSH significantly improves the performance of multiscale face detection in unconstrained settings, especially for small-scale face detection, and state-of-the-art performance is achieved on the unconstrained WIDER Face dataset.

We develop an automatic license plate recognition (ALPR) system for enhancing the investigation capabilities of law enforcing agencies to monitor suspicious vehicles. The recognition performance of real-time ALPR systems is affected to great extent in challenging conditions such as varying illumination, angle-of-view, different sizes of plates, changing contrast, and shadows. Moreover, character segmentation step sensitivity to plate resolution, size of characters, occluded characters, and width between characters makes it difficult to properly isolate the character, which further degrades recognition accuracy. In the first step of the proposed framework, a plate is localized using the faster region-based convolutional neural network method. In the second step, our study proposes a segmentation-free plate recognition approach that applies an adaptive boosting method with linear discriminant analysis for feature selection followed by matching the plates with a database for suspected vehicles and information retrieval. Simulation results show that the proposed framework is more robust to illumination variations, low-resolution images, different orientations, and variable license plate sizes than the conventional ones.

Occasionally, stripe noise infiltrates infrared images during acquisition, causing the image quality to severely deteriorate and affecting subsequent image analysis. To remove stripe noise, an adaptive comb-type notch filtering method is proposed, and a spatial mathematical stripe noise model and the comb-like impulse spectrum characteristics of stripe noise are theoretically and experimentally analyzed. The analysis results demonstrate that when periodic stripe noise consists of only intact stripes, the amplitude spectrum of the periodic stripe noise is an ideal impulse comb spectrum, and the noise frequency energy is concentrated; by contrast, when the periodic stripe noise contains incomplete stripes, spectral leakage occurs, and the noise frequency energy spreads to all frequencies. On this basis, optimal methods of estimating the stripe width and the size of the subimages to be processed are also presented. First, we crop the original image into two subimages of the optimal size to focus the comb-like impulse spectrum characteristics of the stripe noise, thereby enabling easier removal of the stripe noise patterns in the subimages. Then we detect the peaks in the amplitude spectra of the subimages based on the characteristics of the comb-like impulse spectrum and eliminate those peaks using an adaptive comb-type notch filter. Finally, we fuse the two filtered subimages into a final image. Experimental results show that the proposed method achieves a good effect in removing stripe noise from infrared images.

Change detection is the most fundamental component in video surveillance systems. Although many change detection approaches have been proposed, they are often only suitable for particular environments. This paper presents an approach that integrates several detection techniques with scene understanding capability, thereby overcoming the challenges of detecting various scene types and improving overall detection performance. First, a scene-awareness algorithm that incorporates a deep learning-based scene recognition model and support vector machine is developed to classify the monitored scene over time. Then, the appropriate detection technique is automatically adopted to perform scene-specific detection. Experimental results demonstrate that the performance of the proposed method is comparable to that of the state-of-the-art methods and satisfies the requirements of real-time practical applications. Hence, it can serve as an intelligent change detection approach for visual analytics in video surveillance systems.

Visual tracking is a challenging task in computer vision due to various appearance changes of the target. Although correlation filter-based trackers have achieved competitive results, they may easily lead to tracking failure because of the high sensitivity of correlation filter to occlusion. Part-based correlation filter trackers can deal with partial occlusion to some extent, but they may easily drift to the background in the case of fast motion or heavy occlusion. To better solve the above-mentioned problems, a kernelized correlation filter-based tracker that processes both holistic and reliable local parts is proposed. For local parts, reliable parts are identified by peak-to-sidelobe ratio. When all parts are unreliable, we propose to apply a sliding window on each part to generate patches, among which a reliable patch is identified, and the part is replaced by the reliable patch. In holistic level, holistic tracking is performed with the rough position voted by reliable local parts, and then the holistic tracking result is used to provide feedback for parts to update its scale and filter. Moreover, we propose to reset unreliable parts when the holistic tracking result is reliable. The experimental results illustrate that the proposed tracker outperforms those of several state-of-the-art trackers.

Unsupervised domain adaptation aims to utilize knowledge from a source domain to improve learning in a target domain in cases in which abundant labeled samples are available in the source domain, but no labels exist in the target domain. The two domains have the same feature space and label space but different distributions. Our study proposes a semisupervised framework in which labeled samples in the source domain and unlabeled target samples are fully utilized to learn a better classifier. The framework contains two stages: semisupervised feature learning and semisupervised classifier learning. In the first stage, a transformation is learned using labeled source samples and unlabeled target samples to map these data into a representation. In the second stage, a classifier is learned using all samples in the source and target domains of the representation. Furthermore, we propose a semisupervised feature learning approach (i.e., cross-domain discriminative analysis) to learn the transformation in the first stage by reducing the distribution discrepancies between domains and preserving discriminative information in the original data. In our experiments, image classification tasks were conducted using several well-known cross-domain datasets. The proposed method outperformed the state-of-the-art methods in most cases.

No-reference image quality assessment (NR-IQA) aims to predict image quality consistently with subjective scores with no prior knowledge of reference images. However, contrast distortion, which is an uncommon distortion, has been largely overlooked. To address this issue, we explore the NR-IQA metric by predicting the quality of contrast-altered images, using deep-learning techniques. We adopt a two-stage training strategy due to a gap between the deep learning’s sample requirements and the insufficiency of samples in the IQA domain. A deep convolutional neural network (CNN) is first designed and is pretrained to the classification task with the help of an additional synthetic contrast-distorted dataset. Then, the pretrained CNN is fine-tuned on the target IQA dataset using an end-to-end training approach. An effective pooling method is employed to map the image representation into a subjective quality score during the fine-tuning stage. Experimental results on five public IQA databases containing contrast-altered images show that the proposed method achieves competitive results and has good generalization ability compared to other NR-IQA methods.

The loss function plays an important role in model training for the single-image super-resolution task. Most convolutional neural network-based models adopt conventional pixel-wise loss functions to make impressive advances in peak signal-to-noise ratio and structural similarity index. However, these losses tend to find the average of plausible solutions, which lead to overly smoothed SR results with a lower visual perception. We propose a loss function combining the statistics loss with semantic priors and the quality assessment loss to produce an HR image with high visual quality while maintaining natural image statistics, as perceived by human observers. Our statistics loss measures the similarity of deep feature distributions in different semantic blocks and contributes to the maintenance of natural internal statistics in image restoration. Additionally, a no-reference quality metric that focuses on several aspects of human perceptual preferences for lighting, tone, and sharpness is introduced in our loss function to provide a more visually compelling approximation of human visual perception for perceptual image super-resolution. Experiments prove that our loss function can effectively guide the network to generate images of high-perceptual quality while considering the structural distortion for single-image super-resolution.

An adaptive joint sparsity model (JSM) is presented for multimodal image fusion. As a multisignal modeling technique, JSM, which is derived from distributed compressed sensing, has been successfully employed in multimodal image fusion. In traditional JSM-based fusion, a single dictionary learned by K-singular value decomposition (SVD) has higher coherence yet may result in potential visual confusion and misleading. In the proposed model, we first learn a plurality of subdictionaries and use a supervised classification approach based on gradient information. Then, one of the learned subdictionaries is adaptively applied to JSM to obtain the common and innovative sparse coefficients.. Finally, the fused image is reconstructed by the fused sparse coefficients and the adaptive dictionary. Infrared-visible images and medical images were selected to test the proposed approach. The results were compared with those of traditional methods, such as the multiscale transform-based methods, JSM-based method, and adaptive sparse representation (ASR) model-based method. Experimental results on multimodal images demonstrate that the proposed fusion method can obtain better performance than the conventional JSM-based method and ASR-based method in terms of both visual quality and objective assessment.

Low-rank methods have been widely exploited in image denoising and have shown admirable denoising performance, of which weighted Schatten p-norm minimization (WSNM) is particularly effective. However, the WSNM method which applies Frobenius-norm loss model cannot obtain a satisfactory denoising performance when images corrupted by impulse noise. An optimization strategy based on the alternating direction method of multipliers framework is used to solve the proposed model efficiently. Experimental results show that the proposed method outperforms some state-of-the-art denoising methods both quantitatively and qualitatively under various impulsive noise models.

An image stitching technique is introduced to automatically composite a panorama of the corroded surface with a series of scanned microscopic corrosion images. In this method, the features in the overlapping area of adjacent corrosion images are extracted to establish the geometric transformation for registration, and the weighted average method is used to minimize the visible seam existing in the stitching result. A sequence images alignment strategy is introduced to complete the whole stitching process. The effect of overlapping area on stitching accuracy was examined and the advantage of applying image stitching method in corrosion morphology analysis is demonstrated by the segmentation of corrosion image with support vector machine (SVM) based on texture feature. The result shows that the high-resolution corrosion image can be segmented more accurately and easily. Therefore, it is believed that image stitching is a powerful method for the analysis of corrosion morphology in microscopic images, which could make the quantitative study of the corrosion process easier and more efficient.

We present a segmentation method that estimates the relative coverage of each pixel in a sensed image by each image component. The proposed super-resolution blur-aware model (utilizes a priori knowledge of the image blur) for linear unmixing of image intensities relies on a sparsity promoting approach expressed by two main requirements: (i) minimization of Huberized total variation, providing smooth object boundaries and noise removal, and (ii) minimization of nonedge image fuzziness, responding to an assumption that imaged objects are crisp and that fuzziness is mainly due to the imaging and digitization process. Edge fuzziness due to partial coverage is allowed, enabling subpixel precise feature estimates. The segmentation is formulated as an energy minimization problem and solved by the spectral projected gradient method, utilizing a graduated nonconvexity scheme. Quantitative and qualitative evaluation on synthetic and real multichannel images confirms good performance, particularly relevant when subpixel precision in segmentation and subsequent analysis is a requirement.

The unpreceded growth of intelligent surveillance systems has resulted in an urgent need for automatic analysis of the captured scenes. Automatic detection of an abnormal crowd in aerial images can provide useful information to prevent disasters. In fact, aerial images have the advantage of covering a very large view of the people distributed over a scene. Accordingly, we propose a high density crowd detection method in aerial images. In this method, we adapted the bag of words technique using the multiblock local binary pattern as a texture descriptor to extract low-level features. In addition, we also used a three-level classification strategy to reduce confusion between crowd density classes. The experimental results reveal the performance of our method while estimating the crowd density in a challenging context.

Visual attributes, as a bridge between humans and machines, have played an important role in various applications. We focus on attribute-based image retrieval. Unlike early works that either initiate the image search via binary attributes or refine the search results via relative attributes, we propose to realize image retrieval using grouped attribute strength, which are learned from both binary attributes and relative attributes. First, early methods are employed to predict the binary attributes and sort the relative attributes in descending order in different situations. Second, we group the sorted images corresponding to each attribute and combine the learned binary and relative attributes to learn the grouped attribute strength of the images, which result in a new attribute feature vector for each image. Finally, we apply the learned grouped attribute strength to different retrieval models to realize our image retrieval tasks. We demonstrate the approach on LFW-10 and Shoes datasets and show its clear advantages over traditional binary attribute-based retrieval methods.

In free-viewpoint rendering systems, one of the most challenging goals is the creation of virtual views based on available color texture (RGB) and depth data. Conventional depth-image-based rendering (DIBR) approaches have assumed that the virtual camera can only be displaced horizontally, thus leading to fairly simple disocclusion artifacts. However, in free-viewpoint DIBR, the virtual camera can be positioned in an arbitrary way and the respective disocclusion artifacts can exhibit complicated anisotropic appearances. Consequently, conventional approaches for compensating disocclusion holes usually fail in such arbitrary camera motion. We present a disocclusion compensation technique based on texture inpainting. We propose a layered representation of both the color and depth images in local foreground, background, and undefined segments (a trimap). This representation allows for employing an efficient alpha-matting approach for reconstructing the underlying opacity layer followed by a background compensation and layered rendering. The performance of the proposed method is evaluated with respect to the state-of-the-art through objective and subjective tests. The achieved results, especially for large camera displacements, outperform the state-of-the-art. Those results assess the effectiveness of the proposed method and highlight the need for new quality metrics able to address the impairments of this type of content.

Low-light stereo vision is a challenging problem because images captured in dark environment usually suffer from strong random noises. Some widely adopted algorithms, such as semiglobal matching, mainly depend on pixel-level information. The accuracy of local feature matching and disparity propagation decreases when pixels become noisy. Focusing on this problem, we proposed a matching algorithm that utilizes regional information to enhance the robustness to local noisy pixels. This algorithm is based on the framework of ADCensus feature and semiglobal matching. It extends the original algorithm in two ways. First, image segmentation information is added to solve the problem of incomplete path and improve the accuracy of cost calculation. Second, the matching cost volume is calculated with AD-SoftCensus measure that minimizes the impact of noise by changing the pattern of the census descriptor from binary to trinary. The robustness of the proposed algorithm is validated on Middlebury datasets, synthetic data, and real world data captured by a low-light camera in darkness. The results show that the proposed algorithm has better performance and higher matching rate among top-ranked algorithms on low signal-to-noise ratio data and high accuracy on the Middlebury benchmark datasets.

We present a framework based on the development of adaptive scalable kernel (ASK) for hyperspectral image classification, which can achieve an excellent status in removing insignificant details and defending crucial features. The proposed method consists of three steps. First, the spectral feature extraction based on interval gradient and a fast morphological filter is used to reduce the high dimensionality. Second, a powerful spatial structure extraction method based on adaptive scale kernels is adopted to enhance the performance of structure-preserving filtering. Depending on patch-based statistics, this model identifies small-scale texture from large-scale structure and finds an optimal per-pixel smoothing scale. Third, the obtained spectral structure feature maps are classified with the large-margin distribution machine. The experimental results show that the proposed spatial structure extraction method based on ASK achieves the state-of-the-art performance in terms of classification accuracy and computational efficiency.

Recently, image-to-video person reidentification (IVPR) has attracted enormous research interest, and various models are proposed. IVPR is often applied to urgent situations, such as suspect tracking and lost-human locating. Existing IVPR models are under supervised frameworks, which require a large number of labeled image-to-video pairs. This severely limits their real-time efficiency in urgent situations, because annotation is much more time-consuming. To solve the urgent image-to-video person reidentification (UIVPR) problem, we propose a cross-media transfer cycle generative adversarial networks (CTC-GAN) network. Our model aims to alleviate the “media-gap” between image-to-video pairs without newly labeled pairs. We make an existing completely labeled dataset as guidance for CTC-GAN to achieve domain adaptation and make urgent image-to-video matching easier for person reidentification. We introduce cycle GANs for image(video)-to-video(image) translation and extract cross-media features using a triplet constraint in the source domain for different media features. Furthermore, we train the model in the labeled source domain by reconstructing the image (video) as its related video (image). Then, train the model in the unlabeled target domain by reconstructing itself along with source data, so as to ensure that the discriminative model can be used in target domain. Through CTC-GAN, our network can retain pedestrian discriminative information as much as possible, to ensure the matching rate in the target domain. To validate the effectiveness of our approach, we implement substantial experiments on two large-scale person reidentification datasets compared with six existing state-of-the-art unsupervised revised person reidentification models, and experimental results demonstrate that our method can solve UIVPR effectively.