We propose a yet lightweight no-reference (NR) video quality assessment (VQA) method, which uses a convolution neural network (CNN) architecture. The proposed method implements a spatiotemporal saliency patch selection procedure that crops the frame into small nonoverlapping blocks of images (patches) and selects the most perceptually relevant ones. The selected patches are then forwarded to the CNN. To determine which patches are the most relevant, spatial and temporal saliency features are computed for each frame. The proposed method does not require subjective scores to train the CNN. It uses objective quality scores as target quality scores for each video frame, which are computed using an NR image quality assessment method. Given the lack of large annotated video quality databases, this is an advantage of the proposed method. Finally, although it has much smaller cost of data-processing, compared with other state-of-the-art methods, the proposed NR-VQA obtains robust and competitive results.

To improve the facial expression recognition accuracy in resource-constrained and real-time application equipment such as mobile and embedded devices, a lightweight method for facial expression recognition is proposed based on attention mechanism and key regions fusion. To reduce the computation complexity, a lightweight convolutional neural network, mini_Xception, is used as the basic expression recognition model for expression classification. The attention mechanism is introduced to enhance the learning of the important features of the whole face. Then a parameter is introduced to locate the key regions and construct key region models. Finally, to realize the complementarity of models and learn more comprehensive features, the whole facial expression recognition model is fused with the key region models. The proposed method can capture and utilize the important facial expression information in related regions displayed through class activation mapping visualization. The experimental results on JAFFE, CK+ datasets, and a real scene dataset verify the effectiveness of the proposed method.

Lipreading feature extraction is essentially the feature extraction of continuous video frame sequences. A lipreading model based on a two-way convolutional neural network and features is proposed to obtain more reasonable visual-spatial–temporal characteristics. Unlike other lipreading methods based on deep learning, the rank pooling method transforms lip video into a standard RGB image that can be directly input into the convolutional neural network, which effectively reduces the input dimension. In addition, to compensate for the lack of spatial information, the apparent shape and depth features are fused, and then the joint cost function is used to guide the network model learning to obtain more distinguishing features. The experimental results were evaluated on the public GRID database and OuluVS2 database. It shows that the accuracy of the proposed method can reach more than 93%, which validates the effectiveness of the method.

Video frame interpolation is used to generate intermediate frames by estimating the movement of pixels between the input frames. However, problems of blurring, object occlusion, and sudden brightness changes occur in naturally obtained video frames. We propose a context-based video frame interpolation method via depthwise over-parameterized convolution. First, the proposed network obtains the context graphs of the input frames. Subsequently, an adaptive collaboration of flows is adopted to warp the input frames and the context graphs. Then, the frame synthesis network is used to fuse the warped input frames and context graphs to obtain a preliminary estimate of the interpolated frame. Finally, a post-processing module is employed to refine the result. Experimental results on several datasets demonstrate that the proposed method performs qualitatively and quantitatively better than state-of-the-art methods.

With the continuous improvement in the accuracy of steganalysis based on convolutional neural networks (CNNs), the network scale has shown explosive growth. Consequently, CNNs have a high demand for hardware resources and time-consuming training. To reduce the number of CNN parameters and improve the efficiency of steganalysis, we propose a lightweight steganalysis CNN called W-Net. The proposed W-Net first uses grouped convolution and channel shuffling units to extract noise residuals, strengthen the information exchange between groups, and improve feature extraction. In addition, the depth-wise separable convolution is applied to obtain different channels and spaces. The fusion of position information achieves the effect of conventional convolution while reducing the number of network parameters. We verified the effect of activation functions on steganalysis accuracy through experiments. In addition, the proposed W-Net can detect the steganographic data from the S-UNIWARAD spatial steganography algorithm with an embedding rate of 0.4 bpp. Compared with Xu-Net and Zhu-Net, the proposed W-Net improves the detection accuracy by 12.70% and 6.38%, respectively.

A superpixel segmentation algorithm called multi-density peaks clustering (MDPC) is proposed. By selecting a sufficient number of local density maximum pixels from the image as cluster centers to depict the image texture, the boundary of the object can be captured very accurately. The algorithm framework of MDPC is divided into three steps. First, the local density of pixel is defined, and the local density maximum pixels are calculated. Then, the local density maximum pixels are used as cluster centers, and the global optimal search, which is based on the path-to-point idea, is used to complete the clustering of the remaining non-cluster center pixels to realize the initial segmentation. Finally, superpixels are obtained by merging the initial segments according to the size of the segments and the distance between adjacent segments. In quantitative comparisons, MDPC is compared with 13 state-of-the-art superpixel segmentation algorithms in three image segmentation datasets. The experimental results show that MDPC achieves better performance in terms of boundary recall, boundary precision, achievable segmentation accuracy, undersegmentation error, and explained variation. And the qualitative comparisons show that the proposed algorithm has obvious advantages over other superpixel segmentation algorithms in image detail description and boundary adherence. Finally, the practicability and stability of MDPC are further demonstrated by the application of image segmentation. The source code of MDPC will be available at https://github.com/zhaojianaaa.

I present an innovative registration algorithm using the particle flow filter. The particle flow filter is a Bayesian filter that uses particles to represent probability densities. The particle flow filter is not constrained to the highly restrictive unimodal, linear, and Gaussian assumptions of many Bayesian filters such as the Kalman filter. Additionally, the particle flow filter is computationally more efficient than other multimodal filters such as the better-known particle filter (PF). Unlike the PF, the particle flow filter does not require particle resampling or importance weight updates. Rather, the proposal density is formed by flowing the prior probability to the posterior using the Fokker–Planck equation. The particle flow filter algorithms were implemented using MATLAB. Both 2D and 3D rigid body point-set registration were conducted using the Gromov particle flow filter variant. Additionally, the PF method and iterative closest point (ICP) algorithms were implemented for comparison. For the same alignment accuracy, the new particle flow filter approach was 244% faster than the PF for certain challenging problems. For the same alignment time, the particle flow filter reduced misalignment by as much as 35% over that of the PF. The particle flow filter achieved 100% alignment with enough particles, and reduced misalignment by as much as 75% over that of ICP. These results demonstrate that image registration via the particle flow filter significantly outperforms the PF and ICP algorithms in the presence of noise and for a high degree of initial misalignment.

Haze/fog is a common weather phenomenon that may exist in both day and night conditions and presents loss of contrast in captured images. For an imaging device to capture good quality photographs, it is necessary that the captured scene be well-illuminated. But sometimes due to bad illumination, which may be due to natural or manmade conditions, the captured images present degradation. To solve the ill effects of bad illumination, we propose an image enhancement algorithm that sufficiently deals with both hazy and dark images. By combining dynamic stochastic resonance and a fusion method based on illumination estimation, a local contrast enhancement of the image as seen in the fused image is achieved in the first stage. While images that are only dark are sufficiently illuminated, the effect of haze in day/night hazy images remains unchanged. In the second stage, we solve this issue by modifying the atmospheric degradation model to reduce the effect of haze. The proposed method does not rely on the estimation of atmospheric light; rather, a mean preserving algorithm to restore the mean brightness of the image has been applied. The qualitative and quantitative experiments validate the efficacy and robustness of the proposed method.

Few-shot image classification commits to recognizing new concepts from limited annotated samples. Our insight is to obtain a sufficiently powerful embedding network (PEN) to solve few-shot classification tasks. We propose a method to tackle the few-shot classification tasks, namely PENs for few-shot image classification. The key core of PEN is gaining a well-trained embedding network that is capable of extracting strong discriminating representations to represent an image by utilizing two strategies. One strategy is that the multi-scale feature maps are fused instead of only utilizing the final top-level feature maps. We consider that low-level features also play an important role instead of only utilizing top-level representations. Another significant strategy is knowledge distillation (KD). The characteristics of KD can help us get better performance of an embedding network to extract features. Finally, a distance function is employed to classify unlabeled samples. Comprehensive experiments are conducted on few-shot benchmarks. Our method achieves promising performances. The results demonstrate that KD and future fusion are beneficial to gain an expected embedding network for few-shot classification tasks.

There are two application problems in the iris multi-category recognition scenes, namely: when adding a new template category, the expansion convenience problem caused by the difficulty of category expansion, and the environment independence demand caused by the distortion of the iris image information. To solve these two application problems, we propose an iris recognition model. The model is divided into two stages, namely: the first recognition stage and the second recognition stage. According to the orderly arrangement in the same category sample clustering range of each dimensional feature value, a 32-dimensional continuous vector space is formed as the first recognition stage feature knowledge. The 32-dimensional ordered continuous array on the basis of grayscale stable features is used as the feature knowledge in the second recognition stage. The result in the first recognition stage is divided into three types: result category, pending category, and elimination category. The second recognition stage is a specific process that is initiated when the result category is not unique or there is a pending category. Through a specially designed non-template matching function, accurate result can be obtained in the pending categories. The results of experiments with different iris libraries verify that continuous feature space based on image texture can effectively reduce the influence of image information distortion. Additionally, each feature data dimension as a training unit is conducive to the independent training of feature knowledge in single category. It can add new iris categories without interference and solve the problem of expansion convenience.

With the recent development of machine learning theory, many deep learning based methods have been proposed to extract features from hyperspectral sensed images (HSIs) and superior performance has been achieved. Although the data of HSIs is characterized by its high volume as well as with rich spectral and spatial information, the number of the labeled samples is always very limited. But many deep learning methods, especially the supervised approaches, often require numerous labeled samples to extract the effective features, and a prolonged training procedure is inevitable. To expand the set of the labeling samples, a huge amount of manpower and material resources are required, which may make the application unaffordable. In order to alleviate the dependence on HSIs’ labeled samples, in this paper an unsupervised spatial-spectral feature extraction method is proposed. Combining the two-dimensional convolutional neural network (2DCNN) with generative adversarial networks (GANs), the requirement of the labeled samples is reduced for the training of a learning network, and the spatial-spectral characteristic of the HSIs is utilized at the same time. Moreover, the principal component analysis (PCA) is introduced as a preprocess for the high-dimensional hyperspectral data, which not only removes redundant information, but also fits the proposed feature extraction model to hyperspectral data with different dimensions. To improve the generalization capability of the proposed model, a gradient penalty term is further investigated in the objective function of the GAN’s discriminator, which can optimize the discriminator’s convergence and improve the model’s overall generalization capability. Finally, a support vector machine is applied to classify the extracted features. Experimental results on three real data sets show that the proposed method can extract features from the hyperspectral data effectively. Through the GANs-based feature extraction and the specialized regularization, the proposed method achieves a good level of generalization capability, in which a model validated on a single dataset can be extended to different datasets while a satisfactory performance still remained.

The skeleton-based action recognition task has recently drawn much attention in computer vision, and graph convolutional networks (GCNs) have shown great advantages in this task. However, in shallow layers of GCNs, the sharing of convolution kernels in regular graph convolution (RGC) may result in the underutilization of some convolution kernels due to the particularity of the skeleton data. To capture richer locally discriminative action features, we propose the exclusive kernel graph convolution (EKGC) for the shallow layers of GCNs, which assigns the vertexes exclusive convolution kernels for graph convolution. At the same time, following the mixed graph convolution strategy, RGC is retained in the deep layers of the network, and the mixed graph convolutional network (MGCN) is proposed to utilize EKGC and RGC reasonably in all layers of the network. In addition, as a new form of skeleton information in GCN-based methods, angle information is explored in our work. Compared with joint data and bone data, angle data have better invariance against rotation and scaling, and they expand the relationships between skeletons and enrich the data representation of action. Finally, our final model, the multi-stream MGCN, is validated on two challenging datasets, achieving the state-of-the-art performance on the NTU-RGBD dataset and making competitive performance on a kinetics-skeleton dataset.

With the flourishing development of deep learning in the field of computer vision, research of defocus deblurring based on it has gradually become a hotspot. However, most of the research focuses on defocus region detection or defocus map estimation, and algorithms for directly generating restoration images are less studied. Stressing on the problems of defocus deblurring, we propose a defocus deblurring deep model based on multi-scale information and convolution neural network. Concretely, we first perform an efficient and concise multi-scale information fusion by the selective receptive field module, thus the model can adapt to the scale sensitivity of the image defocusing region. We then use the residual channel attention module in the bottleneck module to extract the correlation features between channels, which enhances the effective channels and suppress the useless ones. Finally, a fusion objective function of edge loss and mean square loss is proposed to enhance the edge details of the image. Experimental results on a large-scale defocus deblurring dual-pixel dataset demonstrate that the proposed model has better performance than the traditional and existing deep-based methods. Comparing with the methods of the state of the art, the proposed model has a 0.44-DB improvement in PSNR metric.

Identifying and analyzing the sentiment information of an image is a complicated task in content understanding with nondeterministic features. To establish an effective model for multidimensional emotion recognition of film and television scene images and explore the relationship between the inputs and prediction results, we chose eight feature selection schemes based on principal component analysis, least absolute shrinkage operator (Lasso) regression, Pearson correlation, and the importance of random forest (RF) to establish the input vectors and the relevant model of machining learning. After the test on the data set with fixed combination of dimension reduction of 19 emotional descriptors, the fuse of Lasso regression and the RF importance extraction were proved to have the best performance on accuracy with the regression models. The experimental results showed that our algorithm performed well on the regression task of multidimensional affective dataset, which was convenient and time-saving. The extracted features and relevant dimension reduction methods also achieved good results on the cross-dataset classification tasks.

An accurate iris segmentation method is crucial for the iris recognition system. The conventional iris segmentation algorithms have poor adaptability when applied to heterogeneous iris databases. Therefore, researchers have applied deep learning to the field of iris segmentation. A modified U-Net is proposed to perform the iris segmentation for heterogeneous iris databases, referred to as DropBlock and modified shortcut branch U-Net. The main work is as follows: first, EfficientNetV2 based on DropBlock is used as convolutional blocks of U-Net to improve the ability of feature extraction and generalization of the network. Second, an improved shortcut branch structure is proposed for U-Net to reduce the loss of information during the downsampling process. The experimental results on the CASIA-iris-interval-v4, IITD, and UBIRIS.v2 iris databases demonstrate that this method can not only have good versatility but also provide higher accuracy on heterogeneous databases. Compared with some state-of-the-art fusion methods, the proposed method has a significant performance advantage.

Currently, robust low-rank tensor reconstruction based on tensor singular value decomposition (t-SVD) has made remarkable achievements in the fields of computer vision, image processing, etc. However, existing works mainly concentrate on third-order tensors while order-d (d  ≥  4) tensors are commonly encountered in practical applications, such as fourth-order color videos, fourth-order facial images, fifth-order light-field images and sixth-order bidirectional texture functions. Aiming at this critical issue, we establish a robust order-d tensor reconstruction framework including the model, algorithm and theory by developing a novel algebraic foundation of order-d t-SVD with any invertible linear transforms. Equipped with the newly developed framework, any order-d low-rank tensors can be accurately reconstructed from a sample of its entries corrupted by arbitrary outliers. The superiority and effectiveness of the proposed framework are substantiated in comparison with state-of-the-art alternatives by conducting experiments on both synthetic and real-world tensor data.

We propose a framework named sampling and whole image denoising network based on generative adversarial network (GAN) for compressed sensing image reconstruction (SWDGAN) to reconstruct natural image from compressed sensing (CS) measurements. This work is devoted to balancing the performance of reconstruction quality, practicability, and running efficiency. Different from the recent deep learning reconstruction networks, we further enhance the feature representation and remove the blocking artifacts by introducing a whole image dense residual denoising module without affecting running efficiency. To improve the flexibility of sampling process and the practicability of our algorithm, a fully connected network without bias is applied in the sampling process, whose weight can be extracted separately and used as a measurement matrix. In this way, the measurements can be obtained by matrix multiplication in multiple running environments, not limited to deep learning framework. Besides, the sampling network can also improve the reconstruction quality even at low sampling ratios. Moreover, we remove batch normalization (BN) layer of reconstruction network to avoid the influence of BN artifact on reconstruction image. The experimental results illustrate that our method outperforms the most advanced traditional methods and deep learning-based methods in terms of reconstruction quality and running time.

Due to the intrinsic limitations of hyperparameters, most dehazing algorithms easily suffer from overenhancement or staircase artifacts. To suppress these problems, a spatially adaptive total generalized variational (TGV)-based single image dehazing algorithm is developed. The algorithm contain two modules, i.e., an adaptive coarse transmission map estimation module and a spatially adaptive TGV-based transmission refine module. The adaptive coarse transmission map estimation module is designed to prevent the dehazing result from being overenhanced with a dark channel-based adaptive control parameter. The TGV-based transmission refine module is designed to refine the coarse transmission with a texture prior-based spatially adaptive TGV-regularized variational model. Numerous experiments reveal that the proposed algorithm is comparable with or even outperforms the state-of-the-art techniques.

Distinguishing the target from the background, judging target occlusion, and real-time processing are the problems that the visual tracking algorithm still needs to solve. Color information and position information of the target block are fused as new features to track the target under the framework of particle filtering. First, the hues, saturation, value space, and color integral graph of the image are constructed. The vector representation of the target is obtained on the color integral image by sparse matrix. Then, candidate particles are produced by a particle filter and the sampling mode of particles is adjusted by a uniform acceleration model. The difference of particles reflects the position and scale change of the target. Finally, the candidate with the smallest eigenvector projection error is taken as the tracking target and the feature template is updated based on the tracking results. The presented algorithm can be used to track a single target in the color image sequence and has some robustness to the scale change, occlusion, and morphological change of the target. Experiment results on public datasets show that the proposed algorithm performs favorably in both speed and tracking effect when compared with other conventional trackers.

Correlation filter (CF)-based trackers have demonstrated impressive performance on visual tracking benchmark datasets with high frame rates. A lot of research focuses on extracting strong features to learn a robust model of the object’s appearance. However, the circularly shifted samples used in traditional CF training come from one single image patch and have limited information about the target. Inspired by various data augmentation methods in deep learning, we propose a general multi-templates correlation filter-based tracker that expands the CF tracker with multihandcrafted training samples. We reformulate the original optimization problem and provide a closed-form solution to maintain the high-speed calculation. Several methods to generate the additional samples are evaluated. Comprehensive experiments illustrate that our framework obviously improves the tracking performance and is general enough to be incorporated with classical CF trackers.

We present an online multi-object tracking algorithm to track multiple objects across a large number of image frames. Our work is motivated by the need to study evolution of nanoscale objects by transmission electron microscopy. The proposed approach is based on the existing multi-way data association tracking algorithm that is capable of tracking interacting objects with complex behaviors (i.e., merge, split, overlap, and appearance or disappearance). The multi-way data association is an offline algorithm to associate objects across all image frames at one step with a global optimization, which does not scale very well for large number of image frames. The proposed online tracking algorithm processes image frames as they arrive by detecting all objects in the newly arrived image frame and making the associations of the objects to those detected from the previous frame by the multi-way data association. This frame-by-frame association scheme can cause fragmented traces of the objects that are occasionally misdetected for some image frames. We overcome this issue by allowing previously unassociated objects to be associated when the objects reappear within a fixed number of future image frames, namely the frame-delayed association. We combine the multi-way data association with the frame-delayed association to be able to track interacting objects with accurate handling of object disappearance events. The proposed method is validated through applications to simulated multi-object tracking problem and a real multi-object tracking problem. The outcome of the proposed method is compared with four state-of-the-art algorithms.

Color enhancement factors are spectral components that modulate the spectral power distribution (SPD) of illumination for interactive color saturation control. They can enhance more than one target color simultaneously or independently while maintaining the current color appearance. However, the color enhancement factors obtained in our previous work depended on the lighting system. Moreover, the effects of color muting, such as decreasing saturation of the target color, were limited because the color enhancement factors were optimized to increase color saturation. We overcame these two issues and succeeded in implementing color enhancement factors that can both enhance and mute color saturation. First, we conducted detailed simulations of color saturation enhancement and muting to determine parameters for calculating the color enhancement factors (e.g., the center wavelength and full-width at half-maximum) and assessed their potentiality. The results showed that maximum variations of color muting became almost the same as those of color enhancement. We also found that SPDs of illumination for increasing and decreasing color saturation of red, green, and blue had three peaks, respectively, and that they appeared for enhancement and muting alternately. In addition, the color enhancement factors for enhancing and muting color saturation were roughly symmetric. We then performed experiments using a newly designed 12-color LED lighting system, and the results showed color enhancing and muting effects. Finally, we demonstrated color restoration of discolored cultural heritage and coloring simulation of industrial products.

Currently, the reported image hashing schemes cannot perform satisfactorily in some aspects, such as rotation robustness. To this end, a perceptual color image hashing scheme is proposed. First, the original image is normalized and smoothed by Gaussian low-pass filter. The obtained secondary image is divided into a series of ring-ribbons with different radii and the same number of pixels. Then, textural and color features are extracted in local and global manners. Specifically, local textural features are extracted on luminance values of the ring-ribbons using quadtree decomposition, and global textural features are extracted by gray level co-occurrence matrix. Local color features of significant corner points are extracted on outer boundaries of ring-ribbons through color vector angles, and global color features are extracted by low-order moments. The extracted features are concatenated after quantization and permutation to generate the final hash. Receiver operating characteristic curves verified the effectiveness of our scheme, including robustness, discrimination, and security, which can be effectively applied in content authentication and tampering detection.

With the broad application of iris recognition systems, iris images acquired under cross-sensor and cross-spectrum conditions are increasingly common. Differences in the information stored in these images place high requirements for accuracy and robustness on cross-domain iris recognition algorithms, and current algorithms have not performed well in cross-domain iris recognition. Hence, we propose a dense squeeze and excitation network (DenseSENet) to extract common features in cross-domain iris images and use real-valued features for cross-domain matching. DenseSENet is a deep convolutional neural network based on an attention mechanism. It uses the batch normalization-rectified linear unit-Conv1  ×  1 structure to increase the nonlinearity of the network and repeatedly uses features through dense connections to effectively prevent overfitting. The proposed dense squeeze and excitation structure extracts more accurate and stable features by learning the relations between channels and assigning corresponding attention values. The structure of global average pooling-fully connected-softmax is utilized to obtain compact and normalized real-valued feature templates to effectively reduce the size of the iris template. The proposed cross-domain iris recognition framework based on DenseSENet is end-to-end, which transforms the cross-domain matching problem to a many-to-one classification problem. Experiments are conducted on publicly available cross-sensor and cross-spectral iris databases. The results show that the proposed real-valued feature matching based on DenseSENet has better matching accuracy and robustness than the state-of-art cross-domain matching algorithm on two publicly available cross-sensor and cross-spectral iris databases.

We propose a method that can reason and find hidden objects from a set of images. It discovers relationships between all objects detected by a backbone neural network. We focus on object searching and location determination in the real world. Given the name of an object, the system detects the object and outputs a bounding box containing it if it is in sight. Otherwise, the system outputs bounding boxes containing relevant objects around which the target object is most likely to appear. The primary process of the system includes training and reasoning. The former establishes system experience of object relations, and the latter implements object searching based on this experience. The system consists of a relational discovery module and a searching module. We have tested the proposed method on multiple datasets (COCO, PASCAL VOC, and ImageNet) and multiple backbone networks, and the results show that the proposed method has strong robustness and generalizability. The system can continuously highlight the most relevant objects in the line of sight, thus providing hints for approaching and locating the target objects until they are found, and the proposed method can meet the requirements of real time and accuracy if a suitable backbone network is selected.

Most existing gesture recognition algorithms have low recognition rates under rotation, translation, and scaling of hand images as well as different hand types. We propose a new hand gesture recognition algorithm that combines the hand-type adaptive algorithm and effective-area ratio based on feature matching. Samples are divided into several groups according to the subjects’ palm shapes and the algorithm is trained using self-collected data. The user’s hand type is paired with one of the sample libraries by the hand-type adaptive algorithm. To further improve the accuracy, the effective-area ratio of the gesture is calculated based on the minimum bounding rectangle, and the preliminary gesture is recognized by the effective-area ratio feature method. The results of experiments demonstrate that the proposed algorithm could accurately recognize gestures in real time and exhibits good adaptability to different hand types. The overall recognition rate is over 94%. The recognition rate still exceeds 93% when hand gesture images are rotated, translated, or scaled.

A fast and reliable motion detection method is proposed for moving cameras. Existing methods generally build background models and then apply background subtraction for moving pixels detection. In background modeling, statistical methods and/or features, such as edges, local binary patterns are used. Parameters of the proposed methods need to be updated adaptively for better performance. In recent studies, adaptive learning rates are used with main and candidate background models. Simple and adaptive thresholding methods are used for foreground detection in different studies. We build a statistical background model with hue–saturation–value color space and perform background subtraction with the contribution of optical flow vectors. Dense optical flow is applied between warped the background model and the current frame to estimate optical flow vectors. If a pixel has a larger magnitude, we set a weight value for each pixel in the background subtraction process. Then, we extract two candidate foreground masks and apply some postprocessing techniques to determine the final mask. Our proposed method works fast in real time, and the experimental results on two different datasets show that our approach has the highest performance compared with the methods in the literature.

Document layout analysis, or zoning, is important for textual content analysis such as optical character recognition. Zoning document images such as digitized historical newspaper pages are challenging due to noise and quality of the document images. Recently, effective data-driven approaches, such as leveraging deep learning, have been proposed, albeit with the concern of requiring larger training data and thus incurring additional cost of ground truthing. We propose a zoning solution by incorporating a knowledge-driven document representation, gravity map, into a multimodal deep learning framework to reduce the amount of time and data required for training. We first generate a gravity map for each image, considering the centroid distance and area between a cell in a Voronoi tessellation and its content to encode visual domain knowledge of a zoning task. Second, we inject the gravity maps into a deep convolution neural network (DCNN) during training, as an additional modality to boost performance. We report on two investigations using two state-of-the-art DCNN architectures and three datasets: two sets of historical newspapers and a set of born-digital contemporary documents. Evaluations show that our solution achieved comparable segmentation accuracy using fewer training epochs and less training data compared to a naïve training scheme.

k-Means (KM) is well known for its ease of implementation as a clustering technique. It has been applied for color quantization in RGB, YUV, hyperspectral image, Lab, and other spaces, but this leads to fragmented segments as the pixels are clustered only in the color space without considering connectivity. The problem has been attacked by adding connectivity constraints, or using joint color and spatial features (r, g, b, x, y), which prevent fragmented and nonconvex segments. However, it does not take into account the complexity of the shape itself. The Mumford–Shah model has been earlier used to overcome this problem but with slow and complex mathematical optimization algorithms. We integrate Mumford–Shah model directly into KM context and construct a fast and simple implementation of the algorithm. The proposed approach uses standard KM algorithm with distance function derived from Mumford–Shah model so that it optimizes both the content and the shape of the segments jointly. We demonstrate by experiments that the proposed algorithm provides better results than comparative methods when compared using various error evaluation criteria. The algorithm is applied on 100 images in the Weizmann dataset and two remote sensing images.

Deepfakes and manipulation techniques such as adversarial attacks pose a significant threat to society today. It is essential for the detection methods used against them to be more robust than ever. Generation techniques such as neural texture, along with adversarial attacks, result in visually realistic manipulated media and make it difficult for the current state-of-the-art (SoTA) to detect the manipulation. We propose an ensemble-based approach and a genetic algorithm-based approach for fake imagery detection that has been perturbed using fast gradient sign method and basic iterative method techniques. We collectively put such methods under the umbrella term, “EnsembleDet.” We evaluate the proposed approaches using a perturbed version of the benchmark dataset of FaceForensics++, which consists of data generated from manipulation techniques such as Face2Face, neural textures, among others. We present a comparative analysis of quantitative results with SoTA, XceptionNet, and MesoNet, which depict the superiority of the proposed methods over the current SoTA. We also highlight the aspects where the proposed methods have a scope for improvement.

Recent works have shown that multi-label image recognition is still a challenging task in computer vision due to the complicatedness and diversity of multi-label images. However, the existing works ignore the co-occurrence correlation and global contextual information between image space and objects. We present a model to solve these problems. On the one hand, we devise the graph attention mechanism to compute the hidden representations of different categories in multi-label images. It can specify different weights to different neighbor objects and well model the label dependency. On the other hand, we iterate the global contextual information by the second-order covariance pooling to enhance nonlinear modeling capability and use basic residual network to extract features. The proposed model is thoroughly evaluated on PASCAL VOC 2007 and MS-COCO datasets. Compared with classical ML-GCN, the model can better combine the image features and label embedding. Meanwhile, experiments show that it outperforms the state-of-the-art methods such as residual multi-layer perceptron, EfficientNet, and vision transformer.

In recent years, salient object detection (SOD) in RGB-depth (RGB-D) images has attracted considerable research interest. We present a boundary-enhanced attention-aware network (BANet) for RGB-D SOD by combining boundary and saliency detection networks. In particular, depth maps were used to detect the saliency boundaries effectively, and the corresponding RGB images were used to predict the salient objects. Considering that the data contained in depth maps are insufficient, HSV images were employed as complements to enhance the boundary detection performances. Subsequently, an attention module was used to adaptively weigh the features from the RGB branch and boundary network to improve the SOD performance of the proposed BANet. A loss function combining saliency supervision, background supervision, and boundary supervision was designed to optimize the parameters of the BANet. Extensive experiments were conducted to assess the robustness and effectiveness of the proposed BANet. The results suggest that the proposed BANet shows a significant improvement over other representative SOD approaches.

Remote sensing images are frequently affected by the random haze. An efficient method to remove uneven haze is proposed based on dark channel prior (DCP) corrected by saturation map and atmospheric haze model with consideration of multiple atmospheric scattering effects. First, the non-negligible atmospheric multiple scattering phenomenon is simulated by atmosphere point spread function, and the atmospheric multiple scattering haze model is established. Second, an effect method is proposed to calculate the saturation map to correct the DCP, which is used to eliminate the intensity residual of remote sensing images and re-establish the transmission calculation formula. Finally, the image is entered into the integrated restoration framework to remove the haze. Due to the correction of the saturation map, this method can effectively remove the inhomogeneous haze and retain detailed information in the haze-free areas. Compared with other excellent dehaze approaches, qualitative and quantitative experiments are carried out to demonstrate that the proposed method can effectively recover scenes in hazy regions and can retain more detailed information in haze-free regions.

The challenge of video quality assessment (VQA) modeling for user-generated content (UGC) (i.e., UGC-VQA) is how to accurately extract discriminative features and elaborately quantify interfeature interactions by following the behavior patterns of human eye-brain vision perception. To address this issue, we propose the Deeper Spatial–Temporal Scoring Network (DSTS-Net) to give a precise VQA. Concretely, we first deploy the multiscale feature extraction module to characterize content-aware features accounting for nonlinear reverse hierarchy theory in video perception process, which is not fully considered in the reported UGC-VQA models. Hierarchical handcraft and semantic features are simultaneously considered using content adaptive weighting. Second, we develop a feature integration structure, i.e., deeper gated recurrent unit (DGRU), to fully imitate the interfeature interactions in visionary perception, including feedforward and feedback processes. Third, the dual DGRU structure is employed to further account for interframe interactions of hierarchical features, imitating the nonlinearity of perception as much as possible. Finally, improved pooling is achieved in the local adaptive smoothing module accounting for the temporal hysteresis. Holistic validation of the proposed method on four public challenging UGC-VQA datasets presents a comparable performance over the state-of-the-art no-reference VQA methods, especially our method can give an accurate prediction of the low quality videos with weak temporal correlation. To promote reproducible research and public evaluation, an implementation of our method has been made available online: https://github.com/liu0527aa/DSTS-Net.