Spontaneous smile intensity estimation by fusing saliency maps and convolutional neural networks

Qinglan Wei; Elif Bozkurt; Louis-Philippe Morency; Bo Sun

doi:10.1117/1.JEI.28.2.023031

11 April 2019 Spontaneous smile intensity estimation by fusing saliency maps and convolutional neural networks

Qinglan Wei, Elif Bozkurt, Louis-Philippe Morency, Bo Sun

Author Affiliations +

Journal of Electronic Imaging, Vol. 28, Issue 2, 023031 (April 2019). https://doi.org/10.1117/1.JEI.28.2.023031

Abstract

Smile intensity estimation plays important roles in applications such as affective disorder prediction, life satisfaction prediction, camera technique improvement, etc. In recent studies, many researchers applied only traditional features, such as local binary pattern and local phase quantization (LPQ) to represent smile intensity. To improve the performance of spontaneous smile intensity estimation, we introduce a feature set that combines the saliency map (SM)-based handcrafted feature and non-low-level convolutional neural network (CNN) features. We took advantage of the opponent-color characteristic of SMs and the multiple convolutional level features, which were assumed to be mutually complementary. Experiments were made on the Binghamton-Pittsburgh 4D (BP4D) database and Denver Intensity of Spontaneous Facial Action (DISFA) database. We set the local binary patterns on three orthogonal planes (LBPTOP) method as a baseline, and the experimental results show that the CNN features can better estimate smile intensity. Finally, through the proposed SM-LBPTOP feature fusion with the median- and high-level CNN features, we obtained the best result (52.08% on BP4D, 70.55% on DISFA), demonstrating our hypothesis is reasonable: the SM-based handcrafted feature is a good supplement to CNNs in spontaneous smile intensity estimation.

Citation Download Citation

Qinglan Wei, Elif Bozkurt, Louis-Philippe Morency, and Bo Sun "Spontaneous smile intensity estimation by fusing saliency maps and convolutional neural networks," Journal of Electronic Imaging 28(2), 023031 (11 April 2019). https://doi.org/10.1117/1.JEI.28.2.023031

Received: 31 October 2018; Accepted: 27 March 2019; Published: 11 April 2019

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available

Members: $24.00

Non-members: $28.00 ADD TO CART

JOURNAL ARTICLE
8 PAGES

DOWNLOAD PAPER SAVE TO MY LIBRARY

GET CITATION

RIGHTS & PERMISSIONS

Get copyright permission Get copyright permission on Copyright Marketplace

KEYWORDS

Databases

Samarium

Lawrencium

Video

Data modeling

Feature extraction

Convolutional neural networks

Show All Keywords

Keywords/Phrases

Search In:

Publication Years