Transductive few-shot image recognition with ranking-based multi-modal knowledge transfer

Guangzhi Ye

doi:10.1117/12.2681542

1 June 2023 Transductive few-shot image recognition with ranking-based multi-modal knowledge transfer

Guangzhi Ye

Proceedings Volume 12718, International Conference on Cyber Security, Artificial Intelligence, and Digital Economy (CSAIDE 2023); 127181V (2023) https://doi.org/10.1117/12.2681542
Event: International Conference on Cyber Security, Artificial Intelligence, and Digital Economy (CSAIDE 2023), 2023, Nanjing, China

Abstract

Few-shot image recognition is a challenging task that aims to train a model with limited labeled images for image classification. To overcome the issue of insufficient image information in few-shot learning (FSL), this paper proposes to incorporate textual semantic information for guiding knowledge transfer to achieve more robust models. However, existing multi-modal guidance methods often necessitate aligning feature spaces from different modalities, which not only adds extra model parameters and training costs but also increases overfitting risks in FSL. To address these issues, this paper proposes a multi-modal method based on ranking instead of training, called Ranking-Based Multi-Modal Knowledge Transfer (RMMKT). Specifically, our RMMKT method (1) measures inter-class correlations across different modalities and performs multi-modal fusion based on ranking to avoid the need for training multi-modal feature adapters, (2) utilizes the multi-modal metrics to transfer base classes information to novel classes, which improves the stability of model, and (3) introduces the maximum a posteriori estimation for prototype generation and image classification without training. The proposed method was evaluated on commonly used FSL datasets and achieved state-of-the-art performance. RMMKT provides a training-free transductive few-shot learning strategy that achieves multimodal fusion and knowledge transfer without introducing extra model parameters for novel tasks, thus enhancing model stability and accuracy.

Citation Download Citation

Guangzhi Ye "Transductive few-shot image recognition with ranking-based multi-modal knowledge transfer", Proc. SPIE 12718, International Conference on Cyber Security, Artificial Intelligence, and Digital Economy (CSAIDE 2023), 127181V (1 June 2023); https://doi.org/10.1117/12.2681542

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available

Members: $17.00

Non-members: $21.00 ADD TO CART

PROCEEDINGS
6 PAGES

DOWNLOAD PAPER SAVE TO MY LIBRARY

GET CITATION

RIGHTS & PERMISSIONS

Get copyright permission Get copyright permission on Copyright Marketplace

KEYWORDS

Semantics

Visualization

Prototyping

Information visualization

Data modeling

Visual process modeling

Feature extraction

Show All Keywords

Keywords/Phrases

Search In:

Publication Years