Unveiling insights in source code defect prediction using ChatGPT: moving beyond predictive metrics

Cong Pan; Ai Gu; Yan Gao

doi:10.1117/12.3012284

21 December 2023 Unveiling insights in source code defect prediction using ChatGPT: moving beyond predictive metrics

Cong Pan, Ai Gu, Yan Gao

Proceedings Volume 12970, Fourth International Conference on Signal Processing and Computer Science (SPCS 2023); 129702R (2023) https://doi.org/10.1117/12.3012284
Event: Fourth International Conference on Signal Processing and Computer Science (SPCS 2023), 2023, Guilin, China

Abstract

In the realm of software engineering, defect prediction stands as a vital and valuable research domain. With the emergence of powerful models like ChatGPT, the potential for enhancing software defect prediction becomes significant. Leveraging the GHPR defect prediction dataset, this study delves into the application of ChatGPT for source code-level defect prediction. Different prompt paradigms, including general-purpose prompts, prompts that emphasize minority class, prompts that include few-shot examples, prompts that include target explanations, and prompts using chain-ofthoughts (CoT), are investigated for their impact on predictive outcomes. Furthermore, a comparative analysis is conducted between specific defects predicted by ChatGPT and defects rectified by developers in the GHPR dataset. While direct application of ChatGPT in source code-level defect prediction shows results akin to random guessing, the study reveals that employing CoT prompts unveils a propensity for ChatGPT to identify a higher number of issues, proposing questions with heightened rationale. The findings emphasize the necessity to move beyond predictive metrics when utilizing ChatGPT for source code-based defect prediction, advocating for a more profound analytical approach.

(2023) Published by SPIE. Downloading of the abstract is permitted for personal use only.

Citation Download Citation

Cong Pan, Ai Gu, and Yan Gao "Unveiling insights in source code defect prediction using ChatGPT: moving beyond predictive metrics", Proc. SPIE 12970, Fourth International Conference on Signal Processing and Computer Science (SPCS 2023), 129702R (21 December 2023); https://doi.org/10.1117/12.3012284

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available