Tackling sparse cost in safe reinforcement learning for obstacle avoidance

Quanzhong Li; Xia Zeng; Hengjun Zhao

doi:10.1117/12.2640072

23 May 2022 Tackling sparse cost in safe reinforcement learning for obstacle avoidance

Quanzhong Li, Xia Zeng, Hengjun Zhao

Proceedings Volume 12254, International Conference on Electronic Information Technology (EIT 2022); 122543H (2022) https://doi.org/10.1117/12.2640072
Event: International Conference on Electronic Information Technology (EIT 2022), 2022, Chengdu, China

Abstract

Safe Deep Reinforcement Learning (SDRL) has been studied in many domains. In previous work, based on Augmented Lagrangian Method, we proposed DDPG-ALM algorithm in an industrial oil pump case study to learn a safe controller. In this paper, we apply our algorithm to a more general platform, Safety Gym, to test the performance of our algorithm. To solve the issue of sparse cost in Safety Gym, we propose the notion of State Shaping which aggregates safe states to pay more attention to the unsafe states. Furthermore, to pay more attention to safety-related experience, we combine the advantages of Prioritized Experience Replay and Double Replay Buffer and propose a method called Safe Prioritized Experience Replay (SPER). Our experiments show that, State Shaping and SPER can deal with the sparse cost well and provide safer policies. Ablation studies show that State Shaping reduces average episode cost of all evaluated trajectories from 2.67 to 0.45 and SPER reduces from 0.45 to 0.20 while PER only reduces to 0.30. Moreover, our DDPG-ALM based algorithm not only automatically finds a balance between reward and cost, but also converges faster than other existing benchmark algorithms.

Citation Download Citation

Quanzhong Li, Xia Zeng, and Hengjun Zhao "Tackling sparse cost in safe reinforcement learning for obstacle avoidance", Proc. SPIE 12254, International Conference on Electronic Information Technology (EIT 2022), 122543H (23 May 2022); https://doi.org/10.1117/12.2640072

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available