Evaluating Deep Reinforcement Learning Techniques for Sequential Decision-Making in Arcade Games

Abstract

Reinforcement Learning (RL) is a powerful approach for training agents to make decisions in complex and dynamic environments, particularly those with high-dimensional visual input. However, understanding of architectural changes affect learning performance, stability and efficiency is an important research challenge. This paper compares the five value-based reinforcement learning algorithms, namely, SARSA, Deep Q-Network (DQN), Double Deep Q-Network (DDQN), Dueling DQN, and Dueling DQN with Prioritized Experience Replay (PER) on the MsPacman-v5 environment of Arcade Learning Environment (ALE). To compare fairly, all deep learning models were trained using a consistent pixel-based preprocessing pipeline of frame skipping, grayscale, resizing, and frame stacking while SARSA was represented in a discretized form based on the same environment to ensure fair comparison.

The algorithms were evaluated using metrics such as maximum score, average score, convergence behavior, and training efficiency. The results show that architectural improvements play a significant role in enhancing learning performance. Dueling DQN achieved the best performance, reaching a maximum score of 4650 and an average score of 1225.7, demonstrating its ability to learn more effective state representations. DDQN also showed strong performance, achieving a maximum score of 4000 and improving average performance by 41.7% compared to standard DQN, confirming its effectiveness in reducing overestimation bias. In contrast, SARSA demonstrated stable but limited performance due to its simplified state representation. Overall, the findings highlight the importance of architectural design in improving reinforcement learning performance in complex decision-making environments.