Solving Deep Reinforcement Learning Tasks with Evolution Strategies and Linear Policy Networks
ACM Transactions on Evolutionary Learning and Optimization
Abstract
Although deep reinforcement learning methods can learn effective policies for challenging problems, the underlying algorithms are complex, and training times are often long. This study investigates how several state-of-the-art versions of Evolution Strategies perform compared to gradient-based deep reinforcement learning methods. We use Evolution Strategies to optimise the weights of a neural network via neuroevolution, performing direct policy search. We benchmark deep and linear policy networks for gradient-based methods, including Proximal Policy optimisation, Soft Actor-Critic, and Deep Q-Learning. These are evaluated against linear Evolution Strategies and Augmented Random Search, and additionally against scalable ES for deeper policies, such as LM-MA-ES. Our results reveal that Evolution Strategies can find effective linear policies for many reinforcement learning benchmark tasks, unlike deep reinforcement learning methods that can only find successful policies using much larger networks, suggesting that current benchmarks are easier to solve than previously assumed. Linear Evolution Strategies also outperform scalable ES variants trained on larger neural networks. Interestingly, Evolution Strategies, which do not use any gradient information, also achieve results comparable to gradient-based deep reinforcement learning algorithms for higher-complexity tasks. Furthermore, by directly accessing the memory state of the game, Evolution Strategies can find successful policies in Atari that outperform the policies found by Deep Q-Learning. Evolution Strategies also outperform Augmented Random Search in most benchmarks, demonstrating superior sample efficiency and robustness in training linear policy networks.
Authors 5
-
Affiliation as printed
Leiden Institute of Advanced Computer Science (LIACS), the Netherlands
-
Affiliation as printed
Leiden Institute of Advanced Computer Science (LIACS), the Netherlands
-
Affiliation as printed
Leiden Institute of Advanced Computer Science (LIACS), the Netherlands
-
Affiliation as printed
Leiden Institute of Advanced Computer Science (LIACS), the Netherlands
-
Affiliation as printed
Leiden Institute of Advanced Computer Science (LIACS), the Netherlands
Cited by 0 stored of 0
No patents citing this paper on Lens.org (checked 2026-10-11).