Felix Grün; Muhammad Saif-ur-Rehman; Tobias Glasmachers; Ioannis Iossifidis
Distributional reinforcement learning has delivered strong results in discrete-action benchmarks, largely driven by different ways of representing value distributions (e.g., via quantiles) and comparing them during training. In this work, we bring three prominent distributional methods—QR-DQN, IQN, and FQF—into the continuous action domain by integrating distributional critics into two widely used actor-critic algorithms: TD3 and SAC.
We evaluate whether performance differences known from discrete domains carry over to continuous control tasks (Ant, HalfCheetah, Hopper, Humanoid, Walker2D). A key outcome is a qualitative invariance in the deterministic continuous-action setting with respect to the number and placement of distributional atoms/quantiles, suggesting that these design choices may be less critical than commonly assumed—at least under the tested conditions.
Reference
In: Giuseppe Nicosia; Varun Ojha; Sven Giesselbach; M. Panos Pardalos; Renato Umeton; Emanuele La Malfa; Gabriele La Malfa (Eds.)
Machine Learning, Optimization, and Data Science, pp. 175–190, Springer Nature Switzerland, Cham, 2026.
ISBN: 978-3-032-21477-5.
Links
BibTeX
@inproceedings{gruen2026invariance,
title={Invariance to Quantile Selection in Distributional Continuous Control},
author={Gr{\"u}n, Felix and Saif-ur-Rehman, Muhammad and Glasmachers, Tobias and Iossifidis, Ioannis},
booktitle={Machine Learning, Optimization, and Data Science},
pages={175--190},
year={2026},
publisher={Springer Nature Switzerland}
}