Gradient-Descent for Randomized Controllers under Partial Observability

arXiv (Cornell University)

Abstract

Randomization is a powerful technique to create robust controllers, in particular in partially observable settings. The degrees of randomization have a significant impact on the system performance, yet they are intricate to get right. The use of synthesis algorithms for parametric Markov chains (pMCs) is a promising direction to support the design process of such controllers. This paper shows how to define and evaluate gradients of pMCs. Furthermore, it investigates varieties of gradient descent techniques from the machine learning community to synthesize the probabilities in a pMC. The resulting method scales to significantly larger pMCs than before and empirically outperforms the state-of-the-art, often by at least one order of magnitude.

Authors 5

1. Linus Heck Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University , Aachen , Germany

2. Jip Spel Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University , Aachen , Germany

3. Sebastian Junges

   Radboud University Nijmegen

   Affiliation as printed

   Radboud University , Nijmegen , the Netherlands

4. Joshua Moerman Aachen

   RWTH Aachen University · Open University of the Netherlands

   Affiliation as printed

   Open University of the Netherlands , Heerlen , the Netherlands

   RWTH Aachen University , Aachen , Germany

5. Joost-Pieter Katoen Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University , Aachen , Germany