Toggle Main Menu Toggle Search

Open Access padlockePrints

From Dissipativity Theory to Compositional Construction of Finite Markov Decision Processes

Lookup NU author(s): Dr Sadegh SoudjaniORCiD



This is the authors' accepted manuscript of a conference proceedings (inc. abstract) that has been published in its final definitive form by ACM, 2018.

For re-use rights please refer to the publisher's terms and conditions.


This paper is concerned with a compositional approach for constructing finite Markov decision processes of interconnected discrete-time stochastic control systems. The proposed approach leverages the interconnection topology and a notion of so-called stochastic storage functions describing joint dissipativity-type properties of subsystems and their abstractions. In the first part of the paper, we derive dissipativity-type compositional conditions for quantifying the error between the interconnection of stochastic control subsystems and that of their abstractions. In the second part of the paper, we propose an approach to construct finite Markov decision processes together with their corresponding stochastic storage functions for classes of discrete-time control systems satisfying some incremental passivablity property. Under this property, one can construct finite Markov decision processes by a suitable discretization of the input and state sets. Moreover, we show that for linear stochastic control systems, the aforementioned property can be readily checked by some matrix inequality. We apply our proposed results to the temperature regulation in a circular building by constructing compositionally a finite Markov decision process of a network containing 200 rooms in which the compositionality condition does not require any constraint on the number or gains of the subsystems. We employ the constructed finite Markov decision process as a substitute to synthesize policies regulating the temperature in each room for a bounded time horizon. We also illustrate the effectiveness of our results on an example of fully connected network.

Publication metadata

Author(s): Lavaei A, Soudjani S, Zamani M

Publication type: Conference Proceedings (inc. Abstract)

Publication status: Published

Conference Name: Proceedings of the 21st International Conference on Hybrid Systems: Computation and Control (part of CPS Week)

Year of Conference: 2018

Pages: 21-30

Online publication date: 11/04/2018

Acceptance date: 19/12/2017

Date deposited: 04/11/2019

Publisher: ACM


DOI: 10.1145/3178126.3178135

Library holdings: Search Newcastle University Library for this item

ISBN: 9781450356428