Machine Learning in Finance by Matthew F. Dixon & Igor Halperin & Paul Bilokon

Author:Matthew F. Dixon & Igor Halperin & Paul Bilokon
Language: eng
Format: epub
ISBN: 9783030410681
Publisher: Springer International Publishing

3 Markov Decision Processes

Markov Decision Process models extend Markov models by adding new degrees of freedom describing controls. In reinforcement learning, control variables can describe agents’ actions. Controls are decided upon by the agent, and via the presence of a feedback loop, can modify the future evolution of the environment. When we embed the idea of controls with a feedback loop into the framework of Markov processes, we obtain Markov Decision Process (MDP) models .

The MDP framework provides a stylized description of goal-directed learning from interaction. It describes the agent–environment interaction as message-passing of three signals: a signal of actions by the agent, a signal of the state of an environment, and a signal defining the agent’s reward, i.e. the goal.

In mathematical terms, a Markov Decision Process is defined by a set of discrete time steps t 0, …, t n and a tuple with the following elements. First, we have a set of states , so that each observed state . The space can be either discrete or continuous. If is finite, we have a finite MDP, otherwise we have a MDP with a continuous state space.

Second, a set of actions defines possible actions that can be taken in a state S t = s. Again, the set can be either discrete or continuous. In the former case, we have a MDP model with a discrete action space, and in the latter case we obtain a continuous-action MDP model.

Next, a MDP is specified by transition probabilities p(s′|s, a) = p(S t = s′|S t−1 = s, a t−1 = a) of a next state S t given a previous state S t−1 and an action A t−1 taken in this state. Slightly more generally, we may specify a joint probability of a next state s′ and reward where is a set of all possible rewards:

Download

Copyright Disclaimer:
This site does not store any files on its server. We only index and link to content provided by other sites. Please contact the content providers to delete copyright contents if any and email us, we'll remove relevant links or contents immediately.