In order to solve any reinforcement learning problem, the problem should be defined or modeled as a MDP. A Markov property is termed by the following condition: the future is independent of the past, given the present. This means that the system doesn't depend on any past history of data and the future depends only on the present data. The best example to explain this with is rain prediction. Here, we're considering an analogy and not an actual rain estimation model.

There are various methods in which rain estimation work that may or may not need historical data for estimating "rain measure." We're not going to measure anything here but are instead going to predict whether it is going to rain or not. Hence, considering the MDP equation in terms of this analogy, the equation needs the current state to understand the future and...