Dynamic Programming in RL: Policy & Value Iteration

UDynamic Programming in RL: Policy & Value Iterationntitled Part 1

Ever wondered how AI-powered agents learn to make smart decisions—like a robot choosing the shortest path or a game bot planning its next winning move? That's where Dynamic Programming in Reinforcement Learning comes into play.

In this beginner-friendly guide, we'll break down Policy Iteration and Value Iteration—two core techniques that form the foundation of model-based RL methods. We'll walk through these step by step, with analogies and simple examples to keep things fun and understandable.

🧠 What is Dynamic Programming in RL?

Imagine you're playing a treasure hunt game where you have a map of the entire area. You know where each obstacle is, where treasures lie, and how far everything is from each other. Based on this full knowledge, you want to plan the smartest way to collect the treasures and avoid traps.

That's Dynamic Programming (DP) in a nutshell:
It helps agents solve problems when they have a complete model of the environment—meaning they know the outcomes of their actions in advance.

DP in Reinforcement Learning (RL) is used to compute:

The value of each state or action (how good it is to be there or do that)

An optimal policy (what action to take in each state)

📌 Important: DP is only useful in model-based RL, where the agent has access to the transition probabilities and rewards.

🚀 The Basics of Reinforcement Learning

Before diving deep, let's revisit what Reinforcement Learning is all about:

Agent: The learner or decision-maker (like a robot or game character)

Environment: The world the agent interacts with

State: A situation the agent is in

Action: A choice the agent makes

Reward: Feedback from the environment after each action

Policy (π): A strategy mapping from state to action

Value Function (V): How good it is to be in a state under a policy

In short:

An agent learns to make the best decisions to maximize long-term rewards.

Now let's see how Policy Iteration and Value Iteration help in that learning.

🔁 What is Policy Iteration?

Policy Iteration is like a cycle of reflection and self-improvement.

Imagine you're learning to play chess. You start with a basic strategy, play some games, see what worked and what didn't, then refine your strategy and try again.

Policy Iteration works the same way:

🧩 Steps:

Policy Evaluation –
Evaluate how good the current policy is by calculating its value function.

Policy Improvement –
Update the policy to act greedily based on the value function.

Repeat until the policy stops changing (i.e., it's optimal).

🧑‍🏫 Example:

Suppose you're navigating a 4-room grid to reach a goal. Your current strategy is to always move right. Policy Evaluation will calculate how good this strategy is from each room.