01. Improving policies

Classical methods assume a known model, while deep RL methods even if model-based learn the model through experience. The first is a planning problem while the latter is learning.

Solving an MDP
│
├── 1. Planning with a known model
│   │   You know P(s'|s,a) and R(s,a).
│   │
│   ├── Dynamic Programming
│   │   ├── Value Iteration
│   │   └── Policy Iteration
│   │
│   ├── DAG / finite-horizon planning
│   │
│   └── MCTS
│.      Seen in model based Deep RL (MuZero, etc.)
│
└── 2. Learning from experience, usually without a known model
    │
    ├── Value-based RL
    │   ├── Q-learning
    │   ├── SARSA
    │   └── DQN
    │
    └── Policy Optimization
        ├── REINFORCE
        ├── Actor-Critic
        ├── TRPO
        ├── PPO
        └── SAC / DDPG / TD3, depending on formulation