07. Causal Bounds in Posterior Sampling RL

Vanilla PSRL vs Causal-Truncated PSRL (NOtruncation, Uniform Coverage)

Clanker:

1️⃣ What PSRL normally does

In Posterior Sampling for Reinforcement Learning (PSRL):

1. You maintain a posterior over MDP models

   p(M∣D)p(M \mid \mathcal{D})p(M∣D)

2. At the start of an episode:

   • Sample one model

     M∼p(M∣D)M \sim p(M \mid \mathcal{D})M∼p(M∣D)

   • Compute the optimal policy for that sampled model

   • Execute it

So you are sampling models from the posterior, yes — but you don’t combine multiple samples per episode. You sample one and act optimally in it.

2️⃣ What the figures are showing

The triangles represent:

• A simplex over transition probabilities
  (e.g. for a state-action pair, probabilities over 3 next states)

• Each point in the triangle = one possible transition model

• Contours = posterior density (probability of each model)

So the shaded contours show:

“How likely is each possible transition model under the posterior?”

3️⃣ What the colored regions mean (admissible sets)

The colored regions labeled:

• admissible set 0 (red)

• admissible set 1 (green)

• admissible set 2 (blue)

These represent causal bounds.

They define constraints like:

• “Transition probability to state s₂ cannot exceed X”

• “Certain transitions must respect known structural limits”

• etc.

So:

These regions are sets of models that satisfy some causal constraint.

4️⃣ What “truncate” means here

The slide says:

“truncate → reject if the model does not match causal bounds”

That means:

Instead of sampling from the full posterior

p(M∣D)p(M \mid \mathcal{D})p(M∣D)

you sample from a truncated posterior:

p(M∣D,M∈C)p(M \mid \mathcal{D}, M \in \mathcal{C})p(M∣D,M∈C)

where
C\mathcal{C}C = set of models satisfying causal bounds.

So practically:

1. Sample a model from posterior

2. If it violates causal constraints → reject

3. Resample

4. Use the first valid one

This is rejection sampling.

5️⃣ So are we looking for overlapping zones?

Not exactly.

We are not:

• sampling many models

• intersecting them

• computing overlap

Instead, we are:

• defining allowed regions (colored areas)

• sampling only inside those regions

The overlap between:

• posterior mass (contours)

• admissible region

is what determines:

How much probability mass survives truncation.

If posterior mass lies mostly inside admissible region → little change.

If large mass lies outside → many models get rejected.

6️⃣ Why “data from multiple behavior policies can help”

If you collect data under multiple behavior policies:

• The posterior becomes sharper (less uncertainty)

• It may concentrate inside admissible region

• Fewer rejections happen

So multiple policies → better identification → better overlap with causal constraints.

7️⃣ Intuition of the two triangles

Each triangle corresponds to:

• One action (a₀ on left, a₁ on right)

• Each action has its own posterior over transitions

• Each has its own admissible causal region

The star probably indicates:

• True model

• Or selected sampled model

8️⃣ Big Picture

You are NOT:

• Changing the model repeatedly to find overlap.

You ARE:

• Constraining the posterior using known causal bounds.

• Sampling only from causally plausible models.

• Acting optimistically (like PSRL) but safely.