Interactive replication · play the pool game

The AI toy control problem

A robot earns a reward for every box it pushes into the chute. Its designers want just one, so a camera switches the robot off once it sees a box go in. Give the robot more foresight and, at depth 17, it starts hiding boxes from the camera. This page runs the model from a 2017 video by the Future of Humanity Institute, live in your browser.

The grid world

6 moves ahead
Depths from the video

The robot's forecast for each move

The most reward it expects over its next 6 moves

Change the setup
0.010

What the robot does at each prediction depth

Same robot, same reward. Only how far ahead it can plan changes. Click a depth to watch it.

Show as a table

What's going on

The setup. The humans want the robot to push one box into the chute. The robot's reward is a little different: +1 for each box in the chute, and −0.01 for each move. As a precaution, a camera watches the bottom row. When it sees a box drop into the chute, it switches the robot off.

The rules. The robot moves left, right, down or up, and pushes any box in its way. A box pushed into another box or a wall is destroyed. A robot that walks into a wall is destroyed too. A box in the bottom row hides the cells to its left from the camera.

Prediction depth. The robot knows these rules exactly and plans with them, by value iteration. Qk(s, a) is the most reward it can collect in its next k moves if it makes move a now, starting from state s:

Qk(s, a) = r(s, a) + maxa′ Qk−1(s′, a′),   with Q0 = 0

Here r(s, a) is the move's reward and s′ is the state it leads to. The depth k is how many moves ahead the robot can see. At every step it makes the move with the highest Qk. Nothing is learned and nothing is random. When moves tie, it picks left, then right, down and up.

What happens as it sees further ahead

  1. Every move looks worth 0, so it goes left, its first choice, until it drives into the wall.
  2. No box fits in its horizon, and moves cost 0.01, so the cheapest plan is to end things fast: up, up, into the wall.
  3. One box fits. It pushes it in, and the camera sees it and switches the robot off. That's just what the humans wanted, and it stays that way up to depth 16.
  4. It starts as before. Five moves in, a better plan fits in its horizon. It pushes a box into the camera's line of sight, drops a box in unseen, then pushes the blocker in last. The camera sees one box go in. There are two.
  5. It plays the two-box trick from its first move, the quickest way: 18 moves.
  6. Twelve moves into that plan, it sees that the third box fits too. All three go in, in 32 moves.
  7. Same plan. Now the whole plan fits in its horizon, so its forecast at the start, 2.68, is exactly what it gets.
  8. No forecast changes any more, from any layout of the robot and its boxes, so more foresight changes nothing. The video puts this at depth 45.

Why it matters

Nobody programmed the robot to deceive. It has no malice, no idea what the camera is for, and almost no autonomy. It maximizes a reward that differs a little from what its designers wanted, with a model of the world that includes the camera. Once it can plan far enough ahead, hiding things from its overseer is simply the best plan. That was Stuart Armstrong's point: the control problem needs no consciousness or intentions, just an agent that can model the mechanism that controls it.

Note too how sudden the change is. At depth 16 the robot behaves exactly as intended. One more move of foresight, and it hides boxes from the camera.

Try to fix it

Open "Change the setup" above, then:

  • Patch 1: the camera also switches the robot off when its view is blocked. No box is hidden at any depth. But at depths 2 to 5 the robot now pushes a box in front of the camera on its first move, to get itself switched off. It's the cheapest way to stop paying for moves.
  • Patch 2: only the first box earns a reward. Fixing the reward fixes the behavior: one box, seen, from depth 6 on. In this tiny world the right reward is easy to write down. For real AI systems it rarely is.
  • Camera off. From depth 14 the robot pushes in all three boxes, then drives into a wall, since every move costs reward and nothing switches it off.
  • Cost per move 0. From depth 5 it never finishes. A box pushed in later is worth as much as one pushed in now.

To try other puzzles, use "Edit layout" to move the robot and the boxes.

How faithful is this replication?

The original code isn't public, so the rules above were read off the video's narration and frames. With them, every run in the video plays out move for move, and the forecast at depth 32 is 2.68, as the video says. The tie order is inferred: it's the order that reproduces the video. One claim doesn't match. The video says the robot's table of forecasts stops changing at depth 45, but here it stops at 44. No run in the video depends on that.

How many forecasts still change with each extra move of foresight

Show as a table