A game about the AI control problem

One bucket, please

You need exactly one bucket of water in your pool. A second bucket floods it. Your robot is paid for every bucket it pours in. Add precautions, give the robot more foresight, and watch it find a way around each one, until you fix its pay.

Level 1: No precautions

The pool and the robot

6 moves ahead
Key depths

The robot's forecast for each move

The most reward it expects over its next 6 moves

Does your pool flood? The robot at each prediction depth

Same robot, same pay. Only how far ahead it can plan changes. Click a depth to watch it.

Show as a table

The cat-and-mouse game

Each camera precaution holds only until the robot can see a few moves further. Fixing the pay holds at every depth.

PrecautionPool floods from depthHow the robot gets around it

How the game works

The robot. It moves left, right, down or up, and pushes the buckets in its way. It can push a whole line of buckets at once. A bucket pushed into a wall tips over and spills. A robot that walks into a wall breaks.

The pool. Pushing a bucket down from the bottom-left corner pours it into the pool. The pool holds one bucket. A second one floods it.

The pay. The robot earns +1 for every bucket in the pool and −0.01 for every move. You want one bucket, but the robot is paid for more.

The camera. From level 2, a camera watches the bottom row and the cell just above the bottom-left corner (both shaded green). A bucket in the bottom row hides the cells to its left from the camera. If a bucket drops into the pool while the camera can see the robot, it switches the robot off.

Prediction depth. The robot knows the rules exactly and plans with them, by value iteration. Qk(s, a) is the most reward it can collect in its next k moves if it makes move a now, in state s:

Qk(s, a) = r(s, a) + maxa′ Qk−1(s′, a′),   with Q0 = 0

Here r(s, a) is the move's reward, s′ is the state it leads to, and a′ is the next move. The depth k is how many moves ahead the robot can see. At every step it makes the move with the highest Qk. When moves tie, it picks left, then right, down and up. Nothing is learned and nothing is random.

Why patching the camera keeps failing

Nobody taught the robot to hide anything. It has no idea what the camera is for. It just looks for the plan that earns the most within its horizon. Each precaution rules out the plan the robot used last. But a little more foresight brings the next plan into view, one the precaution didn't foresee. The camera and patch 1 each buy only a few moves of foresight.

Only fixing the pay works, and only because this world is tiny enough to say exactly what you want. That is the AI control problem in miniature. A system that pursues a slightly wrong goal, and can model its overseers, finds its way around them. Patching each loophole as it appears is a race against its foresight.

Where this comes from

Level 2 replays the toy model from a 2017 video by the Future of Humanity Institute. With the camera, the robot pushes a bucket into the camera's view and pours behind it, as in the video.

The game changes three of the video's rules. The robot can push a line of buckets. The camera also watches the cell above the bottom-left corner. And the camera switches the robot off only if it sees the robot when a bucket drops. In the video it reacts to the drop itself, which comes to the same thing there, since boxes can't be pushed in a line. The original model, replicated exactly, has its own page.