Mario's Bombs Away

Double DQN Agent on the Original Game & Watch

A Double DQN agent that learned to play Mario's Bombs Away (Nintendo Game & Watch, 1983) on the original game, emulated in MAME. The game was not reimplemented: the agent presses the same 3 buttons a human would (left, right, bomb) through a Lua script running inside MAME, which talks to Python over named pipes.

The network sees the LCD segments of the screen plus two hidden variables read straight from the RAM of the Sharp SM511 chip, found by reverse-engineering the game's memory.

Key Results

Best Life:

44 bombs delivered without dying (~10 min of play)

Greedy Evaluation (ε = 0):

~8–12 bombs per life

Network:

MLP 38 → 128 → 128 → 4 (22,020 parameters)

Training:

~300K decisions, ~1h20 on a MacBook M2

How It Works

MAME + Lua (the environment server):

Reads the 131 LCD segments and key RAM addresses every frame
Repeats each action for 8 frames (~0.13 s of play)
Detects deaths, deliveries and a cutscene bug
On death, instantly reloads a savestate at the start of Game B, in the same frame
Skips frames during the delivery animation, when no input has any effect

Python:

A Gymnasium environment wraps the text protocol (RESET / STEP over FIFOs), with separate scripts for training and greedy evaluation.

The RL Problem

Episode:

One life, with a random 0–29 no-op steps on reset to desync the chip's timers

Observation (38 values in [0, 1]):

36 game segments: torches, Mario's positions, the cigar man, ground fires and the final man
Speed level (RAM 0x0B)
Tick phase (RAM 0x0C), i.e. how long until the game's next tick

Actions:

Nothing, left, right, button (raise or lower the bomb)

Reward Function

+1Bomb delivered

−1Death (end of episode)

−0.01Every decision, including standing still

ShapingPotential-based, with Φ = 0.05 × Mario's row

F(s, s') = γ × Φ(s') − Φ(s)

Because the shaping is potential-based, it does not change the optimal policy (Ng, Harada & Russell, 1999). It only speeds up learning early on.

Training Parameters

Algorithm:

Double DQN

Replay Buffer:

100,000 transitions

Batch Size:

64, one update per decision

Target Sync:

Every 1,000 steps

Optimizer:

Adam, lr 1e-4

Loss:

Smooth L1 (Huber), gradient clipping at 10

Discount Factor (γ):

0.99

Exploration:

ε-greedy 1.0 → 0.05, then 0.01 floor

Finding the Hidden Variables

The screen does not show everything. The game speeds up with each delivery and an internal clock decides when torches and fires move, so from a single frame the network cannot know how much time it has.

Record all 131 segments and the 128 RAM addresses of the SM511 to a CSV every frame while playing
Look for candidates: automatic correlation scripts were inconclusive, so 0x0B was found by inspecting the data directly, as the address that steps down (5 → 4 → 3) right after each delivery
Prove causality: writing 3 to 0x0B every frame cut the tick from ~28 to ~18 frames, so 0x0B controls the speed

0x0BSpeed level (5 slow … 1 fast), fixed at 3 during training

0x0CTick phase, counts down from 0x0B to 0 between ticks

What Made the Difference

Clean death and delivery detection in Lua, with the reset happening on the exact frame of death
Fixing the speed and feeding the tick phase (0x0C) to the network, so it knows when danger arrives
Lowering the exploration floor from 0.05 to 0.01 in phase 2, which doubled the training average
Keeping truncated episodes in the buffer as non-terminal, instead of discarding the best runs