On 20 July 2026, a small team backed by Reactor pushed something unusual to GitHub: not a model demo, not a paper, but the entire training pipeline for a frontier-level world model — plus a written account of everything that went wrong on the way there.
The project is called OpenDreamer, and it is an open reproduction of DeepMind's Dreamer 4. Within days it had crossed a hundred stars and a browser demo that lets you toggle, frame by frame, between a real Minecraft stream and a stream being hallucinated by a neural network.
For anyone building agents in 2026, this is worth understanding. Not because you are going to train a Minecraft world model next quarter, but because the architecture underneath it is the clearest public blueprint yet for the thing most agent stacks are missing: a simulator the agent can practise inside.
What a world model actually is
An LLM predicts the next token. A world model predicts the next state of an environment, conditioned on an action.
Give it the current frame and the action "move mouse left, hold left-click," and it returns the frame that would follow. Chain those predictions and you have a simulator that was never programmed — it was learned from video.
That distinction matters more than it sounds. Today's agent frameworks handle planning by asking a language model to reason in text about what might happen. A world model lets the agent roll the future forward and look. It can try an action a thousand times inside its own head, cheaply, before committing once in the real environment.
DeepMind's Dreamer 4 proved the approach at a scale nobody had reached before. It became the first agent to obtain diamonds in Minecraft trained purely on offline data — no practising in the live game. That task demands sequences of more than 20,000 mouse and keyboard actions chosen from raw pixels. It beat OpenAI's VPT offline agent while using roughly 100 times less data.
The catch: the paper described the result, not the recipe. Reproducing it was left as an exercise.
What OpenDreamer released
OpenDreamer is a JAX/Flax NNX implementation covering the full pipeline:
- A causal video tokenizer that compresses raw frames into latents
- Tooling to tokenize Minecraft/VPT-style MP4 datasets into ArrayRecord shards
- An action-conditioned latent dynamics model — the world model proper
- Rollout generation and FVD scoring for quality measurement
The authors are Diego Martì Monso, Francesco Sacco and Edward Hu, with compute and runtime support from Reactor. The repository is next-state/open-dreamer, and a separate inference repo runs trained checkpoints against your own video and action sequences.
The behaviour-cloning and RL agent loop — the part that turns the world model into an actual Dreamer 4 agent — is on the roadmap but not yet shipped. What exists today is the hard part: the simulator.
The architecture, briefly
Tokenizer
A transformer-based masked autoencoder achieving roughly 100x compression. The team deliberately avoided KL and adversarial losses in favour of masking, on the grounds that it "makes the latent space more diffusible" — meaning the dynamics model, which is a diffusion model, has an easier target to hit. It also trains faster and scales more predictably.
They prototyped on CoinRun, a procedurally generated 2D platformer that fits on a single GPU, before moving to Minecraft. On CoinRun they measured compute-optimal scaling of roughly N ∝ C^0.56 for model size and D ∝ C^0.44 for data — close to square-root scaling in both.
Dynamics model
This is where the interesting choices live. The dynamics model is a block-causal transformer with two distinct attention paths:
- a space layer, where information propagates within a single frame
- a causal time layer, where information propagates across frames
Sequences are folded into blocks of the form (previous action, state, policy). Each state carries spatial latent tokens for the visual content, plus noise-level and step-size tokens, plus register tokens that act as a shared scratchpad.
Training uses diffusion forcing with flow matching, combined with shortcut models for step-size control. Shortcut forcing is the trick that made Dreamer 4 practical: it collapses the number of denoising steps needed, which is how a diffusion-based video model reaches real-time interactive inference on a single GPU instead of grinding for seconds per frame.
The loss is x-prediction (predict the clean data) evaluated in v-space, which the team found materially more stable than pure v-prediction.
The engineering notes are the real payload
Most open reproductions ship code and stay quiet about the six weeks of failed runs. OpenDreamer's write-up does the opposite, and it reads like a field guide to training diffusion-based video models. A few items that generalise well beyond this project:
Muon over LaProp. LaProp produced random loss spikes. Muon was stable. The comparison alone cost roughly 400 B200-hours — a reminder that optimizer selection at this scale is an experiment, not a preference.
EMA is not optional. Using online weights at inference "yields subpar results." For diffusion models specifically, the exponential moving average of parameters is the artefact you actually ship.
Precision boundaries decide stability. Their split: parameters in float32, matrix multiplications and attention inputs in BF16, normalisation in float32, and the dynamics flow output head in float32. Getting these boundaries wrong destabilised training — a much more specific lesson than "use mixed precision."
Data loading was the bottleneck, not the GPU. Decoding MP4s with ffmpeg on the fly could not keep a B200 fed. The fix was pre-tokenising everything into .arrayrecord format, loading with Grain, and holding a GPU-side prefetch buffer. Result: 57–58% model FLOPs utilisation on B200, sitting firmly in the compute-bound region past the 292 FLOP/byte roofline crossover.
Simple parallelism won. At 1.6B parameters and about 24 GiB of model state, they evaluated FSDP and chose plain data parallelism anyway — communication overhead outweighed the sharding benefit. Long video sequences were handled with activation checkpointing and a batch of one sequence per GPU instead.
μ-parametrisation was skipped, since model sizes spanned only about one order of magnitude and Muon was stable enough without it.
That last set of decisions is the kind of thing you normally only learn by burning a training budget.
Running it
The workflow is four scripts and a set of YAML configs:
pip install uv
uv sync
source .venv/bin/activate
# 1. Train the causal video tokenizer
uv run scripts/train_tokenizer.py
# 2. Encode full episodes into latent ArrayRecords
uv run scripts/tokenize_minecraft_dataset.py
# 3. Train the action-conditioned dynamics model
uv run scripts/train_dynamics.py
# 4. Generate rollouts and score them
uv run scripts/eval_fvd.pyBetween steps 2 and 3 there is a manual handoff that is easy to miss: tokenisation writes metadata/latent_stats.npz, and the latent mean and standard deviation from that file must be copied into the latent dataset config before dynamics training.
import numpy as np
stats = np.load("/path/to/tokenized_data/metadata/latent_stats.npz")
print("latent_mean:", stats["mean"].tolist())
print("latent_std:", stats["std"].tolist())
print("num_videos:", int(stats["num_videos"]))Requirements are Python 3.11, uv, and a CUDA 12-compatible JAX environment. Raw data arrives as shard-*.array_record files, each record a pickled dict of MP4 bytes, video shape, and actions. Pixel statistics are dataset-specific and need recomputing whenever you swap the source video.
One practical note: the tokenizer can train on much shorter windows than the episodes you eventually encode. Fixed 256-frame records train fine with 16-frame windows, then get tokenised as full 256-frame latent episodes. That decoupling saves a lot of memory in the phase where you are still iterating.
Why this matters outside of Minecraft
It is easy to file world models under "games research" and move on. That would be a mistake, and the reason is the shape of the problem rather than the domain.
World models earn their keep when a task needs persistent state, action-conditioned prediction, and cheap counterfactual evaluation. Minecraft is a convenient testbed for exactly those properties — long horizons, sparse rewards, pixel observations — but the properties themselves show up everywhere: warehouse robotics, autonomous driving, industrial process control, and any agent that must commit to an irreversible action.
The 2026 framing is not "world models replace LLMs." It is a division of labour. Language models are good at abstraction, symbolic interfaces, tool orchestration and explaining themselves to humans. World models are good at dynamics, rollouts and counterfactuals inside a narrower domain. A capable agent stack ends up wanting both — the LLM sets the goal and decomposes it, the world model checks whether the plan survives contact with the environment.
This is the same architectural gap we described in Loops to graphs: the AI agent architecture shift and in our look at reasoning models versus fast models. Agent frameworks have gotten good at orchestration. What they still lack is a cheap way to ask "what happens if I do this?" without actually doing it.
There is also a straightforward economic argument. NVIDIA's Cosmos family was trained on something on the order of 20 million hours of real-world video; that is a scale only a handful of organisations can reach. OpenDreamer's contribution is showing that a 1.6B-parameter world model, trained on a public gameplay dataset, gets you a real-time interactive simulator on a single GPU — and publishing every decision that made it work. See our coverage of NVIDIA Cosmos 3 Edge for the industrial end of the same spectrum.
What to watch next
Three things will determine whether this becomes infrastructure or stays a research artefact.
The agent loop. The repository ships the world model but not the behaviour-cloning and RL training loop on top of it. Until that lands, OpenDreamer is a simulator, not an agent. It is on the roadmap.
Domain transfer. Minecraft video is abundant and labelled with actions almost for free. Most real domains are neither. The open question for anyone with a robotics or industrial use case is how much action-labelled video you actually need — and whether the CoinRun-to-Minecraft scaling story holds when the environment stops being a game.
Licensing. GitHub currently reports the repository license as non-standard. If you plan to build commercially on top of it, read the licence file before you read the code.
The takeaway
OpenDreamer is not going to change your production stack this month. What it changes is what you can inspect. A frontier-level world model — tokenizer, dynamics, training recipe, and a candid list of the things that broke — is now public, in readable JAX, with a browser demo that makes the idea concrete in about thirty seconds.
For teams thinking seriously about agents beyond the prompt-and-tool-call pattern, that is a better starting point than another framework.
At Noqta we build AI agents and automation for businesses across Tunisia, Saudi Arabia and the wider MENA region. If you are working out where agents fit in your operations, get in touch.
Sources: OpenDreamer write-up · next-state/open-dreamer on GitHub · Dreamer 4 project page · Training Agents Inside of Scalable World Models (arXiv:2509.24527)