Kunlun Tech launches AI video game model for single RTX 5090

Kunlun Tech released Matrix-Game 4.0, which it calls the first video-model game playable on a single consumer graphics card, and open-sourced the code.

Kunlun Tech has released Matrix-Game 4.0, which it calls the first video-model-driven game to run on a single consumer graphics card: 480p at 60 FPS on an Nvidia RTX 5090, with under 150 milliseconds between a keypress and the screen's response.

The Beijing company is pitching the release as a shift in how interactive world models are built. Instead of asking a model to remember an entire world, Matrix-Game 4.0 lets a game engine hold the world state and rules in code while a video model renders the visuals in real time. The project is open source.

The model runs at a steady 83 FPS for single-step denoising, peaks at 18.5 GiB of video memory and is fast enough for local deployment and real interaction, Kunlun Tech said. Open-source links:

Why earlier world models were hard to play

Kunlun Tech argues the weakness of earlier interactive world models was never image quality. Three problems sit lower down.

First, without an external engine tracking explicit state, objects and scenes persist only in the model's internal representation and memory. A developer cannot reliably read, change or verify the state of a single object, or guarantee that events off-screen keep following the rules.

Second, autoregressive generation builds on its own past output, so small errors accumulate. Long sessions, repeated visits to the same place or objects moving in and out of view can cause the scene, object identity and interaction results to drift. Bigger context windows and memory mechanisms help but do not deliver game-grade consistency.

Third, plausible-looking interactions are not the same as reliable game rules. Health, inventory, quest progress and win conditions must be defined, edited and tested by developers and stay consistent across actions and long runs.

All three point to one gap: games need a world that can be read, modified and verified, not one stored in a model's memory.

Engine runs the world, AI paints it

Matrix-Game 4.0 wraps the video model in a "Harness" that maintains world state. An external game manager supplies spatial layout, tracks state and updates semantics; the video model takes those conditions and generates short, realistic future frames.

The game engine is the referee. It records what happens in the world and decides the outcome of player actions. It does not need to produce detailed visuals; simplified geometry and scene structure describe roads, buildings, characters and interactive objects. Health, backpack, gravity, collision and quest progress are maintained and enforced in code rather than inferred from generated frames. The engine keeps updating the world and outputs structural frames that constrain layout, object motion and camera angle. Objects out of view remain tracked under the rules.

The Coding Agent is the builder. It turns natural-language requests into a working game: adjusting terrain, placing objects, changing NPC behavior or designing new mechanics. "Let monsters swarm from all sides", for example, requires spawn points, movement, attack behavior and win conditions, not just more monsters on screen. The agent also sets scene semantics and visual style — the same simple building can be a seaside shack or a shop in a future city — and these descriptions join the structural frames as text conditions.

The video model is the painter. It turns structural frames into detailed visuals, generating materials, lighting, atmosphere and surface detail while preserving layout and motion constraints. That cuts reliance on hand-built 3D assets. It generates causally and autoregressively from current conditions and past frames, with few-step distillation and inference optimization aimed at real-time rerendering.

The runtime loop: player input → engine updates world state → structural frame and control conditions → video model renders → frame shown to the player.

The split matters. Code carries the rules, the engine holds the state, the video model makes the picture. With a Coding Agent, Kunlun Tech says, more people can turn ideas into playable, editable and iterable games at lower cost.

Data, training and runtime

The team built a structural-to-detailed frame dataset aligned across domains in Unreal Engine 5, covering four settings: the CitySample mega-city, open desert, a complex castle and high-dynamic dogfights.

A lightweight geometry abstraction pipeline keeps camera paths and timestamps in strict spatial and temporal sync. It reduces complex scenes to low-poly representations that preserve spatial topology, rigid-body poses and dynamic trajectories, then aligns them pixel by pixel with high-poly, film-grade footage. In parallel it captures object instance IDs, visibility states and causal interaction events to build a semantically rich stream of dynamic metadata.

Dynamic captions. In a playable game the picture changes constantly: turning the camera reveals different objects, and moving or triggering events changes their state. Fixed text descriptions cannot keep up. Matrix-Game 4.0 links object IDs to semantic descriptions, pulls the currently visible objects and their live states from the engine and fills them into a prompt template at runtime. Structural frames supply position, layout and motion; captions supply what is on screen and in what state.

Controllable generation and spatial memory. The video model is a Wan2.2 5B adapted to video-to-video mode. Structural frames from the engine are concatenated channel-wise with the video to be generated (148 input channels), telling the model what to draw and where. A dynamically composed scene-event prompt, a spatial memory reference image and the first frame handle detail description, scene consistency and frame continuity.

Inference speed. To hit 60 FPS on a consumer card, the team pre-records the DiT backbone with CUDA Graph and replays it in one pass at runtime. It wrote fused GPU kernels for RoPE, normalization and FP8 quantization, with Q, K and V in a layer sharing one input quantization. Text-condition KV is recomputed only when the caption updates; video history KV uses a ring buffer; self-attention is split by KV window and computed in parallel; encoding and decoding use the lightweight TAEHV. The text encoder, DiT and VAE all stay resident on one RTX 5090.

Rewriting a game with a sentence

The most striking part may be how easily the game can be changed. With the Coding Agent, players can customize an existing game in natural language — terrain, scene objects, NPC behavior and interaction logic, or entirely new mechanics. Ideas no longer wait on a 3D asset pipeline; they go straight to a playable, iterable stage.

Cheaper lightweight game engines, as AI coding assistants improve, are what make this practical. Kunlun Tech is encouraging creators to adapt and extend the game code it has released.

Video generation models have made progress in playing games over the past year, but running on an ordinary player's PC is a different matter. In the comparison below, end-to-end latency is about 10.2 seconds for Code World Model, about 545 milliseconds for StreamRender-H3 and about 878 milliseconds for AgentGarten. Matrix-Game 4.0 comes in at 112 milliseconds — roughly 1/90 of the first and 1/5 and 1/8 of the other two, the company says, putting it in the real-time playable range for the first time.

On hardware, the other approaches need a 1× H200 141GB, 8× GB200 or 1× H100 data-center card. Matrix-Game 4.0 runs on a single RTX 5090 with 32GB of memory.

On openness, Code World Model and StreamRender-H3 are already open and AgentGarten is partly open. Matrix-Game 4.0 releases code, weights and structural models together.

What comes next

The team plans to add a refiner for higher resolution, explore multiplayer with consistent views across players, offer finer-grained visual control and extend support to more game genres and scenes.

Going from controllable video to a runnable world may matter more than it looks: the barrier to creation is falling, and the edge of what can be imagined is moving out.

Canonical article