Model Demo: Doomguy, a Pixel ViT That Plays DOOM

NOTE: The Jupyter Notebook below is included in the Chimera SDK and can be run interactively by running the following CLI command:

$ quadric sdk notebook

From the Jupyter Notebook window in your browser, select the notebook named /quadric/sdk-cli/examples/games/doomguy/doomguy.ipynb.

Doomguy Plays by Himself

The Chimera GPNPU renders every frame with the Doom raycaster kernel from ../doom. A small vision transformer, compiled for the same GPNPU, reads that frame together with 12 typed questions ("Shoot?", "Dodge?", "Where is he?", "Is that fireball going to hit me?") and answers all of them in one forward pass. Those answers drive the game. Before a deliberate move he says the question behind it, the network's answer, and the move ("Where is he? On my left. Strafing left.").

What runs where (QC-P: 16x16 PEs, 8 MACs per PE; all GPNPU numbers are exact ISS cycles):

The real-time target is HAPS-200 (QC-P at 50 MHz): section 12 turns the measured cycles into its budget.

1. Setup

All paths live in doomguy/config.py and can be overridden with environment variables. DOOM_TARGET selects the simulated array of the renderer; this notebook uses QC-P throughout.

The notebook presents what run_pipeline.sh builds (hours on a 32-core host). In a work directory without its models, the setup cell downloads the published ones instead: the game-state expert, the int8 ViT with its held-out check frames, and the three voice kernels with their calibration inputs. From those the notebook builds the renderer, compiles every GPNPU graph and runs the renderer, ViT and voice-kernel gates on the ISS. The voice bank and the closed-loop episodes (sections 9-12) need the trained models and hours of ISS time, so they run after run_pipeline.sh; until then their cells name the step that produces their inputs. With the trained models, fetch_assets downloads Pocket TTS, the CC0 reference voice and the MiniLM question embeddings, and decodes the status bar art from the WAD.

import json
import os
import pickle
import subprocess

import numpy as np
from IPython.display import Video, display

os.environ.setdefault("DOOM_TARGET", "QC-P")
from doomguy import config
from doomguy import notebook as NB
from doomguy.notebook import output, sh

print("work directory :", config.WORK)
print("Doom renderer  :", config.DOOM_DIR, "| target", os.environ["DOOM_TARGET"])
READY = (config.MODELS / "small.pt").exists()  # run_pipeline.sh has trained the pixel ViT
if READY:
    sh("python3 -m doomguy.pipeline.fetch_assets", tail=0)
else:
    print(f"no trained models in {config.MODELS}: downloading the published ones")
    NB.download_published()

work directory : /quadric/sdk-cli/examples/games/doomguy/work
Doom renderer  : /quadric/sdk-cli/examples/games/doom | target QC-P
no trained models in /quadric/sdk-cli/examples/games/doomguy/work/models: downloading the published ones
models/expert.npz                                                          0.10 MB  sha256 ok
models/small.log                                                           0.00 MB  sha256 ok
models/gpnpu/small/export.json                                             0.00 MB  sha256 ok
models/gpnpu/small/small_vit_OpSet16_optimized_sym_int8_q_shaped.onnx      2.11 MB  sha256 ok
models/gpnpu/small/small_vit_OpSet16_optimized_sym_int8_q_shaped.tranges   0.02 MB  sha256 ok
models/gpnpu/small/check_frames.npz                                        0.68 MB  sha256 ok
models/gpnpu/kokoro8k_am_liam_r16_t64/export.json                          0.00 MB  sha256 ok
models/gpnpu/kokoro8k_am_liam_r16_t64/a.onnx                               7.87 MB  sha256 ok
models/gpnpu/kokoro8k_am_liam_r16_t64/b.onnx                               5.77 MB  sha256 ok
models/gpnpu/kokoro8k_am_liam_r16_t64/c.onnx                               1.83 MB  sha256 ok
models/gpnpu/kokoro8k_am_liam_r16_t64/calib.npz                           43.64 MB  sha256 ok

2. The Arena and the Game-State Expert

The arena spawns 18 waves of imps and demons around the player (at most 4 alive) and gives health and ammo between waves. The expert sees the game state. It makes committed 1-2 s maneuvers sampled from its move distribution, and dodges when its "dodge now" output crosses its threshold. All three of its heads (fire, move, dodge) are distilled from LAYA's answers on states a scripted scout visits. The expert is built by run_pipeline.sh steps 1-3.

if (config.MODELS / "expert.npz").exists():
    e = json.loads(output("python3 -m doomguy.expert_play one 11").splitlines()[-1])
    print(
        f"expert, seed 11: survived {e['survived']:.0f} s, {e['waves']} waves, {e['kills']} kills, "
        f"{e['fireball_hits']} fireball hits"
    )
    print(
        f"{e['maneuvers']} maneuvers: hold {e['hold']}, dodge left {e['dodge_left']}, "
        f"dodge right {e['dodge_right']}, retreat {e['retreat']}, advance {e['advance']}"
    )
    print(f"{e['dodges']} dodges, {e['false_dodges']} false alarms")
else:
    print("Build the expert first: ./run_pipeline.sh (steps 1-3)")

expert, seed 11: survived 150 s, 11 waves, 28 kills, 4 fireball hits
130 maneuvers: hold 39, dodge left 37, dodge right 29, retreat 11, advance 14
33 dodges, 0 false alarms

3. Frames Rendered by the GPNPU

The renderer is one fused CCL kernel. Each PE renders one pixel of a tile the size of the PE array, so on QC-P a frame is 20x13 tiles of 16x16 pixels (the last tile row is padded past row 200). The cell builds the kernel for QC-P if needed, then renders four moments of an expert episode with enemies in sight on the ISS and reports each frame's cycles. The same scenes are rendered on QC-N as the reference. Gate: every QC-P frame matches its QC-N frame on at least 99.9% of its pixels (a 16x16 tile holds at most 8 sprites against 4 per 8x8 tile, so a very crowded tile can differ).

target = os.environ["DOOM_TARGET"]
if (config.MODELS / "expert.npz").exists():
    NB.build_renderer(target)
    NB.build_renderer("QC-N")  # the reference array
    sh("python3 -m doomguy.expert_play record 11", tail=0)
    frames = pickle.load(open(config.WORK / "expert_seed11.pkl", "rb"))["frames"]
    picked = NB.pick_scenes(frames)
    out, ref = config.WORK / "nb_frames", config.WORK / "nb_frames_qcn"
    pickle.dump({"frames": [frames[i] for i in picked]}, open(f"{out}.pkl", "wb"))
    cyc = NB.render_scenes(f"{out}.pkl", out, target)
    NB.render_scenes(f"{out}.pkl", ref, "QC-N")
    for k, i in enumerate(picked):
        c = cyc[str(k)]
        upload = " (also uploads the textures)" if c["static_load"] else ""
        print(f"t = {i / 15:5.1f} s: {c['cycles']:,} cycles{upload}")
    diff = NB.pixels_differing(out, ref, len(picked))
    print(f"pixels differing from the QC-N reference, per frame: {diff}")
    assert max(diff) <= 64, "a QC-P frame differs from QC-N on more than 0.1% of its 64,000 pixels"
    print("PASS: every QC-P frame matches the QC-N reference on at least 99.9% of its pixels")
    display(NB.frame_grid(out, len(picked)))
else:
    print("Build the expert first: ./run_pipeline.sh (steps 1-3)")

cli/examples/games/doom/renderer_QC-P_1d7_8MB_4kB_128GBps_128GBps_8_OFF_x1_x1/output
[SDK-CLI] : Executing on QC-P simulator
seed: 495626181
Fused renderer: camera=(8,5) angle=0.0, 2 enemies
Saved renderer_frame.ppm
[SDK-CLI] : TotalCycles: 270,327
[SDK-CLI] : Executions/second: 6,289

compute      : ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 99.617K
data_array   : ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 40.673K
mac          :  0
data_ocm     : ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 127.84K
data_external: ▏ 2.196K

[SDK-CLI] : Execution completed.
make: Leaving directory '/quadric/sdk-cli/examples/games/doom'
xamples/games/doom/renderer_QC-N_1d7_4MB_4kB_128GBps_128GBps_8_OFF_x1_x1/output
[SDK-CLI] : Executing on QC-N simulator
seed: 1375162323
Fused renderer: camera=(8,5) angle=0.0, 2 enemies
Saved renderer_frame.ppm
[SDK-CLI] : TotalCycles: 627,812
[SDK-CLI] : Executions/second: 2,708

compute      : ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 294.952K
data_array   : ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 89.959K
mac          :  0
data_ocm     : ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 238.785K
data_external: ▏ 4.115K

[SDK-CLI] : Execution completed.
make: Leaving directory '/quadric/sdk-cli/examples/games/doom'
t =   2.9 s: 293,276 cycles (also uploads the textures)
t =  14.0 s: 287,981 cycles
t =  57.7 s: 269,388 cycles
t = 109.3 s: 320,004 cycles
pixels differing from the QC-N reference, per frame: [0, 0, 0, 0]
PASS: every QC-P frame matches the QC-N reference on at least 99.9% of its pixels

4. The Questions He Asks Himself

Each question is text, embedded once by MiniLM, and enters the ViT as one token. Each option is embedded too; a question token answers by scoring its options. One paraphrase of every question is held out of training.

from doomguy import questions as QB

for name in QB.NAMES:
    kind, phrasings, options, spoken = QB.Q[name]
    print(f"{spoken:36s} [{kind:10s}] options: {', '.join(options)}")

Shoot?                               [decision  ] options: yes, no
What now?                            [decision  ] options: hold, aim_left, aim_right, retreat, dodge_left, dodge_right, grab, scan, advance
Dodge?                               [decision  ] options: sidestep, stay
Which way?                           [decision  ] options: dodge_left, dodge_right, advance, retreat
What's going on?                     [decision  ] options: none, too_close, close, far, fireball_left, fireball_right, fireball_ahead, two, three_plus, imp_left, imp_right, imp_ahead, demon_left, demon_right, demon_ahead, lined_up, imp, demon, health_low, closing_in
Is that fireball going to hit me?    [perception] options: yes, no
Anyone there?                        [perception] options: yes, no
Is he close?                         [perception] options: close, far, none
How many of them?                    [perception] options: none, one, two, three_plus
Where is he?                         [perception] options: left, center, right, none
Where's that fireball coming from?   [perception] options: left, right, ahead, none
Am I hurt?                           [perception] options: yes, no

5. The Pixel ViT

Two stacked 160x100 frames, cut into 10x10 patches (160 tokens), one state token, one perception token and 12 question tokens. All answers come out of one pass, with no autoregressive loop.

import torch
from doomguy import vit

net = vit.PixelBrainQ(d=192, depth=4, heads=3)
tokens = 2 + len(QB.NAMES) + vit.GRID[0] * vit.GRID[1]
d, L = 192, 4
macs = (
    L * (tokens * (4 * d * d + 8 * d * d) + 2 * tokens * tokens * d)
    + vit.GRID[0] * vit.GRID[1] * 600 * d
)
print(
    f"parameters: {vit.n_params(net) / 1e6:.2f}M   tokens: {tokens}   ~{macs / 1e6:.0f}M MACs per decision"
)

parameters: 2.08M   tokens: 174   ~373M MACs per decision

6. Training

run_pipeline.sh steps 4-9: expert play on 160 seeds, frame selection, rendering on the GPNPU kernel, packing, training (15 epochs) and calibration of the dodge and trigger cuts on the validation seeds (the last 8, held out of training). The training log reports validation accuracy per question, including the held-out phrasing.

log = config.MODELS / "small.log"
if log.exists():
    epoch, held, unseen = NB.training_scores(log)
    print(f"{epoch} (last): validation accuracy per question; held-out phrasing in brackets")
    for q, v in held.items():
        if q != "score":
            extra = f"   ({unseen[q]:.3f})" if q in unseen else ""
            print(f"  {NB.question_name(q):36s} {v:.3f}{extra}")
else:
    print("Train first: ./run_pipeline.sh (steps 4-9)")

epoch 14 (last): validation accuracy per question; held-out phrasing in brackets
  Shoot?                               0.935   (0.934)
  What now?                            0.763   (0.768)
  Dodge?                               0.881
  Which way?                           0.590
  What's going on?                     0.792
  Is that fireball going to hit me?    0.912   (0.893)
  Anyone there?                        0.964
  Is he close?                         0.920
  How many of them?                    0.886
  Where is he?                         0.926
  Where's that fireball coming from?   0.854
  Am I hurt?                           0.995
  Dodge? (F1 of sidestep)              0.488   (0.490)

7. One GPNPU Graph

doomguy/deploy.py rewrites the trained student as one graph the Chimera Graph Compiler (CGC) compiles with no custom operators, with the same answers:

• Input = the token rows. One (1, 174, 603) tensor: a row per token in model order. A patch row holds its 600 pixels (two frames, 10x10, channel-major), the state row holds health/ammo/armor; everything else is zero. A stride-10 10x10 convolution is exactly a GEMM over these rows, so one GEMM embeds every token (patch projection and state projection stacked), and the fixed tokens (percept, the 12 baked question tokens, biases, positions) are one constant added after it. No slices or concatenations of computed tensors remain.
• Attention written out: Q, K, V projections, MatMul, Softmax, MatMul.
• Cosine scoring without a reduction: one dense layer emits [a, -a]; its LayerNorm is exactly [a, -a] * sqrt(k) / |a| because its mean is zero, and a MatMul against [options; -options] gives the scaled cosine.

It is quantized to symmetric per-tensor int8 with calibration frames from the training seeds. The graph takes its input already in int8: the host quantizes the token rows with the graph's input scale, exactly as the graph's first QuantizeLinear did, so a decision uploads 105 KB instead of 420 KB and gets the same answers.

exp_json = config.MODELS / "gpnpu" / "small" / "export.json"
if READY and not exp_json.exists():
    sh("python3 -m doomguy.pipeline.export_vit small")
if exp_json.exists():
    exp = json.load(open(exp_json))
    print(f"rewritten graph vs trained model, answers agree: {exp['deploy_vs_trained']:.4f}")
    print("int8 vs float answers agree (500 held-out frames):")
    for q, v in exp["int8_vs_float"].items():
        print(f"  {QB.Q[q][3]:36s} {v:.1%}")
    print(f"perception outputs, mean abs diff: {exp['perception_abs_diff_mean']:.4f}")
else:
    print("Train first: ./run_pipeline.sh (steps 4-9)")

rewritten graph vs trained model, answers agree: 1.0000
int8 vs float answers agree (500 held-out frames):
  Shoot?                               99.0%
  What now?                            96.4%
  Dodge?                               99.2%
  Which way?                           90.6%
  What's going on?                     97.2%
  Is that fireball going to hit me?    99.6%
  Anyone there?                        99.0%
  Is he close?                         96.6%
  How many of them?                    98.4%
  Where is he?                         98.0%
  Where's that fireball coming from?   98.2%
  Am I hurt?                           100.0%
perception outputs, mean abs diff: 0.0202

8. Compile for QC-P and Run on the ISS

ChimeraJob compiles the int8 graph with CGC for QC-P with 8 MACs per PE (no custom operators). The ISS then runs the 16 held-out check frames, the cell prints the job's compile and post-ISS report and the exact cycles of each decision, and compares the GPNPU's outputs with ONNX Runtime running the same int8 graph on the host. Gate: on every frame, all 12 answers equal ONNX Runtime's, and the 43 perception outputs are within 8 int8 steps of ONNX Runtime's (the two int8 implementations are not bit-identical inside the graph; measured up to 5 steps).

import logging

from doomguy import gpnpu
from doomguy.pipeline import compile_vit as CV
from tvm.contrib.epu.chimera_job.chimera_job import ChimeraJob
from tvm.contrib.epu.chimera_job.hw_config import HWConfig

res_json = config.MODELS / "gpnpu" / "small" / "QC-P_m8" / "results.json"
res = json.load(open(res_json)) if res_json.exists() else None
if exp_json.exists() and res is None:
    exp = CV.exported("small")
    with CV.build_dir("small", "QC-P", 8) as build:
        hw = HWConfig(product="QC-P", macs_per_pe=8, ocm_size=gpnpu.ocm_size("QC-P"))
        cgc_job = ChimeraJob(model_p=exp["int8"], trange_file=exp["tranges"], hw_config=hw)
        logging.getLogger("epu").setLevel(logging.WARNING)  # the job sets INFO: lines per frame
        cgc_job.compile(quiet=True)
        res = CV.check(cgc_job, exp)
        json.dump(res, open(build / "results.json", "w"), indent=1)
if res:
    print(res["report"])  # the ChimeraJob after its last ISS run
    NB.show_vit_results(res)
    assert CV.gate_failure(res) is None, CV.gate_failure(res)
    print(
        "PASS: on every frame the GPNPU's answers equal ONNX Runtime's and its perception outputs "
        f"are within {CV.AUX_STEPS} int8 steps"
    )
else:
    print("Export the ViT first (section 7)")

held-out frame 0: 1,971,460 cycles; answers equal ORT int8 1.00; perception max abs diff 0.05530
held-out frame 1: 1,971,460 cycles; answers equal ORT int8 1.00; perception max abs diff 0.04424
held-out frame 2: 1,971,460 cycles; answers equal ORT int8 1.00; perception max abs diff 0.05530
held-out frame 3: 1,971,460 cycles; answers equal ORT int8 1.00; perception max abs diff 0.00000
held-out frame 4: 1,971,460 cycles; answers equal ORT int8 1.00; perception max abs diff 0.00000
held-out frame 5: 1,971,460 cycles; answers equal ORT int8 1.00; perception max abs diff 0.00000
held-out frame 6: 1,971,460 cycles; answers equal ORT int8 1.00; perception max abs diff 0.01106
held-out frame 7: 1,971,460 cycles; answers equal ORT int8 1.00; perception max abs diff 0.02212
held-out frame 8: 1,971,460 cycles; answers equal ORT int8 1.00; perception max abs diff 0.00000
held-out frame 9: 1,971,460 cycles; answers equal ORT int8 1.00; perception max abs diff 0.02212
held-out frame 10: 1,971,460 cycles; answers equal ORT int8 1.00; perception max abs diff 0.05530
held-out frame 11: 1,971,460 cycles; answers equal ORT int8 1.00; perception max abs diff 0.00000
held-out frame 12: 1,971,460 cycles; answers equal ORT int8 1.00; perception max abs diff 0.04424
held-out frame 13: 1,971,460 cycles; answers equal ORT int8 1.00; perception max abs diff 0.05530
held-out frame 14: 1,971,460 cycles; answers equal ORT int8 1.00; perception max abs diff 0.01106
held-out frame 15: 1,971,460 cycles; answers equal ORT int8 1.00; perception max abs diff 0.00000

╒═════════════════════╤════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╕
│ Module Name         │ small_vit_OpSet16_optimized_sym_int8_q_shaped_QC_P_1d7_8MB_4kB_128GBps_128GBps_8_OFF_x1_x1                         │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ ONNX File           │ /quadric/sdk-cli/examples/games/doomguy/work/models/gpnpu/small/small_vit_OpSet16_optimized_sym_int8_q_shaped.onnx │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Product Target      │ QC-P                                                                                                               │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Number of Cores     │ 1                                                                                                                  │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ ISS Clock Frequency │ 1.700                                                                                                              │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ L2M Size            │ 8MB                                                                                                                │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ LRM Size            │ 4kB                                                                                                                │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ External Read BW    │ 128GBps                                                                                                            │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ External Write BW   │ 128GBps                                                                                                            │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ MACS per PE         │ 8                                                                                                                  │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Max L2M             │ 1.485MB                                                                                                            │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Max LRM             │ 0.271kB                                                                                                            │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Max Temp Ext Bytes  │ 0.000MB                                                                                                            │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Network GMACs       │ 0.376                                                                                                              │
╘═════════════════════╧════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╛

╒════╤════════╤════════╤═══════════════╤══════════════════════════╤═══════╕
│    │ Type   │ Name   │ shape         │ type                     │ mse   │
╞════╪════════╪════════╪═══════════════╪══════════════════════════╪═══════╡
│  0 │ Input  │ input  │ [1, 174, 603] │ tensor[int8]             │ n/a   │
├────┼────────┼────────┼───────────────┼──────────────────────────┼───────┤
│  1 │ Output │ logits │ [1, 14, 58]   │ tensor[FixedPoint32<27>] │ n/a   │
├────┼────────┼────────┼───────────────┼──────────────────────────┼───────┤
│  2 │ Output │ aux    │ [1, 14, 43]   │ tensor[FixedPoint32<30>] │ n/a   │
╘════╧════════╧════════╧═══════════════╧══════════════════════════╧═══════╛

Post-ISS Report 1.7 GHz ***
Fully placed-and-routed gate simulation: 
╒══════════════════════════════════╤════════╕
│ Latency (ms)                     │ 1.16   │
├──────────────────────────────────┼────────┤
│ FPS                              │ 862.31 │
├──────────────────────────────────┼────────┤
│ Average Power @ 3nm SSGNP (mW)   │ 867.96 │
├──────────────────────────────────┼────────┤
│ FPS per Watt @ 3nm SSGNP (FPS/W) │ 993.48 │
├──────────────────────────────────┼────────┤
│ Ext Rd Bytes (MB)                │ 3.17   │
├──────────────────────────────────┼────────┤
│ Ext Wr Bytes (MB)                │ 0.01   │
├──────────────────────────────────┼────────┤
│ Avg Ext Rd BW (GBps)             │ 2.67   │
├──────────────────────────────────┼────────┤
│ Avg Ext Wr BW (GBps)             │ 0.00   │
├──────────────────────────────────┼────────┤
│ MAC Utilization                  │ 9.30%  │
╘══════════════════════════════════╧════════╛
*** Data generated using 7nm SSGNP gatesim and scaled to 3nm

[SDK-CLI] : TotalCycles: 1,971,460
[SDK-CLI] : Executions/second: 862.31

compute      : ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 739.883K
data_array   : ▇▇▇▇▇▇▇▇▇▇▇ 172.105K
mac          : ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 297.952K
data_external: ▏ 1.679K
data_ocm     : ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 726.56K

for more information check run directory: /quadric/sdk-cli/examples/games/doomguy/work/models/gpnpu/small/QC-P_m8/ccl_build/small_vit_OpSet16_optimized_sym_int8_q_shaped_QC_P_1d7_8MB_4kB_128GBps_128GBps_8_OFF_x1_x1/run/20261023_002319_f8b66f
QC-P, 8 MACs/PE: 1,971,460 cycles per decision
at 50 MHz: 39.4 ms per decision; at 7.5 decisions/s: 29.6% of the chip
answers equal ONNX Runtime int8 on 16 of 16 frames
perception outputs: up to 5 int8 steps from ONNX Runtime's (one step: 0.01106)
PASS: on every frame the GPNPU's answers equal ONNX Runtime's and its perception outputs are within 8 int8 steps

9. The Voice on the GPNPU

The monologue is Kokoro-82M (am_liam) with its 24 kHz generator replaced by a 3.2M-parameter 8 kHz student distilled from it (doomguy/tts, trained by train + train_gan; the radio voice effect keeps only 250-3400 Hz). Kokoro's text encoder, prosody predictor and pre-decoder stay on the host. The student, the part that turns features into sound, runs on the GPNPU.

doomguy/tts/deploy.py rewrites it into three graphs that CGC compiles with no custom operators:

• Rows of speech. Each call synthesizes 16 chunks of 0.6 s side by side as the rows of 2D tensors (every Conv1d becomes a (1, k) Conv2d), each chunk with 0.1 s of context on both sides, so stitched rows match the whole-line student except at a line's ends; rows from all lines are packed into full calls.
• Snake activations folded. x + sin(a x)^2 / a: a depthwise 1x1 conv computes a x, and 1/a folds into the next conv.
• Polyphase upsampling. Each ConvTranspose (stride 10, then 2) becomes an ordinary conv whose output channels are the output phases; the host interleaves the phases between kernels (a reorder, no arithmetic). Kernel A is the first upsampling, B the 800 frames/s stage, C the 1600 frames/s stage and the spectrum head; the host runs the iSTFT.
• Split spectrum head. Log-magnitudes and phases are two outputs, clipped to [-12, 4] and [-pi, pi], so int8 resolves each on its own scale.

Each kernel is quantized to int8 (calibrated on teacher-data crops), compiled for QC-P and run once on the ISS. Gates: every kernel output on the GPNPU is within 30 dB SNR of ONNX Runtime running the same int8 graph (the snake activations' sin() differs slightly), and the whole voice bank is within 4 dB log-spectral distance of the float student.

VOICE_MODEL = f"kokoro8k_{config.KOKORO_VOICE}_r16_t64"
vsrc = config.MODELS / "gpnpu" / VOICE_MODEL
## the 8 kHz student, trained by the README's Voice steps
STUDENT = config.MODELS / f"kokoro8k_{config.KOKORO_VOICE}.pt"
if STUDENT.exists() and not (vsrc / "export.json").exists():
    sh("python3 -m doomguy.tts.export_gpnpu 16 64 16")
if (vsrc / "export.json").exists():
    for part in "abc":
        if not (vsrc / "QC-P_m8" / part / "results.json").exists():
            sh(f"python3 -m doomguy.tts.compile_gpnpu {VOICE_MODEL} {part}", tail=400)
    vres = {p: json.load(open(vsrc / "QC-P_m8" / p / "results.json")) for p in "abc"}
    worst = NB.show_voice_kernels(vres)
    print(f"GPNPU vs ONNX Runtime int8: {worst:.1f} dB or better on every output (gate 30 dB)")
    assert worst >= 30.0, "GPNPU output below 30 dB against ONNX Runtime int8"
    print("PASS: every voice kernel output reaches 30 dB SNR against ONNX Runtime int8")
else:
    print("Train the student first (README, Voice), then rerun this cell.")

/models/gpnpu/kokoro8k_am_liam_r16_t64/QC-P_m8/a/a_OpSet16_optimized_sym_int8_q_shaped.onnx
{"part": "a", "product": "QC-P", "macs_per_pe": 8, "cycles_per_call": 2113535, "speech_s_per_call": 9.6, "cycles_per_speech_s": 220159.89583333334, "ort_int8_vs_float_snr_db": {"y": 25.105223655700684}, "gpnpu_vs_ort_int8_snr_db": {"y": 301.6880389084278}, "gpnpu_vs_float_snr_db": {"y": 25.105223655700684}}
odels/gpnpu/kokoro8k_am_liam_r16_t64/QC-P_m8/b/b_OpSet16_optimized_sym_int8_q_shaped.onnx
{"part": "b", "product": "QC-P", "macs_per_pe": 8, "cycles_per_call": 25896794, "speech_s_per_call": 9.6, "cycles_per_speech_s": 2697582.7083333335, "ort_int8_vs_float_snr_db": {"y": 23.018357753753662}, "gpnpu_vs_ort_int8_snr_db": {"y": 31.217293739318848}, "gpnpu_vs_float_snr_db": {"y": 23.017501831054688}}
ed.onnx
{"part": "c", "product": "QC-P", "macs_per_pe": 8, "cycles_per_call": 29352519, "speech_s_per_call": 9.6, "cycles_per_speech_s": 3057554.0625, "ort_int8_vs_float_snr_db": {"mag": 27.49713659286499, "phase": 19.543977975845337}, "gpnpu_vs_ort_int8_snr_db": {"mag": 44.21212673187256, "phase": 35.85122108459473}, "gpnpu_vs_float_snr_db": {"mag": 27.486331462860107, "phase": 19.5438814163208}}
kernel                                         cycles per call   int8 vs float
A       first upsampling (x10)                       2,113,535   25.1 dB
B       800 frames/s stage, upsampling (x2)         25,896,794   23.0 dB
C       1600 frames/s stage, spectrum head          29,352,519   mag 27.5 dB, phase 19.5 dB
all     one call = 9.6 s of speech                  57,362,848

5.98 M cycles per second of speech in full calls
GPNPU vs ONNX Runtime int8: 31.2 dB or better on every output (gate 30 dB)
PASS: every voice kernel output reaches 30 dB SNR against ONNX Runtime int8

The voice bank: every monologue line, synthesized once. features runs Kokoro's text side on the host; bank runs every call through the three kernels on simulated QC-P arrays in parallel worker processes and writes the clips with their cycles. The video below uses these clips.

BANK = config.VOICE / f"clips_kokoro8k-gpnpu_{config.KOKORO_VOICE}"
FEATURES = config.VOICE / f"gpnpu_features_{config.KOKORO_VOICE}.npz"
if STUDENT.exists() and not (BANK / "cycles.json").exists():
    if not FEATURES.exists():
        sh("python3 -m doomguy.tts.gpnpu_voice features")
    sh(f"python3 -m doomguy.tts.gpnpu_voice bank {VOICE_MODEL} 16", tail=600)
if (BANK / "cycles.json").exists():
    vb = json.load(open(BANK / "cycles.json"))
    NB.show_voice_bank(vb)
    assert (
        vb["lsd_vs_float_student_db"] <= 4.0
    ), "GPNPU voice: more than 4 dB LSD from the float student"
    print("PASS: the GPNPU voice bank is within 4 dB LSD of the float student")
    NB.voice_example(BANK, FEATURES, "Where is he? On my left. Strafing left.")
else:
    print("The voice bank needs the trained student (README, Voice) and its kernels (above)")

The voice bank needs the trained student (README, Voice) and its kernels (above)

10. Closed Loop on the GPNPU

BRAIN=gpnpu runs the student closed loop with both kernels on the simulated QC-P: every other game tick the renderer draws the frame, every 4th tick the compiled ViT answers all 12 questions about it. Each kernel runs on its own simulated device with the profiler on, so every frame and every decision records its exact cycles. The game (enemies, fireballs, collisions) and the policy stay on the host.

Set SECONDS to the length of play you want. Simulating one game second (15 frames and 7.5 decisions) takes about 38 s of wall time, mostly simulating the ViT, so 8 s of play take about 5 minutes and a full 150 s episode about 1.6 hours. On a HAPS-200 the same second takes 0.42 s of the GPNPU. Each (SEED, SECONDS) pair gets its own episode folder, and the same pair always plays the same game.

import time

SEED = 171  # arena seed; 161-172 were never seen in training
SECONDS = 8  # game seconds to play
## the GPNPU voice bank of section 9, else the same student on the host, else Pocket TTS
VOICE = (
    "kokoro8k-gpnpu"
    if (BANK / "cycles.json").exists()
    else "kokoro8k"
    if STUDENT.exists()
    else "pocket"
)
TAG = f"_nb_gpnpu_{SECONDS}s"
env = {
    "BRAIN": "gpnpu",
    "TAG": TAG,
    "SECONDS": str(SECONDS),
    "GAME_S": str(SECONDS - 1),
    "VOICE_ENGINE": VOICE,
}
ep = config.PLAY / f"small{TAG}_seed{SEED}"
pk = None
if READY and res:
    print(f"seed {SEED}, {SECONDS} s of play: expect about {SECONDS * 38 / 60:.0f} min of ISS time")
    t0 = time.time()
    sh(f"python3 -m doomguy.pixel_play small {SEED}", env, tail=400)
    wall = time.time() - t0
    pk = pickle.load(open(ep / "play.pkl", "rb"))
else:
    print("Needs the trained ViT (run_pipeline.sh) compiled for QC-P (section 8)")

Needs the trained ViT (run_pipeline.sh) compiled for QC-P (section 8)

if pk is not None:
    NB.episode_cycles(pk["frames"], wall)

if pk is not None:
    sh(f"python3 -m doomguy.compose small {SEED}", env, tail=600)
    os.makedirs("videos", exist_ok=True)
    mp4 = f"videos/doomguy_gpnpu_seed{SEED}_{SECONDS}s.mp4"
    subprocess.run(
        [os.environ.get("FFMPEG", "ffmpeg"), "-loglevel", "error", "-y", "-framerate", "15",
         "-i", str(ep / "frames" / "f%05d.png"), "-i", str(ep / "track.wav"), "-c:v", "libx264",
         "-pix_fmt", "yuv420p", "-r", "30", "-c:a", "aac", "-shortest", mp4],
        check=True,
    )  # fmt: skip
    display(Video(mp4, width=960))

    lines = json.load(open(ep / "speech.json"))
    speech_s = sum(line["seconds"] for line in lines)
    print(f"voice: {len(lines)} lines, {speech_s:.1f} s of speech ({VOICE})")
    for line in lines:
        print(f"  at {line['t']:5.1f} s, {line['seconds']:.1f} s long: {line['text']}")
    if VOICE == "kokoro8k-gpnpu":
        vc = vb["cycles_per_speech_s"] * speech_s
        print(
            f"voice generator on the GPNPU: {vc / 1e6:,.0f} M cycles = {vc / NB.CLOCK_HZ:.2f} s at 50 MHz "
            f"for {speech_s:.1f} s of speech"
        )

Where the cycles go. The renderer is re-run on scenes of this episode with its profiling build (make renderer-profile), which reports every stage of the frame; the ViT's per-layer profile comes from the CGC harness run of section 8 (it is the same for every frame).

import matplotlib.pyplot as plt

if pk is not None:
    sh(f"python3 -m doomguy.evaluate.cycle_breakdown small {ep} 24", tail=3000)
    bd = json.load(open(ep / "cycle_breakdown.json"))
    fig, axes = plt.subplots(1, 2, figsize=(13, 3.8))
    parts = (
        (
            "renderer, cycles per frame",
            {k: v for k, v in bd["renderer_stages_mean"].items() if k != "frame"},
        ),
        ("ViT, cycles per decision", bd["vit_layers"]),
    )
    for ax, (title, d) in zip(axes, parts):
        items = sorted(d.items(), key=lambda kv: kv[1])
        ax.barh([k for k, _ in items], [v / 1e3 for _, v in items])
        ax.set_title(title), ax.set_xlabel("K cycles"), ax.grid(alpha=0.3, axis="x")
    plt.tight_layout()
    plt.show()

11. Full Episodes on the GPNPU

run_pipeline.sh step 14 plays the 12 unseen seeds for 150 s each with the renderer and the compiled int8 ViT on the ISS (about 1.7 h with the seeds in parallel), then composes the best one. The table compares them with the same student run in float on the host CPU, then checks how often the int8 GPNPU's answers match the float model's on the exact frames the GPNPU saw. Gate: every answer, and the trigger and dodge cuts, agree with the float model on at least 85% of those frames.

cpu, npu = NB.episode_results("_aimfire"), NB.episode_results("_gpnpu")
if cpu or npu:
    NB.show_episode_table(cpu, npu)
else:
    print("No full episodes yet: ./run_pipeline.sh (steps 10 and 14)")
render, think = NB.gpnpu_cycles(npu) if npu else (np.array([]), np.array([]))
if len(render):
    print(
        f"GPNPU cycles over {len(render):,} frames (texture uploads left out), {len(think):,} decisions:"
    )
    print(
        f"  renderer: mean {render.mean():,.0f}  p95 {np.percentile(render, 95):,.0f}  max {render.max():,}"
    )
    print(f"  ViT     : {', '.join(f'{c:,}' for c in sorted(set(think)))} on every decision")

No full episodes yet: ./run_pipeline.sh (steps 10 and 14)

if npu:  # int8 GPNPU answers vs the float model on the exact frames the GPNPU saw
    seeds = ",".join(map(str, sorted(npu)))
    sh(f"python3 -m doomguy.evaluate.gpnpu_agreement small _gpnpu {seeds}", tail=0)
    ag = json.load(open(config.PLAY / "small_gpnpu_agreement.json"))
    print(f"int8 GPNPU vs float answers on the same {ag['decisions']:,} frames:")
    for q, v in ag["argmax_agreement"].items():
        print(f"  {QB.Q[q][3]:36s} {v:.1%}")
    print(f"  {'trigger cut on the same side':36s} {ag['trigger_cut_agreement']:.1%}")
    print(f"  {'dodge cut on the same side':36s} {ag['dodge_cut_agreement']:.1%}")
    cuts = (ag["trigger_cut_agreement"], ag["dodge_cut_agreement"])
    low = min(*ag["argmax_agreement"].values(), *cuts)
    assert low >= 0.85, f"int8 GPNPU vs float: one decision agrees on only {low:.1%} (gate 85%)"
    print("PASS: every answer and both cuts agree with the float model on at least 85% of frames")

## the best episode as run_pipeline.sh picks it: survived, waves, kills, fewest hits, then the higher seed
best = max(npu, key=lambda s: (*npu[s][:3], -npu[s][3], s)) if npu else None
video = config.OUT / f"doomguy_small_gpnpu_seed{best}.mp4"
if best and video.exists():
    import shutil

    os.makedirs("videos", exist_ok=True)
    shutil.copy(video, f"videos/{video.name}")
    print(f"best episode: seed {best}")
    display(Video(f"videos/{video.name}", width=960))

12. Real-Time Budget on HAPS-200 (QC-P, 50 MHz)

The game shows 15 frames per second and decides 7.5 times per second, and he speaks some of the time. From the measured cycles:

if len(render) and (BANK / "cycles.json").exists():
    NB.show_haps_budget(render, think, json.load(open(BANK / "cycles.json"))["cycles_per_speech_s"])
else:
    print("Needs the GPNPU episodes (run_pipeline.sh step 14) and the voice bank (section 9)")

Needs the GPNPU episodes (run_pipeline.sh step 14) and the voice bank (section 9)

Renderer and ViT together use 41.7% of a QC-P at the HAPS-200 clock, live, leaving room for the game logic. The voice generator adds about 14% while he speaks; it is measured as an offline batch, since the monologue lines are synthesized once ahead of time. Kokoro-82M's own generator would need 24.6 GMAC per second of speech; the 8 kHz student needs about 2 and its convolutions run at up to a third of the int8 peak once 16 rows fill the PE array. Kokoro's text side (about 2.9 GMAC per second of speech) still runs on the host. The ViT is not MAC-bound: most of its cycles are load stalls in the attention and MLP layers (16 MACs per PE saved 3% on the previous training run's weights), so its cost can still come down.

Summary

Key takeaways:

1. A pixel policy fits one graph with no custom operators. Token rows make the patch embedding a single GEMM, and cosine scoring needs no reduction because LayerNorm of [a, -a] is [a, -a] * sqrt(k) / |a|.
2. Renderer and brain fit live on a 50 MHz QC-P with room to spare; the voice fits too as a batch, measured as a phrase bank synthesized ahead of time.
3. Shape the work for the PE array. Sixteen rows of speech fill the 16 PE rows (about a third of the int8 peak), and polyphase convolutions replace transposed convolutions, which CGC runs as slow per-PE gathers.
4. The int8 GPNPU matches ONNX Runtime exactly on the answers; the remaining loss is int8 itself, measured against the float model on the frames the GPNPU saw.

Results

Small ViT (4x192, 2.1M params), 12 unseen seeds, 150 s each:

In int8 on the GPNPU the student plays as well as in float. On the exact frames the GPNPU saw, its int8 answers match the float model's 99.1% of the time for "Shoot?" and 99.3% for "Dodge?"; the small differences compound over an episode. The expert, which reads the game's memory, still out-fights the student; dodging is the student's weakest skill, at about 3x the expert's fireball hits.

Licensing

LAYA (and the laya package that runs it), MiniLM and Kokoro-82M (with its voices) are Apache-2.0. Pocket TTS is CC BY 4.0. The reference voice (Voice-Zero, Bill Boerst) is CC0. The frames are drawn with the DOOM shareware WAD's textures and sprites, which carry no open-source licence; the pixel ViT is trained on such frames, and its check frames are some of them (see ATTRIBUTION.md).