Model Demo: Qwen3-VL-2B Image Description

NOTE: The Jupyter Notebook below is included in the Chimera SDK and can be run interactively by running the following CLI command:

$ quadric sdk notebook

From the Jupyter Notebook window in your browser, select the notebook named /quadric/sdk-cli/examples/models/qwen/qwen3_vl_2b/qwen3_vl_2b.ipynb.

Qwen3-VL-2B Image Description on Chimera GPNPU

This notebook takes Qwen3-VL-2B-Instruct, Alibaba's 2-billion-parameter vision-language model (a 24-block vision transformer feeding a 28-layer language model), from its Hugging Face PyTorch checkpoint to describing a photograph with every model stage on the Chimera GPNPU. One program runs the vision tower and the 128-token prompt prefill from the pixels to the first token; a decode-step program then generates the answer token by token. Both run on the ISS (Instruction Set Simulator).

Pipeline

Model: Qwen3-VL-2B-Instruct (2.1B parameters: a 24-block ViT of width 1024 with three deepstack taps, and a 28-layer language model of width 2048 with grouped-query attention; 256 x 256 input, 256 patches, 64 image tokens)

Every Linear layer, in both the vision tower and the language model, runs W4A16: 4-bit weights with a scale and zero point per block of 128 input channels, and 16-bit activations (fp16 on the matrix engine for the tower and the prefill; a 16-bit fixed-point row for each decode step). The weights come from a quadric_pyquant delivery calibrated on 128 Flickr30k image-caption pairs.

Why the PyTorch path?

The model never leaves PyTorch. Each stage is a ChiPy custom op: a torch.nn.Module whose forward() computes exactly what its CCL (Chimera Compute Language) kernel computes, so the same code is the device program and its host-CPU reference. There is no ONNX export to maintain, and Qwen3-VL's interleaved M-RoPE, deepstack feature injection and fused attention are written once, as kernels.

Note — split pipeline.

• On the Chimera GPNPU: the vision tower, the prompt prefill with the embedding lookup and the LM head, and every decode step
• On the host CPU: the image resize and patchify, the greedy argmax, and passing the KV cache between ISS runs

The full pipeline can run end-to-end on the GPNPU — see yolox_e2e_chipy.ipynb or retina_net_e2e_chipy.ipynb for examples that compose CCL custom ops with ChiPy to keep every stage on-chip.

1. Setup

qwen3vl_w4a16_text holds the language model's ChiPy ops and program builders, qwen3vl_w4a16_vision the vision tower's, qwen3vl_w4a16_runner the ISS drivers and their gates, and qwen3vl_w4a16_helpers the downloads, the golden references and the plots. The GPNPU target is QC-U with 4 MB of OCM (On-Chip Memory), 16 MACs (multiply-accumulate units) per PE (processing element) and 32 GB/s of DDR bandwidth: one core at 1.7 GHz.

import os

import numpy as np
import torch

import qwen3vl_w4a16_helpers as qh
import qwen3vl_w4a16_runner as qr
import qwen3vl_w4a16_vision as qv

%matplotlib inline

NEW_TOKENS = 20  # answer tokens generated on the ISS: one from the prefill, the rest decode steps
NUM_LAYERS = None  # language model layers; None runs all 28
VISION_BLOCKS = None  # vision tower blocks; None runs all 24
CLOCK_HZ = 1.7e9
DDR_BYTES_PER_S = 32e9  # the 32 GB/s DDR budget: 18.82 bytes per cycle at 1.7 GHz
torch.set_num_threads(min(16, os.cpu_count()))

2. Download the Model

From the public sdk-cli-models bucket, at s3://sdk-cli-models/qwen3_vl_2b/w4a16/, come the two files of the quadric_pyquant delivery that inference reads, full_model.pth (the int4 weight codes, the norm weights and the biases) and quant_params.pth (the block scales, the zero points and the calibrated activation ranges), and the golden references (section 3). From the Hugging Face repository (Qwen/Qwen3-VL-2B-Instruct, Apache-2.0), pinned to a fixed revision, comes the tokenizer.

bundle_dir = qh.download_bundle()
golden = qh.load_golden()
tokenizer = qh.decoder()
state_dict, quant_params, cfg = qr.load_bundle(bundle_dir)
depth = NUM_LAYERS or cfg.num_layers
print(
    f"language model: {cfg.num_layers} layers, width {cfg.hidden_size}, vocabulary {cfg.vocab_size}"
)

downloading full_model.pth ...
  full_model.pth: 2.58 GB
downloading quant_params.pth ...
  quant_params.pth: 0.21 GB
downloading qwen3_vl_2b_w4a16_golden.pt ...



Fetching 9 files:   0%|          | 0/9 [00:00<?, ?it/s]



merges.txt: 0.00B [00:00, ?B/s]



config.json: 0.00B [00:00, ?B/s]



generation_config.json:   0%|          | 0.00/269 [00:00<?, ?B/s]



preprocessor_config.json:   0%|          | 0.00/390 [00:00<?, ?B/s]



tokenizer.json: 0.00B [00:00, ?B/s]



video_preprocessor_config.json:   0%|          | 0.00/385 [00:00<?, ?B/s]



chat_template.json: 0.00B [00:00, ?B/s]



tokenizer_config.json: 0.00B [00:00, ?B/s]



vocab.json: 0.00B [00:00, ?B/s]


language model: 28 layers, width 2048, vocabulary 151936

The photograph is held out from calibration: it is the last file of the repository's shared coco-like photo set, and the delivery was calibrated on Flickr30k, which does not contain it. It is resized to 256 x 256, the image processor's minimum size, and cut into a 16 x 16 grid of patches, which the tower merges 2 x 2 into 64 image tokens. The question fills the rest of a 128-token prompt exactly, so the prefill runs one full block with no padding.

ids, pixels, grid, image_start = (
    golden["ids"],
    golden["pixels"],
    golden["grid"],
    golden["image_start"],
)
print(f"photo {golden['photo']}; question: {golden['question']!r}")
print(
    f"{len(ids)} prompt tokens: {ids.count(qv.IMAGE_TOKEN_ID)} image tokens from row {image_start}"
)
print(f"patch grid {grid[1]} x {grid[2]}: pixel patches {tuple(pixels.shape)}")

photo 34501842524_3c858b3080_k.jpeg; question: 'Describe this photograph in detail: the main subject and what it is doing, the setting and background, the lighting and colours, any text or signs you can read, and the mood. Then say in one short sentence what the photograph is about and why someone took it.'
128 prompt tokens: 64 image tokens from row 4
patch grid 16 x 16: pixel patches (256, 1536)

3. The Reference Answers

Two PyTorch models answered the question offline, and their answers ship as the golden references (python qwen3vl_w4a16_reference.py golden writes them). The Hugging Face model in fp32 is the original. The W4A16 model is the same Hugging Face model with the delivery's weights dequantized into it: the quantized network, computed in float, independent of any GPNPU code. Where the two differ, the difference comes from the 4-bit weights, and the device is expected to follow the W4A16 model. The same run recorded both models' logits along the ChiPy chain's own answer, which section 8 scores the ISS against.

On this photograph, teacher-forced on the ChiPy chain's own 24-token answer, the W4A16 model (the delivery's weights, computed in float) picks the same next token as fp32 at 91.7% of positions (KL 0.17 nats per token): the answers describe the same scene, with different wording and some details changed.

print(f"fp32 : {tokenizer.decode(golden['fp32_ids'])!r}")
print(f"W4A16: {tokenizer.decode(golden['w4a16_ids'])!r}")

fp32 : 'This photograph captures a quiet, everyday scene on a suburban road, likely in the United States, with a strong focus on'
W4A16: 'A detailed description of the photograph is as follows:\n\nThe main subject is a red, octagonal stop sign with the word'

4. The CCL Kernels

Two headers hold the kernels. Every one is a @chipy.torch_ccl_custom_op in qwen3vl_w4a16_text or qwen3vl_w4a16_vision whose forward() is its host-CPU reference. Each stage runs a few fused kernels, so the intermediate activations stay in OCM and only the weights stream from DDR.

Two W4 matmuls do the work. Decode reads one row, so matMulNBitsAsym streams the int4 codes once and multiplies them directly against the row, split into two int8 planes, applying each block's zero point and scale afterwards. The prefill and the tower process 128 and 256 rows, so pfGemmW4Core gives each PE an output column, dequantizes a block of weights into its local memory, and broadcasts the activation rows to the whole array.

qh.show_source("qwen3vl_w4a16_ops.hpp", r"void matMulNBitsAsym\(AShape& matA,", lines=11)

## qwen3vl_w4a16_ops.hpp, line 726
__attribute__((noinline)) void matMulNBitsAsym(AShape& matA,
                                               BPackedShape& bPacked,
                                               ScalesI32Shape& scalesI32,
                                               ZeroPointsShape& zeroPoints,
                                               OutShape& matC,
                                               OcmAllocatorType& ocmMem) {
  PROFILE_SCOPE("matMulNBitsAsym");
  auto noPostProcData = emptyTensor();
  matMulNBitsAsymCore<blockSize, scaleFracBits, outFracBits>(
    matA, bPacked, scalesI32, zeroPoints, matC, ocmMem, EmptyType(), noPostProcData);
}

5. The ChiPy Program

Each op is a module like FinalNormOp below: the decorator names the CCL kernel and which inputs are read-only, template_param_factory supplies the kernel's template arguments (here the row count and the binary points taken from the delivery's calibrated ranges), and forward() is the reference. assert_fits raises during the host run if a value would overflow its fixed-point format, which the device would clamp silently.

qh.show_source(
    "qwen3vl_w4a16_text.py", r'ccl_func_name="qwen3vl::finalRmsNorm"', lines=43, before=1
)

## qwen3vl_w4a16_text.py, line 614
@chipy.torch_ccl_custom_op(
    ccl_func_name="qwen3vl::finalRmsNorm",
    reserved_l2m=0,
    reserved_ext=0,
    io_in_ext_mem=True,
    read_only=("inp", "weight"),
)
class FinalNormOp(torch.nn.Module):
    """Final RMSNorm over the last token row only, FX32 in -> FX32 out.

    Prefill samples one token, so slicing before the head keeps lm_head at a single row.
    """

    def __init__(
        self,
        seq_len: int,
        out_frac: int,
        sq_sum_frac: int = SQ_SUM_FRAC,
        edge: str = "final_norm_out",
    ):
        """Bind the block length to slice from and the normalized value's binary point."""
        super().__init__()
        self._edge = edge
        self._seq_len = seq_len
        self._out_frac = out_frac
        self._sq_sum_frac = sq_sum_frac

    @property
    def _frac_bits(self):
        return (self._out_frac,)

    def template_param_factory(self):
        """Return (seqLength, sqSumFracBits, normOutFracBits)."""
        return (self._seq_len, self._sq_sum_frac, self._out_frac)

    def forward(self, inp, weight):
        """Apply the op to the last row of `inp`."""
        x_t = to_torch(inp).float().reshape(self._seq_len, -1)[-1:]
        w_t = to_torch(weight).float()
        rms = torch.sqrt(torch.mean(x_t * x_t, dim=-1, keepdim=True) + RMS_NORM_EPS)
        out = ((x_t / rms) * w_t).reshape(1, 1, 1, x_t.shape[-1])
        assert_fits(self._edge, out, self._out_frac)
        return out

The image program chains the tower (24 blocks and the four mergers), the splice of its 64 rows into the prompt's embedding, the 28 text layers with the deepstack additions, and the LM head. Its inputs are the pixel patches, the token ids, the M-RoPE tables and the empty KV caches.

prompt = (ids, pixels, grid, image_start)
setup = qr.image_setup(cfg, state_dict, quant_params, depth, len(ids), VISION_BLOCKS, prompt=prompt)
print(f"program {setup['module']}: inputs {', '.join(setup['names'])}")

program qwen3vl_w4a16_image24v28_s128: inputs pixel_patches, token_ids, cos, sin, k_cache, v_cache

6. Image to First Token on the ISS

CGC (the Chimera Graph Compiler) compiles the image program into one GPNPU kernel and the ISS runs it. The same program first runs on the host CPU as the ChiPy chain, the reference every device result is gated on. Gate: the next-token logits against the chain, cosine >= 0.9999 and the same top-1 token; the chain's logits one position earlier must fail that cosine, so the gate can tell neighbouring outputs apart.

prefill = qr.prefill_on_iss(setup, cfg, depth, profile=True)
assert prefill is not None, "the image-to-first-token program did not complete on the ISS"
print(
    f"compiled in {prefill['compile_s']:.0f} s;"
    f" {prefill['cycles'] / CLOCK_HZ * 1e3:,.0f} ms at 1.7 GHz"
)
assert prefill["ok"], "the ISS prefill logits do not match the ChiPy chain"

  prefill on the ISS in 12075 s, 128,157,706 cycles
  trunk delta cosine   = 0.99744880   delta gain 0.993917
  trunk rows           : median 0.999969, worst 0.613066 at position 65, 2 of 128 below 0.99
  logits cosine        = 0.99997602   (gate 0.9999)
  top-1 agrees         = True   (reference margin 1.053)
  vs previous position = 0.400823   (inline control, must be < 0.9999: OK)
compiled in 313 s; 75 ms at 1.7 GHz

7. Generate the Answer on the ISS

The decode step runs once per token: the host passes it the previous token, its cache slot and M-RoPE position, and the KV cache the last run returned, and takes the argmax of the logits it returns. The ChiPy chain runs alongside, fed the device's tokens and carrying its own cache. Gate per step: logits cosine >= 0.999 and the same top-1 token, unless the chain's own top-2 logits are within 0.1, a near-tie that fixed-point rounding may flip; the device cache must grow by exactly one slot.

record = "qwen3_vl_2b_w4a16_run.npz"  # tokens and logits, rewritten after every step
result = qr.generate_on_iss(
    prefill, setup, cfg, state_dict, quant_params, depth, NEW_TOKENS, record, profile=True
)
assert result is not None, "a decode step did not complete on the ISS"
iss_ids = result["device_tokens"]
print(f"\nISS answer ({len(iss_ids)} tokens): {tokenizer.decode(iss_ids)!r}")

  step  0 pos 128: device  11682 chain  11682  logits cos 0.999929 (margin 1.246)  cache 129/129 slot cos 0.999909  OK  49,355,941 cycles  721 s
  step  1 pos 129: device   4008 chain   4008  logits cos 0.999979 (margin 2.523)  cache 130/130 slot cos 0.999997  OK  49,366,305 cycles  125 s
  step  2 pos 130: device    315 chain    315  logits cos 0.999992 (margin 7.654)  cache 131/131 slot cos 0.999997  OK  49,372,538 cycles  125 s
  step  3 pos 131: device    279 chain    279  logits cos 0.999987 (margin 9.520)  cache 132/132 slot cos 0.999995  OK  49,377,188 cycles  125 s
  step  4 pos 132: device  10300 chain  10300  logits cos 0.999989 (margin 2.483)  cache 133/133 slot cos 0.999992  OK  49,377,726 cycles  125 s
  step  5 pos 133: device    374 chain    374  logits cos 0.999986 (margin 0.106)  cache 134/134 slot cos 0.999996  OK  49,383,828 cycles  125 s
  step  6 pos 134: device    438 chain    438  logits cos 0.999994 (margin 3.405)  cache 135/135 slot cos 0.999995  OK  49,400,274 cycles  125 s
  step  7 pos 135: device  11017 chain  11017  logits cos 0.999976 (margin 10.488)  cache 136/136 slot cos 0.999978  OK  49,383,826 cycles  125 s
  step  8 pos 136: device   1447 chain   1447  logits cos 0.999982 (margin 3.892)  cache 137/137 slot cos 0.999998  OK  49,499,021 cycles  125 s
  step  9 pos 137: device    785 chain    785  logits cos 0.999994 (margin 3.092)  cache 138/138 slot cos 0.999993  OK  49,503,281 cycles  126 s
  step 10 pos 138: device   1887 chain   1887  logits cos 0.999979 (margin 0.921)  cache 139/139 slot cos 0.999993  OK  49,502,001 cycles  125 s
  step 11 pos 139: device   3832 chain   3832  logits cos 0.999992 (margin 5.581)  cache 140/140 slot cos 0.999997  OK  49,505,201 cycles  126 s
  step 12 pos 140: device    374 chain    374  logits cos 0.999992 (margin 1.863)  cache 141/141 slot cos 0.999995  OK  49,502,275 cycles  126 s
  step 13 pos 141: device    264 chain    264  logits cos 0.999978 (margin 3.748)  cache 142/142 slot cos 0.999971  OK  49,505,235 cycles  126 s
  step 14 pos 142: device   2518 chain   2518  logits cos 0.999911 (margin 0.054)  cache 143/143 slot cos 0.999950  OK  49,503,813 cycles  126 s
  step 15 pos 143: device     11 chain     11  logits cos 0.999965 (margin 1.037)  cache 144/144 slot cos 0.999975  OK  49,484,097 cycles  126 s
  step 16 pos 144: device  18491 chain  18491  logits cos 0.999891 (margin 6.230)  cache 145/145 slot cos 0.999962  OK  49,633,423 cycles  127 s
  step 17 pos 145: device  23450 chain  23450  logits cos 0.999994 (margin 6.652)  cache 146/146 slot cos 0.999997  OK  49,639,037 cycles  127 s
  step 18 pos 146: device   2936 chain   2936  logits cos 0.999987 (margin 4.375)  cache 147/147 slot cos 0.999983  OK  49,637,753 cycles  127 s

ISS answer (20 tokens): 'A detailed description of the photograph is as follows:\n\nThe main subject is a red, octagonal stop'

8. Accuracy

• ISS vs ChiPy chain checks the port against the kernels' own PyTorch forward() code (sections 6 and 7), asserted below. It cannot catch a mistake that a kernel and its forward() share.
• ISS vs the W4A16 model is the independent check, asserted below on a full run: the Hugging Face model with the delivery's weights shares no code with the kernels or their forward(). The ISS must pick the same next token as this model at every generated position, with a logits cosine of at least 0.9999 at each (the two differ only by rounding).
• ISS vs fp32 is the accuracy of the deployed model; W4A16 vs fp32 is the part of it that 4-bit weights cost.

Every gate can fail: validate.py checks each one with a negative control that corrupts the device output (--negative-control zero, or one-word for a single cache word) and must turn it to FAIL.

The golden logits follow the chain's answer, so the ISS is scored on every position up to and including the first token where it chooses differently (all of them when it agrees). Top-1 is the share of positions where the two pick the same next token; KL (Kullback-Leibler divergence) is in nats per token.

scores, rows = qh.score_run(result, golden)
print(f"scored on {rows} of {len(iss_ids)} positions")
print(f"{'':20s} {'top-1':>7s} {'cos mean':>9s} {'cos min':>9s} {'KL':>8s}")
for label, score in scores.items():
    print(
        f"{label:20s} {score['top1']:7.1%} {score['cos_mean']:9.5f} {score['cos_min']:9.5f}"
        f" {score['kl_mean']:8.4f}"
    )
assert result["ok"], "the ISS generation does not match the ChiPy chain"
## The golden references are the full model's, so the independent gate needs a full run.
if NUM_LAYERS is None and VISION_BLOCKS is None:
    w4a16 = scores["ISS vs W4A16 model"]
    assert rows == len(iss_ids), "the ISS answer leaves the ChiPy chain's tokens"
    assert w4a16["top1"] == 1.0, "the ISS picks a different token than the W4A16 model"
    assert (
        w4a16["cos_min"] >= 0.9999
    ), f"the ISS logits are off the W4A16 model: {w4a16['cos_min']:.6f}"
print("PASS: the ISS answer matches the PyTorch reference")

scored on 20 of 20 positions
                       top-1  cos mean   cos min       KL
ISS vs ChiPy chain    100.0%   0.99997   0.99989   0.0001
ISS vs W4A16 model    100.0%   0.99998   0.99993   0.0001
ISS vs fp32            95.0%   0.93731   0.78317   0.1499
W4A16 vs fp32          91.7%   0.93790   0.78507   0.1661
PASS: the ISS answer matches the PyTorch reference

The answers side by side: the ISS, the W4A16 model on the host, and the original model in fp32.

fig = qh.plot_caption(
    golden["image"],
    {
        f"ISS ({len(iss_ids)} tokens on the GPNPU)": tokenizer.decode(iss_ids),
        "W4A16 model (PyTorch)": tokenizer.decode(golden["w4a16_ids"][:NEW_TOKENS]),
        "Hugging Face, fp32": tokenizer.decode(golden["fp32_ids"][:NEW_TOKENS]),
    },
)

9. Performance

Cycles per program on QC-U 4 MB, 16 MACs per PE, one core at 1.7 GHz with 32 GB/s of DDR bandwidth. The decode step is memory bound: it streams every W4 weight and the KV cache once per token.

decode_cycles = int(np.mean(result["decode_cycles"]))
stages = qh.stage_cycles(setup["module"])
fig = qh.plot_stage_cycles(
    {
        "vision tower": stages["vision tower"],
        "prompt prefill + LM head": stages["prompt prefill + head"],
        "one decode step": decode_cycles,
    },
    CLOCK_HZ,
)
first = prefill["cycles"]
print(f"image -> first token {first:>14,} cycles  {first / CLOCK_HZ * 1e3:8.1f} ms")
print(
    f"decode step          {decode_cycles:>14,} cycles  {decode_cycles / CLOCK_HZ * 1e3:8.1f} ms"
    f"  = {CLOCK_HZ / decode_cycles:.1f} tokens/s"
)
step_cycles, step_bytes, _ = qh.kernel_profile(result["module"])
bytes_per_cycle = step_bytes / step_cycles
roofline = DDR_BYTES_PER_S / CLOCK_HZ
print(f"decode DDR traffic {step_bytes / 1e9:.2f} GB/token -> {bytes_per_cycle:.1f} B/cycle")
print(f"({bytes_per_cycle / roofline:.0%} of the 32 GB/s DDR roofline)")

image -> first token    128,157,706 cycles      75.4 ms
decode step              49,470,145 cycles      29.1 ms  = 34.4 tokens/s
decode DDR traffic 0.92 GB/token -> 18.5 B/cycle
(98% of the 32 GB/s DDR roofline)

Summary

Key takeaways

1. A vision-language model runs from pixels to generated text on the Chimera GPNPU: the vision tower, the prompt prefill and each decode step are GPNPU kernels, with no ONNX in between.
2. The ChiPy chain makes every kernel's reference the same PyTorch code, so each ISS run is gated step by step, and a difference from the fp32 model can be traced to the 4-bit weights.
3. Fusing whole layers keeps activations on chip, so the decode step runs at the DDR floor and the prefill and the tower spend their cycles in the matrix engine, not in data movement.

Citation

@misc{bai2025qwen3vl,
  title         = {{Qwen3-VL} Technical Report},
  author        = {Bai, Shuai and Cai, Yuxuan and Chen, Ruizhe and others},
  year          = {2025},
  eprint        = {2511.21631},
  archivePrefix = {arXiv},
  url           = {https://arxiv.org/abs/2511.21631}
}

Model: Qwen/Qwen3-VL-2B-Instruct (Apache-2.0). Photograph: "Oak Ridge_14" (Flickr 34501842524), Simon Edelman, U.S. Department of Energy, a U.S. government work.