Model Demo: Qwen3.5-4B Image Captioning

NOTE: The Jupyter Notebook below is included in the Chimera SDK and can be run interactively by running the following CLI command:

$ quadric sdk notebook

From the Jupyter Notebook window in your browser, select the notebook named /quadric/sdk-cli/examples/models/qwen3_5_4b_vl/qwen3_5_4b_vl.ipynb.

Qwen3.5-4B Image Captioning on Chimera GPNPU

This notebook takes Qwen3.5-4B, Alibaba's natively multimodal 4-billion-parameter model (a 24-block vision transformer feeding a 32-layer hybrid language model of Gated DeltaNet and gated attention), from its Hugging Face PyTorch checkpoint to image captioning with every model stage on the Chimera GPNPU: one program runs the vision tower and the prompt prefill from the image to the first token, and a decode-step program generates the caption token by token, all on the ISS (Instruction Set Simulator).

Pipeline

Model: Qwen3.5-4B (4.5B parameters: a 24-block ViT of width 1024 and a 32-layer hybrid language model of width 2560; 224 x 224 input, 49 image tokens; calibrated on held-out COCO 2017 images)

The language model runs W4A16: 4-bit weights (the community AWQ (Activation-aware Weight Quantization) checkpoint cyankiwi/Qwen3.5-4B-AWQ-4bit, imported code for code) with 16-bit activations. The vision tower runs int8 per-channel weights (SmoothQuant) on the fp16 matrix engine.

Why the PyTorch path?

The model never leaves PyTorch. The same torch.nn.Module code is the float reference and the host-CPU oracle of every GPNPU kernel: each ChiPy op is a module whose forward() computes exactly what its CCL (Chimera Compute Language) kernel computes. There is no ONNX export to maintain, and Qwen3.5's Gated DeltaNet layers, with a 128 x 128 recurrent state per head and no standard ONNX form, are just another module.

What runs where

Note — split pipeline.

• On the Chimera GPNPU: the vision tower, the prompt prefill with the LM head, and every decode step
• On the host CPU: the image patchify, the greedy argmax, and passing the state buffer between ISS runs

The full pipeline can run end-to-end on the GPNPU — see yolox_e2e_chipy.ipynb or retina_net_e2e_chipy.ipynb for examples that compose CCL custom ops with ChiPy to keep every stage on-chip.

1. Setup

qwen35_helpers holds the downloads, the ISS drivers and the plots; qwen35_twin is the plain-PyTorch reference; qwen35_chipy holds the language model's ChiPy ops (decode step and prompt prefill) and qwen35_vision the vision tower's. The GPNPU target is QC-P with 8 MB of OCM (On-Chip Memory) and 8 MACs (multiply-accumulate units) per PE (processing element): one core at 1.7 GHz.

import os
import time

import numpy as np
import torch

import qwen35_chipy as qc
import qwen35_helpers as qh
import qwen35_twin as ref

%matplotlib inline

IMAGE = 0  # golden image captioned on the ISS
NEW_TOKENS = 8  # caption tokens generated on the ISS in section 9
CHAIN_TOKENS = 32  # caption tokens the host chain generates in section 10
torch.manual_seed(0)
torch.set_num_threads(min(16, os.cpu_count()))  # the chain oracle is many small ops

2. Download the Model

The checkpoint and tokenizer come from Hugging Face (Qwen/Qwen3.5-4B, Apache-2.0), pinned to a fixed revision. From the public sdk-cli-models bucket come the device parameters (section 4), the Hugging Face model's golden outputs for eight COCO 2017 validation images (preprocessed to 224 x 224 -> 196 patches -> 49 image tokens), and eight held-out COCO images for calibration.

assets = qh.download_assets()
golden, golden_acts = qh.load_golden(assets)
calib = qh.load_calibration(assets)
tokenizer = qh.load_tokenizer(assets)

sample = golden[IMAGE]
ids = sample["input_ids"].reshape(-1)
n_image = int((ids == ref.IMAGE_TOKEN).sum())
print(f"prompt: {sample['text_prompt']!r}")
print(f"{ids.numel()} prompt tokens ({n_image} image), COCO image {sample['image_id']}")
print(f"HF fp32 caption: {tokenizer.decode(sample['gen_ids'].tolist())!r}")

3. A PyTorch Twin

qwen35_twin restates the model in plain PyTorch in the form the device runs. The changes are exact in real arithmetic: the RMSNorm gains that feed Linears are folded into those Linears (or, for the imported AWQ weights, applied by the device's RMSNorm), the ViT LayerNorms are folded into the next Linear, and the patch-embed convolution is a Linear on the processor's patches. The interleaved M-RoPE tables of the language model (image tokens get 2-D positions, text 1-D) and the ViT's 2-D rotary tables are host constants.

Here the vision tower runs in fp32 and the language model in bf16 (4B parameters in fp32 need 16 GB of host RAM). Gates, against the Hugging Face fp32 run: vision tower cosine > 0.9999, and the same next token from the language model.

t0 = time.time()
twin = ref.Twin()  # bf16 language model + fp32 vision tower
with torch.no_grad():
    vision_embeds = twin.vision(sample["pixel_values"])
    prompt_embeds = twin.embed_inputs(ids, golden_acts["image_embeds"])
    logits = twin.lm(
        prompt_embeds, ref.rope_positions(ids), twin.new_state(qc.MAXSEQ), all_logits=True
    )
vision_cosine = qh.cosine(vision_embeds, golden_acts["image_embeds"])
next_token, hf_next_token = int(logits[-1].argmax()), int(golden_acts["logits"][-1].argmax())
print(f"twin loaded and run in {time.time() - t0:.0f} s")
print(f"vision tower vs HF:     cosine {vision_cosine:.7f}")
print(f"LM logits vs HF (bf16): cosine {qh.cosine(logits[-1], golden_acts['logits'][-1]):.6f}")
print(f"next token {next_token} vs HF {hf_next_token}")
assert vision_cosine > 0.9999, f"vision tower cosine {vision_cosine:.6f} <= 0.9999"
assert next_token == hf_next_token, f"next token {next_token} != HF {hf_next_token}"
del twin, logits

4. The Device Parameters

The language model's 7 Linears per layer run W4 with block-32 scales. Their int4 codes and scales are imported unchanged from the community AWQ checkpoint cyankiwi/Qwen3.5-4B-AWQ-4bit (the scale format is exactly the GEMV (matrix-vector product) kernel's block-32 symmetric W4). AWQ folds its smoothing scales into the RMSNorm gains, so the device applies those gains in its RMSNorm kernels rather than folding them into the weights. The rest was quantized by Quadric: the int8 embedding table (per-row scale), the W4 asymmetric LM head (the tied embedding times the final norm) and the W4 DeltaNet gate projections.

Measured offline on 32 held-out COCO captions (421 tokens, teacher-forced), against the fp32 model, as KL (Kullback-Leibler) divergence and top-1 agreement:

params = qh.load_params(assets / "golden" / "qwen35_w4a16_params.pt")
for kind, (n, mb) in qh.describe_params(params).items():
    print(f"{kind:22s} {n:4d} tensors  {mb:8.0f} MB")

5. The CCL Kernels

Three headers hold the kernels; every one is a @chipy.torch_ccl_custom_op in qwen35_chipy / qwen35_vision whose forward() is its oracle. Data moves DDR (external memory) -> OCM by the ELS (external load/store) engine, OCM -> each PE's LRM (local register memory) by the PLS (PE load/store) flows, and the MEU (matrix engine unit) runs the multiply-accumulates.

Decode step (qwen35_ops.hpp)

• w4gemvT / w4gemvTRes: the column-per-PE W4 x A16 GEMV. Each PE owns one output column; K-chunks of its weights, with their block scales packed into the same chunk, stream DDR -> OCM (one ELS copy per chunk) -> LRM (async PLS, double buffered) while the activation's int16 is split into exact hi/lo int8 planes in registers and broadcast. Epilogues: plain, SwiGLU, and residual + the next RMSNorm (with its gain) kept in registers. The DeltaNet in-projection fuses q|k|v|z|b|a into one GEMV. The loop runs at about 93% of the rate at which a PE can load its weight rows.
• dnMix: one Gated DeltaNet token: causal conv1d (4 taps) against a 3-row ring buffer, the gates g = -exp(A_log) softplus(a + dt), beta = sigmoid(b), a fused single-pass gated delta rule on the 32 x 128 x 128 state S^T (two heads per 256-PE pass, each state column read once for k.S and written once as S' while q.S' accumulates, the state streaming DDR <-> OCM under the compute), and the gated RMSNorm.
• attnGated: q/k per-head RMSNorm, partial M-RoPE (64 of 256 dims), fp16 KV append, attention over the cache, and the sigmoid output gate.
• embedI8, rms, snapshot.

Prompt prefill (qwen35_prefill.hpp)

• pfGemmDq: the fused W4 x fp16 GEMM (matrix-matrix product). Each PE owns one output column of a 256-column tile: it dequantizes a 512-row chunk of its W4 column into fp16 in its LRM and runs one fp16 MEU conv per prompt row against the broadcast activation chunk, all 72 accumulators staying in the LRM; the next tile's weights stage into OCM underneath. About 580 logical MAC/cycle.
• dnPrefill: the DeltaNet over 64-token segments with the SDK's chunked (WY/UT) gated delta rule; padded rows are inert (g = 0, beta = 0) so the state after the prompt is exact.
• attnPrefill: batched gated attention over the prompt: q/k RMSNorm and M-RoPE on all rows, K (transposed on-chip) and V written to the KV cache in one ELS copy each, and the SDK's causal flashAttentionGroup once per KV head (4 query heads each).
• pfSiluMul, pfResRms, rmsRows, w4gemvRow (the LM head on the last row), and embedPrompt (in qwen35_ops.hpp): the prompt's rows, text from the int8 table, image from the tower.

Vision tower (qwen35_vision_ops.hpp): Linears on the fp16 MEU (int8 per-channel weights, stored as fp16 code x scale), GELU by table in the fc1 epilogue, a fused one-pass residual add + LayerNorm, and 2-D-rope fp16 flash attention.

All three share one DDR state buffer layout: per DeltaNet layer its conv ring and S^T, per attention layer its fp16 K and V caches (128 positions).

The heart of pfGemmDq: per K chunk, each PE dequantizes its column's nibbles into fp16 in its LRM, then one fp16 MEU conv per prompt row accumulates that row's output for the PE's column.

qh.show_source(qh.HERE / "qwen35_prefill.hpp", r"for\(std::int32_t ch = 0;", lines=27)

6. The ChiPy Chains

build_decode_step composes one decode token from 163 custom ops (the embedding, five per layer and the LM head); build_prefill wraps each of them in its batched prefill counterpart (sharing weights and calibration, so the prefill writes the state buffer in exactly the formats the decode step reads). VisionChain composes the tower. Each op is a module like EmbedPromptOp below: the decorator names the CCL kernel, template_param_factory its template arguments, and forward() its oracle.

qh.show_source(qh.HERE / "qwen35_chipy.py", r"^class EmbedPromptOp", lines=40, before=6)

Build the decode-step chain from the parameters and calibrate the vision tower on the held-out images.

t0 = time.time()
step = qc.build_decode_step(params, head=True, snapshot=True)
del params
vision = qh.build_vision([s["pixel_values"].float() for s in calib])
print(f"chains built and the vision tower calibrated in {time.time() - t0:.0f} s")
print(
    f"decode step: {sum(1 for _ in qc.all_ops(step))} ops; state buffer {qc.STATE_BYTES / 2**20:.0f} MB"
)

7. Calibrate the Language Model

Every CalibrationPoint records the value range of one tensor while the chain runs on the host, then fixes that tensor's FixedPoint32 format. Two held-out COCO captions (vision rows from the tower's oracle) are enough: the formats have one bit of headroom and are shared by all layers of a kind.

t0 = time.time()
qh.calibrate_lm(step, vision, calib[:2])
print(f"calibrated in {time.time() - t0:.0f} s")

8. Image to First Token on the ISS: One Program

The vision tower, the prompt's embedding rows and the 32-layer prompt prefill compile, with CGC (the Chimera Graph Compiler), into one ChiPy program (qwen35_image_prefill.hpp): its inputs are the image patches, the prompt's token ids and the M-RoPE tables; its outputs are the next-token logits, the filled state buffer and the tower's 49 output rows. On the device, embedPrompt looks the 21 text tokens up in the int8 table and places the tower's rows at the image-token span; nothing goes back to the host in between.

Gate: the tower's rows on the ISS against the chain oracle, cosine > 0.999.

The full image-to-first-token ISS cell can take over three hours with the stock simulator (the earlier complete run took 12,302 seconds). This notebook sets a bounded 18,000-second cell timeout so CI can reach the vision, logits and caption assertions.

pixel_patches = sample["pixel_values"].float()
image_prefill, ip_inputs, ip_ranges = qh.build_image_prefill(step, vision, ids, pixel_patches)
pbuild = qh.compile_chain(
    image_prefill, "q35_image_prefill", qh.HERE / "qwen35_image_prefill.hpp", ip_inputs, ip_ranges
)
print(f"image-to-first-token program compiled in {pbuild['compile_s']:.0f} s")
outs, prefill_cycles, wall = qh.run_on_iss(pbuild, {k: v.values for k, v in ip_inputs.items()})
prefill_logits = outs[0].astype(np.float64).ravel()
state = outs[1].astype(np.int8).reshape(qc.STATE_SHAPE)
image_embeds = outs[2].astype(np.float64).reshape(ref.NIMG, ref.D)
print(f"ISS {wall:.0f} s: {prefill_cycles:,} cycles ({prefill_cycles / qh.CLOCK_HZ * 1e3:,.0f} ms)")
rows_cosine = qh.cosine(image_embeds, vision.cpu(pixel_patches).numpy())
print(f"vision rows, ISS vs chain oracle:  cosine {rows_cosine:.6f}")
print(
    f"vision rows, ISS vs HF fp32 tower: cosine {qh.cosine(image_embeds, golden_acts['image_embeds']):.6f}"
)
assert rows_cosine > 0.999, f"vision rows cosine {rows_cosine:.6f} <= 0.999"

9. Generate the Caption on the ISS

The decode step runs once per token: the host passes it the previous token, its position and the state buffer the last run returned, and takes the argmax of the logits it returns.

dbuild = qh.compile_chain(
    step,
    "q35_decode",
    qh.HERE / "qwen35_ops.hpp",
    qc.step_inputs(1, 0, ref.rope_positions(ids)[:, 0]),
    {"sin": [-1.0, 1.0], "cos": [-1.0, 1.0]},
)
print(f"decode step compiled in {dbuild['compile_s']:.0f} s")
iss_ids, iss_logits, decode_cycles = qh.decode_on_iss(
    dbuild, int(prefill_logits.argmax()), ids, state, NEW_TOKENS
)
print(f"\nISS caption ({len(iss_ids)} tokens): {tokenizer.decode(iss_ids)!r}")

10. Compare Against the PyTorch Reference

The ChiPy chain on the host (every kernel's oracle, the vision tower's included) captions the same image. Checks, all asserted:

1. Vision rows: cosine > 0.999 against the chain (section 8).
2. Prefill: logit cosine >= 0.995 against the chain and the same next token. The device's FixedPoint32 / fp16 rounding drifts from the chain's float64 arithmetic layer by layer: the cosine is 0.9999 after 4 layers and about 0.998 after all 32.
3. Generation: at least 90% of the ISS tokens equal the chain's.
4. Negative control: the prefill logits against the chain's logits one position later must fail the cosine gate, so the gate can tell outputs apart.

The caption is also compared with the original Hugging Face model in fp32; that one is reported, not asserted: a differing word there is a quantization effect, which the chain reproduces.

chain_ids, chain_logits, attn_maps = qh.cpu_caption(
    step, image_prefill, ip_inputs, ids, CHAIN_TOKENS, capture=True
)
prefill_cosine = qh.cosine(prefill_logits, chain_logits[0])
negative_cosine = qh.cosine(prefill_logits, chain_logits[1])
gen_agree = np.mean([a == b for a, b in zip(iss_ids, chain_ids)])
hf_ids = sample["gen_ids"].tolist()
print(
    f"prefill logits: cosine {prefill_cosine:.6f}, next token {int(prefill_logits.argmax())} vs {chain_ids[0]}"
)
print(f"negative control (vs the next position): cosine {negative_cosine:.4f}")
print(f"generation: {gen_agree:.0%} of {len(iss_ids)} ISS tokens match the chain")
print(f"\nISS    : {tokenizer.decode(iss_ids)!r}")
print(f"chain  : {tokenizer.decode(chain_ids)!r}")
print(f"HF fp32: {tokenizer.decode(hf_ids)!r}")
print(f"chain vs HF fp32: {sum(a == b for a, b in zip(chain_ids, hf_ids))}/{len(chain_ids)} tokens")
assert prefill_cosine >= 0.995, f"prefill logits cosine {prefill_cosine:.6f} < 0.995"
assert int(prefill_logits.argmax()) == chain_ids[0], "prefill next token differs from the chain"
assert negative_cosine < 0.995, f"negative control passed the gate: {negative_cosine:.6f}"
assert gen_agree >= 0.9, f"only {gen_agree:.0%} of the ISS tokens match the chain"
print("PASS: the ISS caption matches the PyTorch reference")

The three captions side by side: the ISS, the host chain, and the Hugging Face model in fp32.

fig = qh.plot_caption(
    sample["image_224"],
    {
        f"ISS ({len(iss_ids)} tokens on the GPNPU)": tokenizer.decode(iss_ids),
        "ChiPy chain (host CPU)": tokenizer.decode(chain_ids),
        "Hugging Face, fp32": tokenizer.decode(hf_ids),
    },
)

11. What the Model Looks At

Every generated word attends over the whole prompt, the 49 image tokens included. The chain (the device arithmetic on the host) records those attention weights: here, for each word of the caption, its attention to the image tokens, averaged over the 16 heads and the 8 attention layers and upsampled from the 7 x 7 token grid onto the 224 x 224 input.

n_show = 12
words = [tokenizer.decode([t]).strip() or repr(tokenizer.decode([t])) for t in chain_ids[:n_show]]
fig = qh.plot_word_attention(sample["image_224"], words, attn_maps[:n_show])

12. Performance

Cycles per program on QC-P 8 MB, 8 MACs per PE, one core at 1.7 GHz (the ISS models 128 GB/s DDR). The decode step is memory bound: it streams every W4 weight and the DeltaNet states once per token.

decode_cycles_mean = int(np.mean(decode_cycles))
fig = qh.plot_stage_cycles(
    {
        "image -> first token (vision tower + 70-row prefill)": prefill_cycles,
        "one decode step": decode_cycles_mean,
    }
)
print(
    f"image -> first token {prefill_cycles:>14,} cycles  {prefill_cycles / qh.CLOCK_HZ * 1e3:8.1f} ms"
)
print(
    f"decode step          {decode_cycles_mean:>14,} cycles  {decode_cycles_mean / qh.CLOCK_HZ * 1e3:8.1f} ms"
    f"  = {qh.CLOCK_HZ / decode_cycles_mean:.1f} tokens/s"
)
bytes_per_cycle = qh.DECODE_BYTES_PER_TOKEN / decode_cycles_mean
roofline = qh.DDR_BYTES_PER_S / qh.CLOCK_HZ
print(
    f"decode DDR traffic {qh.DECODE_BYTES_PER_TOKEN / 1e9:.2f} GB/token -> {bytes_per_cycle:.0f} B/cycle"
)
print(f"({bytes_per_cycle / roofline:.0%} of the 128 GB/s DDR roofline)")

Summary

Key takeaways

1. A natively multimodal 4B model runs every model stage on the GPNPU from PyTorch source: the vision tower and the prompt prefill are one program, and the decode step carries the DeltaNet states and KV caches in one DDR buffer.
2. Community quantized weights can be imported as they are when their format matches a kernel's: the AWQ W4 block-32 codes are the GEMV's format and cost only KL 0.102 against fp32.
3. The prefill is compute bound and the decode step memory bound; the kernels differ accordingly (a fused GEMM that keeps its accumulators in the LRM, a GEMV that streams each weight once).
4. Every kernel's oracle is its module's forward(), so the ISS is checked against the same PyTorch code that defines the model, with a negative control on the gate.

Citation

@misc{qwen3.5,
  title  = {{Qwen3.5}: Towards Native Multimodal Agents},
  author = {{Qwen Team}},
  month  = {February},
  year   = {2026},
  url    = {https://qwen.ai/blog?id=qwen3.5}
}
@article{yang2024gated,
  title   = {Gated Delta Networks: Improving Mamba2 with Delta Rule},
  author  = {Yang, Songlin and Kautz, Jan and Hatamizadeh, Ali},
  journal = {arXiv preprint arXiv:2412.06464},
  year    = {2024}
}
@article{lin2023awq,
  title   = {AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration},
  author  = {Lin, Ji and Tang, Jiaming and Tang, Haotian and Yang, Shang and Chen, Wei-Ming and
             Wang, Wei-Chen and Xiao, Guangxuan and Dang, Xingyu and Gan, Chuang and Han, Song},
  journal = {arXiv preprint arXiv:2306.00978},
  year    = {2023}
}
@article{lin2014coco,
  title   = {Microsoft COCO: Common Objects in Context},
  author  = {Lin, Tsung-Yi and Maire, Michael and Belongie, Serge and Bourdev, Lubomir and Girshick, Ross
             and others},
  journal = {arXiv preprint arXiv:1405.0312},
  year    = {2014}
}

Model: Qwen/Qwen3.5-4B (Apache-2.0). AWQ weights: cyankiwi/Qwen3.5-4B-AWQ-4bit (Apache-2.0). Images: COCO 2017 validation set; captions CC BY 4.0, COCO Consortium. The captioned image is "Sink Cat" by Jason Rojas, CC BY 2.0, resized and centre-cropped to 224 x 224. Every image's credit is in qwen3_5_4b_vl-NOTICE_ccby.txt, downloaded with the assets.