NOTE: The Jupyter Notebook below is included in the Chimera SDK and can be run interactively by running the following CLI command:
$ quadric sdk notebook
From the Jupyter Notebook window in your browser, select the notebook named /quadric/sdk-cli/examples/models/whisper/whisper_tutorial.ipynb.
Whisper-Tiny Speech Recognition on Chimera GPNPU
This notebook walks through deploying OpenAI's Whisper-Tiny speech-recognition model on the Quadric Chimera GPNPU, end-to-end, using the Quadric SDK.
Both transformer stages — the encoder and the decoder prefill — run on the GPNPU under the Instruction Set Simulator (ISS). The encoder's ISS output is fed directly into the decoder, so the accuracy numbers reflect the full hardware pipeline rather than per-stage measurements against a floating-point reference.
INT4 weight quantization (MatMulNBits) and FP16 self-/cross-attention keep the full pipeline resident in On-Chip Memory (OCM). Multicore execution scales to 4 cores via a single sdk source --num-cores N flag — no wrapper recompile required.
Pipeline
| Stage | Runs on | Input | Output |
|---|---|---|---|
| Mel spectrogram | host CPU | Raw audio | [1, 80, 3000] |
| Conv front-end | host CPU (ORT) | Mel spectrogram | [1, 1500, 384] post-conv features |
| Encoder (4 layers) | Chimera GPNPU (ISS) | Post-conv features | [1, 1500, 384] hidden states |
| Decoder prefill (4 layers) | Chimera GPNPU (ISS) | Hidden states + 4 prompt tokens | [1, 4, 51865] logits |
Encoder
Decoder
Model: Whisper-Tiny — 4 encoder + 4 decoder layers, embed_dim = 384, 39 M parameters, INT4 weights, FP16 attention. Pre-trained on 680 k hours of multilingual audio.
1. Setup
Install the Python dependencies and download the Whisper-Tiny encoder and decoder ONNX models from the Quadric model S3 bucket.
%pip install -q -r whisper_requirements.txt
import glob
import shutil
import urllib.request
from pathlib import Path
import numpy as np
import onnx
import onnxruntime as ort
from tvm.contrib.epu import onnx_util
from tvm.contrib.epu.chimera_job.chimera_job import ChimeraJob
from tvm.contrib.epu.chimera_job.hw_config import HWConfig
from whisper_helper.encoder_helper import replace_encoder_custom_ops
from whisper_helper.decoder_helper import replace_decoder_custom_ops
from whisper_helper.generate_tranges import generate_tranges
[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv[0m[33m
[0mNote: you may need to restart the kernel to use updated packages.
S3_BASE = "https://sdk-cli-models.s3.amazonaws.com"
S3_MODELS = {
"encoder_model.onnx": f"{S3_BASE}/whisper_encoder_model.onnx",
"decoder_prefill.onnx": f"{S3_BASE}/whisper_decoder_prefill.onnx",
}
def download_models(model_dir: Path, models: dict = S3_MODELS) -> None:
"""Download model files from S3 if not already present locally.
Parameters
----------
model_dir : Path
Local directory where model files are stored.
models : dict, optional
Mapping of ``{local_filename: s3_url}``. Defaults to ``S3_MODELS``.
"""
for local_name, url in models.items():
local_path = model_dir / local_name
if local_path.exists():
print(f" {local_name}: exists ({local_path.stat().st_size:,} bytes)")
continue
print(f" Downloading {local_name}...")
urllib.request.urlretrieve(url, str(local_path))
print(f" {local_name}: {local_path.stat().st_size:,} bytes")
MODEL_DIR = Path(".")
print("Checking / downloading model files...")
download_models(MODEL_DIR)
ENCODER_ONNX = "encoder_model.onnx"
ENCODER_TRANGES = "encoder_model.tranges"
ENCODER_MATCHED = "encoder_model_matched.onnx"
ENCODER_CUT = "encoder_model_matched_cut.onnx"
DECODER_ONNX = "decoder_prefill.onnx"
DECODER_TRANGES = "decoder_prefill.tranges"
DECODER_CUT = "decoder_model_cut.onnx"
DECODER_MATCHED = "decoder_model_matched.onnx"
ENCODER_ONNX_PATH = MODEL_DIR / ENCODER_ONNX
DECODER_ONNX_PATH = MODEL_DIR / DECODER_ONNX
hw_config = HWConfig(product="QC-P", ocm_size="32MB", macs_per_pe=16, num_cores=1)
WHISPER_MODEL_ID = "openai/whisper-tiny"
results = {}
Checking / downloading model files...
Downloading encoder_model.onnx...
encoder_model.onnx: 8,331,061 bytes
Downloading decoder_prefill.onnx...
decoder_prefill.onnx: 95,466,860 bytes
2. Audio & Calibration
Whisper consumes 16 kHz mono audio as an 80-channel log-mel spectrogram ([1, 80, 3000] for 30 seconds of audio).
INT4 weight quantization needs per-tensor min/max statistics to choose fractional-bit allocations. whisper_helper/data_prep.py loads the LibriSpeech "dummy" validation split and runs each sample through the floating-point encoder once to produce a representative set of activations for the calibration step in section 3.
from whisper_helper.data_prep import prepare_whisper_data
SAMPLE_INDEX = 0
NUM_CALIBRATION_SAMPLES = 73
data = prepare_whisper_data(
encoder_onnx=ENCODER_ONNX_PATH,
decoder_onnx=DECODER_ONNX_PATH,
num_calibration_samples=NUM_CALIBRATION_SAMPLES,
sample_index=SAMPLE_INDEX,
)
processor = data.processor
encoder_session = data.encoder_session
decoder_session = data.decoder_session
input_features = data.input_features
calibration_mels = data.calibration_mels
calibration_encoder_outputs = data.calibration_encoder_outputs
Primary: 1272-128104-0000.flac | ref: MISTER QUILTER IS THE APOSTLE OF THE MIDDLE CLASSES AND WE ARE GLAD TO WELCOME HIS GOSPEL
Calibration: 73 mels, 73 encoder outputs
3. Tensor-Range Calibration
Tensor ranges (tranges) record per-tensor min/max statistics from the calibration data. The compiler uses these to assign fixed-point fractional bits — wider ranges get fewer fractional bits, tighter ranges get more.
Encoder ranges come from the mel spectrograms; decoder ranges come from the encoder's floating-point outputs (which give realistic encoder_hidden_states distributions for the decoder's cross-attention inputs).
print(f"Generating encoder tranges ({len(calibration_mels)} samples)...")
generate_tranges(
str(ENCODER_ONNX_PATH),
ENCODER_TRANGES,
num_samples=len(calibration_mels),
mel_spectrograms=calibration_mels,
)
print(f"Generated {ENCODER_TRANGES}")
print(f"\nGenerating decoder tranges ({len(calibration_encoder_outputs)} samples)...")
generate_tranges(
str(DECODER_ONNX_PATH),
DECODER_TRANGES,
num_samples=len(calibration_encoder_outputs),
encoder_outputs_list=calibration_encoder_outputs,
)
print(f"Generated {DECODER_TRANGES}")
Generating encoder tranges (73 samples)...
Processing encoder_model.onnx...
Inputs: [('input_features', [1, 80, 3000])]
Using 73 real audio samples
Running calibration with 73 real audio samples...
Saved 224 tensor ranges to encoder_model.tranges
Generated encoder_model.tranges
Generating decoder tranges (73 samples)...
Processing decoder_prefill.onnx...
Inputs: [('input_ids', [1, 4]), ('encoder_hidden_states', [1, 1500, 384])]
Using 73 real encoder outputs
Running calibration with 73 random samples...
Saved 345 tensor ranges to decoder_prefill.tranges
Generated decoder_prefill.tranges
4. Encoder
The encoder begins with a small convolutional front-end (Conv1 → GELU → Conv2 → GELU → Transpose → PositionalEmbedding) that runs on the host CPU via ONNX Runtime. The GPNPU compilation begins at the post-conv edge /Add_2_output_0 and covers the 4 transformer layers — INT4 MatMulNBits projections and FP16 self-attention.
4.1 Custom Op Replacement
Each MatMulNBits subgraph is replaced with a linalg::matMulNBits custom op, and each self-attention core is replaced with nn::whisperAttention. The convolutional front-end is then cut from the graph so only the transformer block is sent to the compiler.
encoder_graph_info = replace_encoder_custom_ops(
encoder_onnx=ENCODER_ONNX,
encoder_tranges=ENCODER_TRANGES,
encoder_matched_path=ENCODER_MATCHED,
encoder_cut_path=ENCODER_CUT,
)
WARNING:root:ONNX node does not have a name, deriving temp name from its outputs. Node was named to: _v_392__v_393__v_394
WARNING:root:ONNX node does not have a name, deriving temp name from its outputs. Node was named to: _v_391
WARNING:root:ONNX node does not have a name, deriving temp name from its outputs. Node was named to: _v_350__v_351__v_352
WARNING:root:ONNX node does not have a name, deriving temp name from its outputs. Node was named to: _v_349
WARNING:root:ONNX node does not have a name, deriving temp name from its outputs. Node was named to: _v_315__v_316__v_317
WARNING:root:ONNX node does not have a name, deriving temp name from its outputs. Node was named to: _v_314
WARNING:root:ONNX node does not have a name, deriving temp name from its outputs. Node was named to: _v_310__v_312__v_318
WARNING:root:ONNX node does not have a name, deriving temp name from its outputs. Node was named to: _v_309
L0 QKV proj (+bias) → linalg::matMulNBits<31, 31, 28>
L0 attn core
L0 outproj → linalg::matMulNBits<31, 31, 28>
L0 fc1 → linalg::matMulNBits<31, 31, 29>
L0 fc2 → linalg::matMulNBits<31, 31, 29>
Layer 0: all custom ops replaced
L1 QKV proj (+bias) → linalg::matMulNBits<31, 31, 28>
L1 attn core
L1 outproj → linalg::matMulNBits<31, 31, 30>
L1 fc1 → linalg::matMulNBits<31, 31, 28>
L1 fc2 → linalg::matMulNBits<31, 31, 28>
Layer 1: all custom ops replaced
L2 QKV proj (+bias) → linalg::matMulNBits<31, 31, 28>
L2 attn core
L2 outproj → linalg::matMulNBits<31, 31, 28>
L2 fc1 → linalg::matMulNBits<31, 31, 27>
L2 fc2 → linalg::matMulNBits<31, 31, 28>
Layer 2: all custom ops replaced
L3 QKV proj (+bias) → linalg::matMulNBits<31, 31, 28>
L3 attn core
L3 outproj → linalg::matMulNBits<31, 31, 29>
L3 fc1 → linalg::matMulNBits<31, 31, 28>
L3 fc2 → linalg::matMulNBits<31, 31, 27>
Layer 3: all custom ops replaced
Saved matched model to encoder_model_matched.onnx
Cut model: 129 nodes, input=/Add_2_output_0, output=last_hidden_state
4.2 Compilation
Compile the matched encoder through the Chimera Graph Compiler (CGC). CGC emits the kernel body and const_tensor_data.bin — the packed-weights blob the wrapper loads at runtime.
for stale_dir in glob.glob("ccl_build/encoder_model_matched_cut_*"):
shutil.rmtree(stale_dir, ignore_errors=True)
encoder_model_full = onnx.load(str(ENCODER_ONNX_PATH))
frontend_model = onnx_util.cut_onnx(encoder_model_full, [], [encoder_graph_info["conv_output"]])
frontend_session = ort.InferenceSession(frontend_model.SerializeToString())
conv_frontend_output = frontend_session.run(None, {"input_features": input_features})[0]
print(f"Conv front-end output: {conv_frontend_output.shape}")
print("\nCompiling encoder...")
try:
encoder_job = ChimeraJob(
ENCODER_CUT,
hw_config=hw_config,
trange_file=ENCODER_TRANGES,
attn_stub_src_path=str(Path("whisper_matmul_stubs.hpp").resolve()),
bypass_quantization_checks=True,
target_lang="QIL",
validate_iss=False,
io_to_ignore=["__placeholder_to_trigger_iomodifier__"],
)
encoder_job.compile()
except Exception as cgc_error:
results["encoder"] = {"compiled": False}
raise
results["encoder"] = {"compiled": True}
Conv front-end output: (1, 1500, 384)
Compiling encoder...
2026-06-19 03:56 - INFO - epu - chimera_job - START==================================onnx_ingest
2026-06-19 03:56 - INFO - epu - chimera_job - Numerical ranges provided
2026-06-19 03:56 - INFO - epu - codegen - START===============================optimize_relay
2026-06-19 03:56 - INFO - epu - codegen - START====================quantize_to_cpu_runnable_fx
2026-06-19 03:56 - INFO - epu - fx -
Source name Op Output 0 Range Output 0 Frac Bits
-------------------------------------------------------- ----------------------------- ---------------------- --------------------
/layers.0/self_attn_layer_norm/Add_1 nn.layer_norm [-16.5387f, 11.5336f] 26
CustomOp/linalg::matMulNBits<31, 31, 28>0 contrib.epu.quadric_custom_op [-10.2091f, 8.31001f] 27
CustomOp/nn::whisperAttention<384, 6, 1500, 14, false>1 contrib.epu.quadric_custom_op [-5.05174f, 5.65756f] 28
CustomOp/linalg::matMulNBits<31, 31, 28>2 contrib.epu.quadric_custom_op [-4.96675f, 2.94481f] 28
/layers.0/Add add [-2.08293f, 3.24672f] 27
/layers.0/final_layer_norm/Add_1 nn.layer_norm [-14.6758f, 21.1295f] 26
CustomOp/linalg::matMulNBits<31, 31, 29>3 contrib.epu.quadric_custom_op [-10.8754f, 12.0852f] 27
/layers.0/activation_fn/Mul_1 contrib.epu.gelu [-0.169971f, 12.0852f] 27
CustomOp/linalg::matMulNBits<31, 31, 29>4 contrib.epu.quadric_custom_op [-4.11117f, 4.20129f] 28
/layers.0/Add_1 add [-4.7491f, 3.81342f] 27
/layers.1/self_attn_layer_norm/Add_1 nn.layer_norm [-10.1508f, 14.0432f] 27
CustomOp/linalg::matMulNBits<31, 31, 28>5 contrib.epu.quadric_custom_op [-9.54884f, 7.00364f] 27
CustomOp/nn::whisperAttention<384, 6, 1500, 14, false>6 contrib.epu.quadric_custom_op [-2.58438f, 2.65559f] 29
CustomOp/linalg::matMulNBits<31, 31, 30>7 contrib.epu.quadric_custom_op [-1.57448f, 2.00309f] 29
/layers.1/Add add [-4.03927f, 3.76297f] 27
/layers.1/final_layer_norm/Add_1 nn.layer_norm [-13.3634f, 16.9462f] 26
CustomOp/linalg::matMulNBits<31, 31, 28>8 contrib.epu.quadric_custom_op [-10.0729f, 13.8156f] 27
/layers.1/activation_fn/Mul_1 contrib.epu.gelu [-0.169971f, 13.8156f] 27
CustomOp/linalg::matMulNBits<31, 31, 28>9 contrib.epu.quadric_custom_op [-6.80352f, 7.78972f] 28
/layers.1/Add_1 add [-10.0094f, 9.13924f] 27
/layers.2/self_attn_layer_norm/Add_1 nn.layer_norm [-21.9874f, 22.29f] 26
CustomOp/linalg::matMulNBits<31, 31, 28>10 contrib.epu.quadric_custom_op [-9.56785f, 11.4681f] 27
CustomOp/nn::whisperAttention<384, 6, 1500, 14, false>11 contrib.epu.quadric_custom_op [-5.1613f, 5.73556f] 28
CustomOp/linalg::matMulNBits<31, 31, 28>12 contrib.epu.quadric_custom_op [-4.55164f, 3.54009f] 28
/layers.2/Add add [-9.32197f, 9.04343f] 27
/layers.2/final_layer_norm/Add_1 nn.layer_norm [-26.9777f, 24.1529f] 26
CustomOp/linalg::matMulNBits<31, 31, 27>13 contrib.epu.quadric_custom_op [-40.481f, 128.961f] 23
/layers.2/activation_fn/Mul_1 contrib.epu.gelu [-0.169971f, 128.961f] 23
CustomOp/linalg::matMulNBits<31, 31, 28>14 contrib.epu.quadric_custom_op [-27.8089f, 247.797f] 23
/layers.2/Add_1 add [-30.3758f, 253.198f] 22
/layers.3/self_attn_layer_norm/Add_1 nn.layer_norm [-21.1174f, 33.7284f] 25
CustomOp/linalg::matMulNBits<31, 31, 28>15 contrib.epu.quadric_custom_op [-12.3535f, 13.794f] 27
CustomOp/nn::whisperAttention<384, 6, 1500, 14, false>16 contrib.epu.quadric_custom_op [-7.21264f, 8.97383f] 27
CustomOp/linalg::matMulNBits<31, 31, 29>17 contrib.epu.quadric_custom_op [-5.37972f, 11.7859f] 27
/layers.3/Add add [-29.5759f, 264.796f] 22
/layers.3/final_layer_norm/Add_1 nn.layer_norm [-94.3298f, 120.531f] 24
CustomOp/linalg::matMulNBits<31, 31, 28>18 contrib.epu.quadric_custom_op [-121.067f, 329.001f] 22
/layers.3/activation_fn/Mul_1 contrib.epu.gelu [-0.169971f, 329.001f] 22
CustomOp/linalg::matMulNBits<31, 31, 27>19 contrib.epu.quadric_custom_op [-121.728f, 52.5459f] 24
/layers.3/Add_1 add [-131.778f, 276.011f] 22
/layer_norm/Add_1 nn.layer_norm [-17.5703f, 19.0185f] -
2026-06-19 03:56 - INFO - epu - codegen - START====================build_cpu_runnable_fx_relay
2026-06-19 03:56 - INFO - epu - codegen - START=======================quantize_to_chimera_fx
2026-06-19 03:56 - INFO - epu - codegen - START=================================relay_to_tir
2026-06-19 03:56 - INFO - epu - codegen - START===========================relay_to_epu_relay
2026-06-19 03:56 - INFO - epu - codegen - START==============================adapt_and_order
2026-06-19 03:56 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-06-19 03:56 - INFO - epu - codegen - START=============================plan_lrm_virtual
2026-06-19 03:56 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-06-19 03:57 - INFO - epu - codegen - START===============================lrm_alloc_loop
2026-06-19 03:57 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-06-19 03:57 - INFO - epu - codegen - START================================lrm_splitting
2026-06-19 03:57 - INFO - epu - codegen - START==============================ext_split_relay
2026-06-19 03:58 - INFO - epu - codegen - File copied from /quadric/sdk-cli/examples/models/whisper/whisper_matmul_stubs.hpp to /quadric/sdk-cli/examples/models/whisper/ccl_build/encoder_model_matched_cut_QC_P_1d7_32MB_4kB_128GBps_128GBps_16_OFF_x1_x1/attention_stubs.hpp
2026-06-19 03:58 - INFO - epu - codegen - START====================================build_tir
2026-06-19 03:58 - INFO - epu - chimera_job - Compilation of encoder_model_matched_cut_QC_P_1d7_32MB_4kB_128GBps_128GBps_16_OFF_x1_x1 successful
4.3 Build, Run on ISS, and Compare
The same wrapper builds on 1, 2, or 4 cores via sdk source --num-cores N. The next cell compares the GPNPU output against the floating-point ONNX Runtime reference.
Reference accuracy
| Stage | PSNR |
|---|---|
| Single attention-layer kernel only | ~58.5 dB |
| 4-layer encoder, up to final LayerNorm | ~45 dB |
| 4-layer encoder, full output | ~25.9 dB |
ort_ref_output = calibration_encoder_outputs[SAMPLE_INDEX]
## Conv front-end (Conv1 + GELU + Conv2 + GELU + Transpose + PositionalEmbedding)
## already ran on the host CPU via ONNX Runtime; its output is what feeds the
## GPNPU encoder. Quantize that output to FxP28 int32 and drop it next to
## whisper_encoder.cpp.
np.round(conv_frontend_output.astype(np.float64) * (1 << 28)).astype(np.int32).tofile(
"_Add_2_output_0.bin"
)
%%bash
set -euo pipefail
sdk source -v whisper_encoder.cpp \
--include-cgc-headers \
--target QC-P \
--num-cores 4 \
--ocm-size 32MB \
--macs-per-pe 16 \
--clock-freq-ghz 1.7 \
--ext-read-bw 128GBps \
--ext-write-bw 128GBps \
--quiet
2026-06-19 03:58 - DEBUG - sdk - cli - Executing command: cmake CMakeLists.txt -B /quadric/sdk-cli/examples/models/whisper/whisper_encoder_QC-P_32MB_4kB_128GBps_128GBps/build -DNUM_GPNPUS=4 -DNUM_CORES=16 -DNUM_BORDERS=2 -DEPU_VERSION=2.0.0 -DQLLVM_ROOT_PATH=/quadric/llvm -DOCM_SIZE_KIBIBYTES=32768 -DNUM_PE_MACS=16 -DASSERT_MLS_WIDTH_LINE_ALIGN=ON -DHARDWARE_TARGET=OFF
2026-06-19 03:58 - DEBUG - sdk - cli - Executing command: make -j8
[SDK-CLI] : Executing on QC-P simulator
2026-06-19 03:58 - DEBUG - sdk - cli - Executing command: ./whisper_encoder_host -c --ddrRdBwTotal 1048576.0 --ddrWrBwTotal 1048576.0 --ddrAxiWidth 128 --instMemDepth 1310720 --ocmSize 33554432 --cycleTimeNS 0.5882352941176471 --no-check --ddrRdAvgPct 100 --ddrRdMaxPct 100 --ddrWrAvgPct 100 --ddrWrMaxPct 100 --postKernelFlowTimeoutCycles 4000000 --clusterSize 4 --numClusters 1
2026-06-19 04:27 - WARNING - epu - core - No profile.json found. Not able to show performance results.
2026-06-19 04:27 - INFO - epu - core - If you would like to see performance results, add profiling statements to your source code.
[SDK-CLI] : Execution completed.
## Encoder ISS output (FxP26 int32 → fp32) vs ORT.
iss_output = (
(np.fromfile("last_hidden_state.bin", dtype=np.int32).astype(np.float64) / float(1 << 26))
.astype(np.float32)
.reshape(ort_ref_output.shape)
)
abs_diff = np.abs(ort_ref_output - iss_output)
mse = float(np.mean((ort_ref_output - iss_output) ** 2))
mae = float(np.mean(abs_diff))
max_diff = float(np.max(abs_diff))
signal_range = float(ort_ref_output.max() - ort_ref_output.min())
psnr = 10.0 * np.log10(signal_range**2 / mse) if mse > 0 else float("inf")
print("Encoder ORT vs ISS:")
print(f" MAE: {mae:.6f} PSNR: {psnr:.2f} dB Max diff: {max_diff:.6f}")
results["encoder"].update({"mse": mse, "mae": mae, "max_diff": max_diff, "psnr": psnr})
ENCODER_PSNR_MIN_DB = 20.0
assert (
psnr >= ENCODER_PSNR_MIN_DB
), f"Encoder PSNR regression: {psnr:.2f} dB < {ENCODER_PSNR_MIN_DB} dB"
## MSE catches blow-ups that PSNR can mask when the signal range is wider than
## expected (the encoder output's range grows with calibration drift, which
## can keep PSNR steady while errors climb in absolute terms).
ENCODER_MSE_MAX = 5 # ~5× headroom over current MSE; PSNR 20 dB equivalent
assert mse <= ENCODER_MSE_MAX, f"Encoder MSE regression: {mse:.4f} > {ENCODER_MSE_MAX}"
Encoder ORT vs ISS:
MAE: 0.032195 PSNR: 54.50 dB Max diff: 10.286570
5. Decoder
The decoder prefill processes the four Whisper bootstrap tokens (<|startoftranscript|>, <|en|>, <|transcribe|>, <|notimestamps|>) with cross-attention to the encoder's hidden states. Each of the four layers contains causal self-attention, cross-attention, and a feed-forward block. INT4 weights (MatMulNBits) and FP16 attention apply throughout.
The decoder has 8 projections per layer compared with the encoder's 4 because of the separate Q/K/V/output projections in cross-attention.
5.1 Custom Op Replacement
Same pattern as the encoder, with the addition of the cross-attention variant of nn::whisperAttention.
encoder_outputs = iss_output
## Whisper prompt: <|startoftranscript|>, <|en|>, <|transcribe|>, <|notimestamps|>
decoder_input_ids = np.array([[50258, 50259, 50359, 50363]], dtype=np.int32)
replace_decoder_custom_ops(
decoder_onnx=DECODER_ONNX,
decoder_tranges=DECODER_TRANGES,
decoder_cut_path=DECODER_CUT,
decoder_matched_path=DECODER_MATCHED,
)
WARNING:root:ONNX node does not have a name, deriving temp name from its outputs. Node was named to: _v_612__v_614__v_615
WARNING:root:ONNX node does not have a name, deriving temp name from its outputs. Node was named to: _v_489__v_490__v_491
WARNING:root:ONNX node does not have a name, deriving temp name from its outputs. Node was named to: _v_480__v_481__v_482
WARNING:root:ONNX node does not have a name, deriving temp name from its outputs. Node was named to: _v_471__v_472__v_473
/decoder/layers.0/encoder_attn/k_proj/MatMul_Q4 → linalg::matMulNBits<31, 31, 31>
/decoder/layers.0/encoder_attn/v_proj/MatMul_Q4 → linalg::matMulNBits<31, 31, 31>
/decoder/layers.1/encoder_attn/k_proj/MatMul_Q4 → linalg::matMulNBits<31, 31, 31>
/decoder/layers.1/encoder_attn/v_proj/MatMul_Q4 → linalg::matMulNBits<31, 31, 31>
/decoder/layers.2/encoder_attn/k_proj/MatMul_Q4 → linalg::matMulNBits<31, 31, 31>
/decoder/layers.2/encoder_attn/v_proj/MatMul_Q4 → linalg::matMulNBits<31, 31, 31>
/decoder/layers.3/encoder_attn/k_proj/MatMul_Q4 → linalg::matMulNBits<31, 31, 31>
/decoder/layers.3/encoder_attn/v_proj/MatMul_Q4 → linalg::matMulNBits<31, 31, 31>
fused_qkv/_v_611 → linalg::matMulNBits<31, 31, 28>
/decoder/layers.0/self_attn/out_proj/MatMul_Q4 → linalg::matMulNBits<31, 31, 31>
/decoder/layers.0/encoder_attn/q_proj/MatMul_Q4 → linalg::matMulNBits<31, 31, 28>
/decoder/layers.0/encoder_attn/out_proj/MatMul_Q4 → linalg::matMulNBits<31, 31, 31>
/decoder/layers.0/fc1/MatMul_Q4 → linalg::matMulNBits<31, 31, 28>
/decoder/layers.0/fc2/MatMul_Q4 → linalg::matMulNBits<31, 31, 30>
fused_qkv/_v_488 → linalg::matMulNBits<31, 31, 27>
/decoder/layers.1/self_attn/out_proj/MatMul_Q4 → linalg::matMulNBits<31, 31, 31>
/decoder/layers.1/encoder_attn/q_proj/MatMul_Q4 → linalg::matMulNBits<31, 31, 27>
/decoder/layers.1/encoder_attn/out_proj/MatMul_Q4 → linalg::matMulNBits<31, 31, 30>
/decoder/layers.1/fc1/MatMul_Q4 → linalg::matMulNBits<31, 31, 27>
/decoder/layers.1/fc2/MatMul_Q4 → linalg::matMulNBits<31, 31, 29>
fused_qkv/_v_479 → linalg::matMulNBits<31, 31, 28>
/decoder/layers.2/self_attn/out_proj/MatMul_Q4 → linalg::matMulNBits<31, 31, 29>
/decoder/layers.2/encoder_attn/q_proj/MatMul_Q4 → linalg::matMulNBits<31, 31, 28>
/decoder/layers.2/encoder_attn/out_proj/MatMul_Q4 → linalg::matMulNBits<31, 31, 30>
/decoder/layers.2/fc1/MatMul_Q4 → linalg::matMulNBits<31, 31, 28>
/decoder/layers.2/fc2/MatMul_Q4 → linalg::matMulNBits<31, 31, 29>
fused_qkv/_v_470 → linalg::matMulNBits<31, 31, 28>
/decoder/layers.3/self_attn/out_proj/MatMul_Q4 → linalg::matMulNBits<31, 31, 27>
/decoder/layers.3/encoder_attn/q_proj/MatMul_Q4 → linalg::matMulNBits<31, 31, 28>
/decoder/layers.3/encoder_attn/out_proj/MatMul_Q4 → linalg::matMulNBits<31, 31, 27>
/decoder/layers.3/fc1/MatMul_Q4 → linalg::matMulNBits<31, 31, 28>
/decoder/layers.3/fc2/MatMul_Q4 → linalg::matMulNBits<31, 31, 29>
lm_head → linalg::matMulNBits<31, 31, 31>
L0 self-attn FP16 MHA core
L0 cross-attn FP16 MHA core
L1 self-attn FP16 MHA core
L1 cross-attn FP16 MHA core
L2 self-attn FP16 MHA core
L2 cross-attn FP16 MHA core
L3 self-attn FP16 MHA core
L3 cross-attn FP16 MHA core
{'num_layers': 4, 'embed_dim': 384, 'model_output': 'logits'}
5.2 Compilation
The decoder reuses whisper_matmul_stubs.hpp for its custom-op implementations.
for stale_dir in glob.glob("ccl_build/decoder_model_matched_*"):
shutil.rmtree(stale_dir, ignore_errors=True)
print("Compiling decoder...")
try:
decoder_job = ChimeraJob(
DECODER_MATCHED,
hw_config=hw_config,
trange_file=DECODER_TRANGES,
bypass_quantization_checks=True,
attn_stub_src_path=str(Path("whisper_matmul_stubs.hpp").resolve()),
target_lang="QIL",
validate_iss=False,
io_to_ignore=["__placeholder_to_trigger_iomodifier__"],
)
decoder_job.compile()
except Exception as cgc_error:
results["decoder"] = {"compiled": False}
raise
results["decoder"] = {"compiled": True}
Compiling decoder...
2026-06-19 04:27 - INFO - epu - chimera_job - START==================================onnx_ingest
2026-06-19 04:27 - INFO - epu - chimera_job - Numerical ranges provided
2026-06-19 04:27 - INFO - epu - codegen - START===============================optimize_relay
2026-06-19 04:27 - INFO - epu - codegen - START====================quantize_to_cpu_runnable_fx
2026-06-19 04:27 - INFO - epu - fx -
Source name Op Output 0 Range Output 0 Frac Bits
------------------------------------------------- ----------------------------- ----------------------- --------------------
/decoder/embed_tokens/Gather contrib.epu.embedding [-0.154297f, 0.20166f] 31
/decoder/Add add [-0.586182f, 0.470642f] 31
/decoder/layers.0/self_attn_layer_norm/Add_1 nn.layer_norm [-5.01829f, 4.52495f] 28
CustomOp/linalg::matMulNBits<31, 31, 28>8 contrib.epu.quadric_custom_op [-7.66943f, 8.27715f] 27
CustomOp/nn::whisperAttention<384, 6, 4, 14>33 contrib.epu.quadric_custom_op [-0.863014f, 1.08392f] 30
CustomOp/linalg::matMulNBits<31, 31, 31>9 contrib.epu.quadric_custom_op [-0.691889f, 0.544409f] 31
/decoder/layers.0/Add add [-0.680281f, 0.579606f] 30
/decoder/layers.0/encoder_attn_layer_norm/Add_1 nn.layer_norm [-8.14458f, 7.72507f] 27
CustomOp/linalg::matMulNBits<31, 31, 28>10 contrib.epu.quadric_custom_op [-9.41189f, 8.99864f] 27
CustomOp/linalg::matMulNBits<31, 31, 31>0 contrib.epu.quadric_custom_op [-6.85451f, 9.78807f] 27
CustomOp/linalg::matMulNBits<31, 31, 31>1 contrib.epu.quadric_custom_op [-3.03744f, 3.36402f] 29
CustomOp/nn::whisperAttention<384, 6, 1500, 14>34 contrib.epu.quadric_custom_op [-1.37888f, 0.944248f] 30
CustomOp/linalg::matMulNBits<31, 31, 31>11 contrib.epu.quadric_custom_op [-0.551486f, 0.72956f] 31
/decoder/layers.0/Add_1 add [-0.881128f, 1.28411f] 29
/decoder/layers.0/final_layer_norm/Add_1 nn.layer_norm [-8.30888f, 20.6246f] 26
CustomOp/linalg::matMulNBits<31, 31, 28>12 contrib.epu.quadric_custom_op [-7.38382f, 11.0261f] 27
/decoder/layers.0/activation_fn/Mul_1 contrib.epu.gelu [-0.169971f, 11.0261f] 27
CustomOp/linalg::matMulNBits<31, 31, 30>13 contrib.epu.quadric_custom_op [-11.5326f, 1.70436f] 27
/decoder/layers.0/Add_2 add [-11.726f, 1.55193f] 27
/decoder/layers.1/self_attn_layer_norm/Add_1 nn.layer_norm [-9.21633f, 7.00148f] 27
CustomOp/linalg::matMulNBits<31, 31, 27>14 contrib.epu.quadric_custom_op [-6.00798f, 9.36193f] 27
CustomOp/nn::whisperAttention<384, 6, 4, 14>35 contrib.epu.quadric_custom_op [-0.677473f, 0.542494f] 31
CustomOp/linalg::matMulNBits<31, 31, 31>15 contrib.epu.quadric_custom_op [-0.601211f, 1.41293f] 30
/decoder/layers.1/Add add [-11.953f, 1.41258f] 27
/decoder/layers.1/encoder_attn_layer_norm/Add_1 nn.layer_norm [-31.8444f, 18.763f] 26
CustomOp/linalg::matMulNBits<31, 31, 27>16 contrib.epu.quadric_custom_op [-8.70127f, 11.7968f] 27
CustomOp/linalg::matMulNBits<31, 31, 31>2 contrib.epu.quadric_custom_op [-6.54958f, 5.89434f] 28
CustomOp/linalg::matMulNBits<31, 31, 31>3 contrib.epu.quadric_custom_op [-4.18588f, 3.59232f] 28
CustomOp/nn::whisperAttention<384, 6, 1500, 14>36 contrib.epu.quadric_custom_op [-1.3987f, 2.01703f] 29
CustomOp/linalg::matMulNBits<31, 31, 30>17 contrib.epu.quadric_custom_op [-1.27409f, 0.777316f] 30
/decoder/layers.1/Add_1 add [-11.7993f, 1.31637f] 27
/decoder/layers.1/final_layer_norm/Add_1 nn.layer_norm [-34.3886f, 17.1081f] 25
CustomOp/linalg::matMulNBits<31, 31, 27>18 contrib.epu.quadric_custom_op [-18.8776f, 71.4612f] 24
/decoder/layers.1/activation_fn/Mul_1 contrib.epu.gelu [-0.169971f, 71.4612f] 24
CustomOp/linalg::matMulNBits<31, 31, 29>19 contrib.epu.quadric_custom_op [-113.282f, 8.93458f] 24
/decoder/layers.1/Add_2 add [-114.834f, 10.1161f] 24
/decoder/layers.2/self_attn_layer_norm/Add_1 nn.layer_norm [-9.23447f, 15.4071f] 26
CustomOp/linalg::matMulNBits<31, 31, 28>20 contrib.epu.quadric_custom_op [-5.46188f, 5.14847f] 28
CustomOp/nn::whisperAttention<384, 6, 4, 14>37 contrib.epu.quadric_custom_op [-0.822397f, 0.738742f] 31
CustomOp/linalg::matMulNBits<31, 31, 29>21 contrib.epu.quadric_custom_op [-0.471116f, 3.29417f] 29
/decoder/layers.2/Add add [-114.76f, 9.82762f] 24
/decoder/layers.2/encoder_attn_layer_norm/Add_1 nn.layer_norm [-32.0417f, 18.2027f] 25
CustomOp/linalg::matMulNBits<31, 31, 28>22 contrib.epu.quadric_custom_op [-10.4317f, 8.56416f] 27
CustomOp/linalg::matMulNBits<31, 31, 31>4 contrib.epu.quadric_custom_op [-7.25804f, 6.81882f] 28
CustomOp/linalg::matMulNBits<31, 31, 31>5 contrib.epu.quadric_custom_op [-4.18824f, 4.76663f] 28
CustomOp/nn::whisperAttention<384, 6, 1500, 14>38 contrib.epu.quadric_custom_op [-1.28361f, 1.73447f] 30
CustomOp/linalg::matMulNBits<31, 31, 30>23 contrib.epu.quadric_custom_op [-1.62225f, 1.06592f] 30
/decoder/layers.2/Add_1 add [-115.403f, 10.153f] 24
/decoder/layers.2/final_layer_norm/Add_1 nn.layer_norm [-27.5384f, 13.6033f] 26
CustomOp/linalg::matMulNBits<31, 31, 28>24 contrib.epu.quadric_custom_op [-7.36775f, 2.62154f] 28
/decoder/layers.2/activation_fn/Mul_1 contrib.epu.gelu [-0.169971f, 2.61007f] 29
CustomOp/linalg::matMulNBits<31, 31, 29>25 contrib.epu.quadric_custom_op [-3.5966f, 1.07429f] 29
/decoder/layers.2/Add_2 add [-115.526f, 9.63642f] 24
/decoder/layers.3/self_attn_layer_norm/Add_1 nn.layer_norm [-15.502f, 13.7208f] 27
CustomOp/linalg::matMulNBits<31, 31, 28>26 contrib.epu.quadric_custom_op [-7.84249f, 6.81374f] 28
CustomOp/nn::whisperAttention<384, 6, 4, 14>39 contrib.epu.quadric_custom_op [-2.42135f, 1.45664f] 29
CustomOp/linalg::matMulNBits<31, 31, 27>27 contrib.epu.quadric_custom_op [-2.11393f, 3.09569f] 29
/decoder/layers.3/Add add [-115.514f, 9.92158f] 24
/decoder/layers.3/encoder_attn_layer_norm/Add_1 nn.layer_norm [-26.1919f, 9.08276f] 26
CustomOp/linalg::matMulNBits<31, 31, 28>28 contrib.epu.quadric_custom_op [-8.86917f, 9.42494f] 27
CustomOp/linalg::matMulNBits<31, 31, 31>6 contrib.epu.quadric_custom_op [-8.37619f, 7.85888f] 27
CustomOp/linalg::matMulNBits<31, 31, 31>7 contrib.epu.quadric_custom_op [-5.48935f, 6.18549f] 28
CustomOp/nn::whisperAttention<384, 6, 1500, 14>40 contrib.epu.quadric_custom_op [-2.70527f, 3.26581f] 29
CustomOp/linalg::matMulNBits<31, 31, 27>29 contrib.epu.quadric_custom_op [-4.75347f, 1.91847f] 28
/decoder/layers.3/Add_1 add [-115.245f, 10.3928f] 24
/decoder/layers.3/final_layer_norm/Add_1 nn.layer_norm [-46.1616f, 11.8378f] 25
CustomOp/linalg::matMulNBits<31, 31, 28>30 contrib.epu.quadric_custom_op [-17.4479f, 17.3023f] 26
/decoder/layers.3/activation_fn/Mul_1 contrib.epu.gelu [-0.169971f, 17.3023f] 26
CustomOp/linalg::matMulNBits<31, 31, 29>31 contrib.epu.quadric_custom_op [-8.86752f, 11.8442f] 27
/decoder/layers.3/Add_2 add [-110.492f, 8.81477f] 24
/decoder/layer_norm/Add_1 nn.layer_norm [-239.063f, 80.9414f] 23
CustomOp/linalg::matMulNBits<31, 31, 31>32 contrib.epu.quadric_custom_op [-21.6856f, 33.7606f] 25
2026-06-19 04:27 - INFO - epu - codegen - START====================build_cpu_runnable_fx_relay
2026-06-19 04:27 - INFO - epu - codegen - START=======================quantize_to_chimera_fx
2026-06-19 04:27 - INFO - epu - codegen - START=================================relay_to_tir
2026-06-19 04:27 - INFO - epu - codegen - START===========================relay_to_epu_relay
2026-06-19 04:27 - INFO - epu - codegen - START==============================adapt_and_order
2026-06-19 04:27 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-06-19 04:27 - INFO - epu - codegen - START=============================plan_lrm_virtual
2026-06-19 04:27 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-06-19 04:27 - INFO - epu - codegen - START===============================lrm_alloc_loop
2026-06-19 04:28 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-06-19 04:28 - INFO - epu - codegen - START================================lrm_splitting
2026-06-19 04:28 - INFO - epu - codegen - START==============================ext_split_relay
2026-06-19 04:28 - INFO - epu - codegen - File copied from /quadric/sdk-cli/examples/models/whisper/whisper_matmul_stubs.hpp to /quadric/sdk-cli/examples/models/whisper/ccl_build/decoder_model_matched_QC_P_1d7_32MB_4kB_128GBps_128GBps_16_OFF_x1_x1/attention_stubs.hpp
2026-06-19 04:28 - INFO - epu - codegen - START====================================build_tir
2026-06-19 04:28 - INFO - epu - chimera_job - Compilation of decoder_model_matched_QC_P_1d7_32MB_4kB_128GBps_128GBps_16_OFF_x1_x1 successful
5.3 Build, Run on ISS, and Compare
whisper_decoder.cpp consumes the encoder's GPNPU output (the ISS result) rather than the floating-point ONNX Runtime reference. The accuracy comparison is therefore end-to-end on the GPNPU pipeline.
The wrapper expects two input binaries on disk:
| Bin | Contents |
|---|---|
input_ids.bin (16 B) | 4 bootstrap tokens |
encoder_hidden_states.bin (~2.3 MB) | Encoder ISS output |
The next cell computes ORT-reference logits, reads the GPNPU output, and checks two conditions:
| Check | Threshold |
|---|---|
| Top-1 predicted token matches the ORT reference (LibriSpeech sample 0) | strict equality |
| PSNR of the GPNPU logits vs ORT logits | ≥ 20 dB |
Reference accuracy (single-core baseline)
| Stage | PSNR |
|---|---|
| Single attention-layer kernel only | ~70.3 dB |
| 4-layer decoder, fed ORT encoder output | ~48.8 dB |
| 4-layer decoder, fed ISS encoder output (end-to-end) | ~24 dB |
## input_ids.bin + encoder_hidden_states.bin (FxP26) for whisper_decoder.cpp.
decoder_input_ids.astype(np.int32).tofile("input_ids.bin")
np.round(encoder_outputs.astype(np.float64) * (1 << 26)).astype(np.int32).tofile(
"encoder_hidden_states.bin"
)
%%bash
set -euo pipefail
sdk source -v whisper_decoder.cpp \
--include-cgc-headers \
--target QC-P \
--num-cores 4 \
--ocm-size 32MB \
--macs-per-pe 16 \
--clock-freq-ghz 1.7 \
--ext-read-bw 128GBps \
--ext-write-bw 128GBps \
--quiet
2026-06-19 04:28 - DEBUG - sdk - cli - Executing command: cmake CMakeLists.txt -B /quadric/sdk-cli/examples/models/whisper/whisper_decoder_QC-P_32MB_4kB_128GBps_128GBps/build -DNUM_GPNPUS=4 -DNUM_CORES=16 -DNUM_BORDERS=2 -DEPU_VERSION=2.0.0 -DQLLVM_ROOT_PATH=/quadric/llvm -DOCM_SIZE_KIBIBYTES=32768 -DNUM_PE_MACS=16 -DASSERT_MLS_WIDTH_LINE_ALIGN=ON -DHARDWARE_TARGET=OFF
2026-06-19 04:28 - DEBUG - sdk - cli - Executing command: make -j8
[SDK-CLI] : Executing on QC-P simulator
2026-06-19 04:29 - DEBUG - sdk - cli - Executing command: ./whisper_decoder_host -c --ddrRdBwTotal 1048576.0 --ddrWrBwTotal 1048576.0 --ddrAxiWidth 128 --instMemDepth 1310720 --ocmSize 33554432 --cycleTimeNS 0.5882352941176471 --no-check --ddrRdAvgPct 100 --ddrRdMaxPct 100 --ddrWrAvgPct 100 --ddrWrMaxPct 100 --postKernelFlowTimeoutCycles 4000000 --clusterSize 4 --numClusters 1
2026-06-19 04:32 - WARNING - epu - core - No profile.json found. Not able to show performance results.
2026-06-19 04:32 - INFO - epu - core - If you would like to see performance results, add profiling statements to your source code.
[SDK-CLI] : Execution completed.
ort_decoder_output = decoder_session.run(
None,
{"input_ids": decoder_input_ids.astype(np.int64), "encoder_hidden_states": encoder_outputs},
)[0].astype(np.float32)
decoder_iss_output = (
(np.fromfile("logits.bin", dtype=np.int32).astype(np.float64) / float(1 << 25))
.astype(np.float32)
.reshape(ort_decoder_output.shape)
)
abs_diff = np.abs(ort_decoder_output - decoder_iss_output)
mse = float(np.mean((ort_decoder_output - decoder_iss_output) ** 2))
mae = float(np.mean(abs_diff))
max_diff = float(np.max(abs_diff))
signal_range = float(ort_decoder_output.max() - ort_decoder_output.min())
psnr = 10.0 * np.log10(signal_range**2 / mse) if mse > 0 else float("inf")
print("Decoder ORT vs ISS:")
print(f" MAE: {mae:.6f} PSNR: {psnr:.2f} dB Max diff: {max_diff:.6f}")
results["decoder"].update({"mse": mse, "mae": mae, "max_diff": max_diff, "psnr": psnr})
dec_iss_top5 = np.argsort(-decoder_iss_output[0, -1, :])[:5]
dec_ref_top5 = np.argsort(-ort_decoder_output[0, -1, :])[:5]
print(f" ISS top-5: {dec_iss_top5.tolist()}")
print(f" ORT top-5: {dec_ref_top5.tolist()}")
print(f" Top-5 set match: {set(dec_iss_top5.tolist()) == set(dec_ref_top5.tolist())}")
DECODER_PSNR_MIN_DB = 20.0
assert (
psnr >= DECODER_PSNR_MIN_DB
), f"Decoder PSNR regression: {psnr:.2f} dB < {DECODER_PSNR_MIN_DB} dB"
## MSE on logits — useful because logits' signal range varies sample-to-sample,
## so a fixed PSNR threshold can drift; an MSE bound on absolute logit error is
## more directly tied to top-k stability.
DECODER_MSE_MAX = 10 # ~3-4× headroom over current MSE; catches catastrophic drift
assert mse <= DECODER_MSE_MAX, f"Decoder MSE regression: {mse:.4f} > {DECODER_MSE_MAX}"
Decoder ORT vs ISS:
MAE: 0.204787 PSNR: 45.45 dB Max diff: 0.583006
ISS top-5: [2221, 440, 503, 13094, 639]
ORT top-5: [2221, 440, 503, 13094, 639]
Top-5 set match: True
Summary
| Model | Whisper-Tiny (39 M parameters, 4 encoder + 4 decoder layers, embed_dim 384) |
| Pipeline | (1) Mel spectrogram extraction — host CPU (transforms 30 s of 16 kHz audio into [1, 80, 3000] log-mel features)(2) Conv front-end — host CPU (ORT) ( Conv1 + GELU + Conv2 + GELU + Transpose + PositionalEmbedding produces [1, 1500, 384] post-conv features)(3) Encoder, 4 layers — Chimera GPNPU (ISS) (self-attention + FFN per layer, INT4 weights, FP16 attention; outputs [1, 1500, 384] hidden states)(4) Decoder prefill, 4 layers — Chimera GPNPU (ISS) (self-attention + cross-attention over encoder hidden states + FFN per layer; outputs [1, 4, 51865] logits for the next-token prediction) |
| Target | QC-P, 32 MB OCM, 16 MACs/PE, 4 cores |
| Quantization | INT4 weights via MatMulNBits, FP16 self- and cross-attention |
| Custom Ops | linalg::matMulNBits, nn::whisperAttention (self- and cross-attention variants) |
| End-to-end accuracy | Top-1 token match against ORT reference on the validation sample, PSNR ≥ 20 dB |
The same wrapper builds on 1, 2, or 4 cores via sdk source --num-cores N, with no code change required to scale.
