NOTE: The Jupyter Notebook below is included in the Chimera SDK and can be run interactively by running the following CLI command:
$ quadric sdk notebook
From the Jupyter Notebook window in your browser, select the notebook named /quadric/sdk-cli/examples/models/qwen/qwen3_8b/qwen3_8b.ipynb.
Qwen3-8B on Chimera GPNPU
This notebook compiles Qwen3-8B with CGC (Chimera Graph Compiler) and runs it on the ISS (Instruction Set Simulator) in a 4-core QC-P configuration — both halves of LLM inference, from one W4A8 checkpoint:
- Decode — the autoregressive loop: one token in, one token out, the KV cache carried on-chip between steps. Roughly 18–19 tokens/s average (20–21 peak) at 1 GHz.
- Prefill — the whole 1024-token prompt through all 36 decoder layers in one pass: one layer validated on the ISS against ONNX Runtime, then the complete prefill program compiled through CGC.
Why one notebook for decode and prefill?
An LLM serves a prompt in two phases that share everything except their shapes. Prefill processes the full prompt at once (sequence_length = 1024, empty cache) to fill the KV cache; decode then produces one token per pass (sequence_length = 1, past_sequence_length = 1023) against that cache. The quantized weights, the tensor ranges and the custom operators are identical — only the shape-fixing step and the host runner differ. Keeping both phases together shows how far a single quantized export carries.
Pipeline
Model: Qwen3-8B (8.2B parameters, 36 decoder layers, hidden size 4096, 32 query / 8 KV heads, 151,936-token vocabulary), W4A8 SmoothQuant (α = 0.36), 1024-token context
| Phase | Graph shape | On the ISS | Sections |
|---|---|---|---|
| Decode | 1 token in, 1023 cached | The full autoregressive loop on 4× QC-P, checked against ONNX Runtime | 3–9 |
| Prefill | 1024 tokens in, empty cache | One decoder layer, validated against ONNX Runtime; all 36 compiled | 10–11 |
1. Setup
Imports, the model's architecture constants and the hardware target. The QC-P configuration below is shared by every compile and ISS run in this notebook: 4 cores, 2 MB OCM (On-Chip Memory) per core, 16 MACs per PE, 1 GHz, 24 GB/s of DDR bandwidth per core over a 256-bit AXI interface. CGC needs a deep stack to schedule an 8-billion-parameter graph, so the limit is raised once here.
import re
from pathlib import Path
import numpy as np
from IPython.display import Image, display
from tvm.contrib.epu.chimera_job.chimera_job import ChimeraJob
from tvm.contrib.epu.chimera_job.hw_config import HWConfig
from examples.models.qwen.qwen3_8b.prefill.custom_op_match import (
qwen3_custom_op_replacer as qwen3_prefill_custom_op_replacer,
)
from examples.models.qwen.qwen3_8b.prefill.run_decoder_pipeline import run_pipeline
from examples.models.qwen.qwen_helpers import (
compare_decode_with_ort,
kv_cache_io_names,
plot_decode_profile,
raise_stack_limit,
report_decode_throughput,
stage_const_tensor,
)
from custom_op_match import qwen3_custom_op_replacer
from fix_shapes import fix_shapes
from qwen3_8b_helpers import (
download_decode_model,
download_prefill_model,
prepare_prefill_prompt,
project_prefill_latency,
)
%matplotlib inline
NUM_DECODERS = 36
EMBED_DIM = 4096
NUM_KV_HEADS = 8 # 32 query heads share 8 key/value heads (grouped-query attention)
SEQ_LEN = 1024 # context window: prompt + generated tokens for decode, prompt length for prefill
NEW_TOKENS = 1
HW_CONFIG = HWConfig(product="QC-P", ocm_size="2MB", macs_per_pe=16)
PREFILL_DIR = Path("prefill") # prefill graph, tokenized prompt and per-decoder artifacts live here
raise_stack_limit()
Stack limit raised to 32 MB for CGC compilation
2. Download the W4A8 Checkpoint
Qwen3-8B was quantized offline to W4A8 — 4-bit per-channel weights, 8-bit activations — with SmoothQuant (α = 0.36) migrating activation outliers into the weights before quantization. The export also lifts the RoPE sin/cos tables out as graph inputs, so the runner supplies the rotary angle for each position, and carries an attention mask shaped for both the prefill and the decode stage. Three files make up the decode graph:
| File | Contents |
|---|---|
qwen_0_36.onnx | The quantized graph with dynamic batch and sequence dimensions |
9b9822ba-….data | External weight data (~6 GB) |
qwen_0_36.onnx.tranges | Per-tensor value ranges that parameterize the fixed-point custom ops |
decode_onnx, _decode_weights, tranges_path = map(str, download_decode_model())
Downloading qwen_0_36.onnx...
Downloaded qwen_0_36.onnx (0.02 GB)
Downloading 9b9822ba-a73f-11f0-9995-10ffe060dbe7.data...
Downloaded 9b9822ba-a73f-11f0-9995-10ffe060dbe7.data (6.27 GB)
Downloading qwen_0_36.onnx.tranges...
Downloaded qwen_0_36.onnx.tranges (0.00 GB)
3. Fix Shapes for Autoregressive Decode
CGC compiles static graphs. The downloaded model still has symbolic batch_size, sequence_length and past_sequence_length dimensions, so fix_shapes pins them for the decode step — one new token attending to a 1023-entry KV cache — and runs ONNX Runtime's quantization pre-processing (shape inference and constant folding) so every tensor has a known shape before custom-op matching. The same helper produces the prefill layout with mode="prefill" (1024 tokens in, empty cache); section 10 downloads that graph pre-built.
Runtime: ~1 minute
decode_seq_onnx = "qwen3_seq1024.onnx"
fix_shapes(decode_onnx, decode_seq_onnx, seq_len=SEQ_LEN, mode="autoregressive")
Loading model from qwen_0_36.onnx...
Fixing shapes for autoregressive mode: seq_len=1024
Saving intermediate fixed model to fixed.onnx...
Running quantization pre-processing...
Shape fixing complete! Output saved to qwen3_seq1024.onnx
4. Replace Attention, Projections and the LM Head with Custom Ops
CGC compiles most of the graph on its own. Three patterns are swapped for CCL (Chimera Compute Language) kernels from the SDK's neural-network block library, because a fused kernel keeps the whole operation inside OCM and reads each weight exactly once per token:
| ONNX subgraph | Custom op | What it buys |
|---|---|---|
| Q/K/V projections → QK-RMSNorm → RoPE → grouped-query attention over the KV cache | nn::qwen3Attention | One kernel per layer appends to the cache and computes attention in OCM |
| Gate and up projections + SiLU | nn::gateProj | The SwiGLU pair in a single pass over the hidden state |
| Every INT4-weight MatMul, including the 4096 × 151,936 LM head | linalg::channelwiseQuantMatMul | INT8 activations against INT4 weights packed eight per 32-bit word (v8i4), per-channel scales |
qwen3_custom_op_replacer pattern-matches these subgraphs across all 36 layers, turns the tensor ranges into the fixed-point fractional bits each kernel expects as template arguments, and packs the INT4 weights — the packing is what takes the time.
Runtime: ~30–45 minutes
decode_custom_onnx = "qwen3_custom_ops_seq1024.onnx"
qwen3_custom_op_replacer(
decode_seq_onnx,
decode_custom_onnx,
tranges_path,
num_heads=NUM_KV_HEADS,
embed_dim=EMBED_DIM,
seq_length=SEQ_LEN,
num_decoders=NUM_DECODERS,
)
Successfully saved modified model to qwen3_custom_ops_seq1024.onnx
The weight packing is the one piece worth seeing in full. Eight signed 4-bit values are stacked along the input dimension into one int32, so a [K, N] weight becomes [K/8, N] words that the kernel unpacks on the fly — the same layout the v8i4 vector type uses on the GPNPU:
source = Path("custom_op_match.py").read_text()
print(re.search(r"def pack_row_v8i4.*?\n return output\n", source, re.DOTALL).group(0))
def pack_row_v8i4(input_array):
"""Pack 8 rows of 4-bit values into 32-bit integers.
Parameters
----------
input_array : numpy.ndarray
The input 2D array with shape (height, width) to pack.
Returns
-------
numpy.ndarray
The packed array with shape (ceil(height/8), width).
"""
pack_factor = 8
height, width = input_array.shape
output_height = math.ceil(height / pack_factor)
shift_granularity = 32 // pack_factor
mask = (1 << shift_granularity) - 1
# Pad input if height is not divisible by pack_factor
if height % pack_factor != 0:
padding = pack_factor - (height % pack_factor)
input_array = np.pad(
input_array, ((0, padding), (0, 0)), mode="constant", constant_values=0
)
# Reshape to group rows by pack_factor
reshaped = input_array.reshape(output_height, pack_factor, width)
# Convert to int32 and mask to get unsigned values
unsigned_vals = reshaped.astype(np.int32) & mask
# Create shift amounts for each position in the pack
shifts = np.arange(pack_factor) * shift_granularity
shifts = shifts.reshape(pack_factor, 1)
# Apply shifts to each group
shifted = unsigned_vals * (1 << shifts)
# Combine all values in each group using bitwise OR (sum works since non-overlapping bits)
output = shifted.sum(axis=1, dtype=np.int32)
return output
5. Compile the Decode Graph with CGC
CGC turns the custom-op graph into C++ for the QC-P target: it schedules the 36 layers, tiles every tensor to fit the 2 MB OCM, and emits the kernel plus a const_tensor_data.bin holding the packed weights. The KV cache tensors and the attention mask are listed in io_to_ignore — the autoregressive runner in section 7 owns them across token steps, so CGC leaves them out of the kernel's I/O.
Memory: ~55 GB peak · Runtime: ~25 minutes
decode_job = ChimeraJob(
decode_custom_onnx,
hw_config=HW_CONFIG,
trange_file=tranges_path,
target_lang="QIL", # Quadric Intermediate Language
io_to_ignore=kv_cache_io_names(NUM_DECODERS),
)
decode_job.compile()
print(decode_job)
2026-09-22 03:27 - INFO - epu - chimera_job - START==================================onnx_ingest
2026-09-22 03:27 - INFO - epu - chimera_job - Numerical ranges provided
/usr/local/lib/python3.10/dist-packages/tvm/relay/frontend/onnx.py:6272: UserWarning: This protobuf of onnx model is too large (>2GB). Call check_model with model path instead.
warnings.warn(str(e))
2026-09-22 03:28 - INFO - epu - codegen - START===============================optimize_relay
2026-09-22 03:28 - INFO - epu - codegen - START====================quantize_to_cpu_runnable_fx
2026-09-22 03:29 - INFO - epu - fx -
Source name Op Output 0 Range Output 0 Frac Bits
--------------------------------------------------- ----------------------------- ----------------------- --------------------
/model/embed_tokens/Gather contrib.epu.embedding [-0.628906f, 0.8125f] 31
/model/layers.0/input_layernorm/Mul_1 contrib.epu.rms_norm [-0.359118f, 0.37561f] 31
/model/layers.0/self_attn/q_proj/MatMul_smooth_mul multiply [-0.287793f, 0.307279f] 31
/model/layers.0/self_attn/k_proj/MatMul_smooth_mul multiply [-0.247679f, 0.223768f] 31
/model/layers.0/self_attn/v_proj/MatMul_smooth_mul multiply [-0.100749f, 0.107016f] 31
CustomOp/linalg::channelwiseQuantMatMul<43>3 contrib.epu.quadric_custom_op [-2.25614f, 3.85482f] 29
/model/layers.0/Add add [-2.30253f, 4.25466f] 28
/model/layers.0/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-1.23024f, 1.59211f] 29
/model/layers.0/mlp/gate_proj/MatMul_smooth_mul multiply [-0.682836f, 0.548619f] 28
CustomOp/nn::gateProj<41, 29>1 contrib.epu.quadric_custom_op [-0.278465f, 3.85478f] 29
/model/layers.0/mlp/up_proj/MatMul_smooth_mul multiply [-0.421256f, 0.474667f] 29
CustomOp/linalg::channelwiseQuantMatMul<42>0 contrib.epu.quadric_custom_op [-3.34433f, 3.38188f] 29
/model/layers.0/mlp/Mul multiply [-11.3828f, 9.95375f] 27
/model/layers.0/mlp/down_proj/MatMul_smooth_mul multiply [-2.19349f, 2.29969f] 26
CustomOp/linalg::channelwiseQuantMatMul<38>2 contrib.epu.quadric_custom_op [-5.18762f, 16.7897f] 26
/model/layers.0/Add_1 add [-6.67269f, 19.2419f] 26
/model/layers.1/input_layernorm/Mul_1 contrib.epu.rms_norm [-1.30106f, 1.14032f] 30
/model/layers.1/self_attn/q_proj/MatMul_smooth_mul multiply [-0.549389f, 0.618498f] 28
/model/layers.1/self_attn/k_proj/MatMul_smooth_mul multiply [-0.547089f, 0.290293f] 28
/model/layers.1/self_attn/v_proj/MatMul_smooth_mul multiply [-0.18635f, 0.285787f] 28
CustomOp/linalg::channelwiseQuantMatMul<43>7 contrib.epu.quadric_custom_op [-1.41003f, 1.53298f] 30
/model/layers.1/Add add [-6.78994f, 19.2634f] 26
/model/layers.1/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-51.933f, 47.6116f] 25
/model/layers.1/mlp/gate_proj/MatMul_smooth_mul multiply [-3.17403f, 3.52375f] 22
CustomOp/nn::gateProj<38, 27>5 contrib.epu.quadric_custom_op [-0.278465f, 7.77009f] 28
/model/layers.1/mlp/up_proj/MatMul_smooth_mul multiply [-2.03129f, 2.22485f] 22
CustomOp/linalg::channelwiseQuantMatMul<39>4 contrib.epu.quadric_custom_op [-13.6112f, 16.4607f] 26
/model/layers.1/mlp/Mul multiply [-22.4947f, 19.6069f] 23
/model/layers.1/mlp/down_proj/MatMul_smooth_mul multiply [-3.95853f, 6.61841f] 24
CustomOp/linalg::channelwiseQuantMatMul<37>6 contrib.epu.quadric_custom_op [-12.4679f, 38.1379f] 25
/model/layers.1/Add_1 add [-18.1108f, 57.4013f] 25
/model/layers.2/input_layernorm/Mul_1 contrib.epu.rms_norm [-0.862898f, 1.28555f] 29
/model/layers.2/self_attn/q_proj/MatMul_smooth_mul multiply [-0.369643f, 0.667969f] 28
/model/layers.2/self_attn/k_proj/MatMul_smooth_mul multiply [-0.560969f, 0.593967f] 28
/model/layers.2/self_attn/v_proj/MatMul_smooth_mul multiply [-0.209467f, 0.313265f] 28
CustomOp/linalg::channelwiseQuantMatMul<42>11 contrib.epu.quadric_custom_op [-1.03252f, 1.31911f] 30
/model/layers.2/Add add [-18.1085f, 57.4384f] 25
/model/layers.2/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-82.5232f, 81.3135f] 24
/model/layers.2/mlp/gate_proj/MatMul_smooth_mul multiply [-12.8995f, 14.1477f] 25
CustomOp/nn::gateProj<35, 24>9 contrib.epu.quadric_custom_op [-0.278465f, 7.80494f] 28
/model/layers.2/mlp/up_proj/MatMul_smooth_mul multiply [-12.5991f, 14.1477f] 25
CustomOp/linalg::channelwiseQuantMatMul<35>8 contrib.epu.quadric_custom_op [-122.194f, 103.963f] 24
/model/layers.2/mlp/Mul multiply [-26.306f, 24.5606f] 21
/model/layers.2/mlp/down_proj/MatMul_smooth_mul multiply [-5.16373f, 2.29959f] 22
CustomOp/linalg::channelwiseQuantMatMul<37>10 contrib.epu.quadric_custom_op [-10.8904f, 26.5761f] 26
/model/layers.2/Add_1 add [-23.2706f, 81.9635f] 24
/model/layers.3/input_layernorm/Mul_1 contrib.epu.rms_norm [-1.24747f, 3.00854f] 28
/model/layers.3/self_attn/q_proj/MatMul_smooth_mul multiply [-0.43238f, 0.730476f] 27
/model/layers.3/self_attn/k_proj/MatMul_smooth_mul multiply [-0.371823f, 1.19244f] 27
/model/layers.3/self_attn/v_proj/MatMul_smooth_mul multiply [-0.246321f, 0.362407f] 27
CustomOp/linalg::channelwiseQuantMatMul<42>15 contrib.epu.quadric_custom_op [-1.84038f, 1.97427f] 30
/model/layers.3/Add add [-23.1594f, 81.7177f] 24
/model/layers.3/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-53.3995f, 62.6125f] 25
/model/layers.3/mlp/gate_proj/MatMul_smooth_mul multiply [-3.54248f, 4.5413f] 22
CustomOp/nn::gateProj<37, 26>13 contrib.epu.quadric_custom_op [-0.278465f, 8.11204f] 27
/model/layers.3/mlp/up_proj/MatMul_smooth_mul multiply [-2.58039f, 4.31592f] 22
CustomOp/linalg::channelwiseQuantMatMul<37>12 contrib.epu.quadric_custom_op [-17.4614f, 10.6391f] 26
/model/layers.3/mlp/Mul multiply [-23.0629f, 23.7062f] 23
/model/layers.3/mlp/down_proj/MatMul_smooth_mul multiply [-3.35231f, 3.48879f] 25
CustomOp/linalg::channelwiseQuantMatMul<39>14 contrib.epu.quadric_custom_op [-7.28584f, 7.4731f] 28
/model/layers.3/Add_1 add [-23.3718f, 85.3067f] 24
/model/layers.4/input_layernorm/Mul_1 contrib.epu.rms_norm [-1.80589f, 2.00268f] 29
/model/layers.4/self_attn/q_proj/MatMul_smooth_mul multiply [-0.898383f, 0.659137f] 28
/model/layers.4/self_attn/k_proj/MatMul_smooth_mul multiply [-0.707868f, 1.08651f] 28
/model/layers.4/self_attn/v_proj/MatMul_smooth_mul multiply [-0.439939f, 0.234896f] 28
CustomOp/linalg::channelwiseQuantMatMul<42>19 contrib.epu.quadric_custom_op [-3.53964f, 2.62418f] 29
/model/layers.4/Add add [-23.2038f, 85.5677f] 24
/model/layers.4/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-32.953f, 41.0647f] 25
/model/layers.4/mlp/gate_proj/MatMul_smooth_mul multiply [-3.60791f, 4.06511f] 25
CustomOp/nn::gateProj<37, 27>17 contrib.epu.quadric_custom_op [-0.278465f, 9.32974f] 27
/model/layers.4/mlp/up_proj/MatMul_smooth_mul multiply [-2.50043f, 4.23358f] 25
CustomOp/linalg::channelwiseQuantMatMul<37>16 contrib.epu.quadric_custom_op [-8.54877f, 10.1004f] 27
/model/layers.4/mlp/Mul multiply [-20.2387f, 20.039f] 24
/model/layers.4/mlp/down_proj/MatMul_smooth_mul multiply [-2.69153f, 2.8874f] 26
CustomOp/linalg::channelwiseQuantMatMul<39>18 contrib.epu.quadric_custom_op [-6.69227f, 12.1684f] 27
/model/layers.4/Add_1 add [-22.4392f, 87.0669f] 24
/model/layers.5/input_layernorm/Mul_1 contrib.epu.rms_norm [-2.14681f, 3.19879f] 28
/model/layers.5/self_attn/q_proj/MatMul_smooth_mul multiply [-1.45607f, 0.922822f] 28
/model/layers.5/self_attn/k_proj/MatMul_smooth_mul multiply [-1.13808f, 1.1609f] 28
/model/layers.5/self_attn/v_proj/MatMul_smooth_mul multiply [-0.502765f, 0.449179f] 28
CustomOp/linalg::channelwiseQuantMatMul<42>23 contrib.epu.quadric_custom_op [-6.97809f, 4.76181f] 28
/model/layers.5/Add add [-21.9713f, 84.54f] 24
/model/layers.5/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-24.6416f, 19.9332f] 26
/model/layers.5/mlp/gate_proj/MatMul_smooth_mul multiply [-3.30293f, 4.25595f] 25
CustomOp/nn::gateProj<37, 27>21 contrib.epu.quadric_custom_op [-0.278465f, 7.52237f] 28
/model/layers.5/mlp/up_proj/MatMul_smooth_mul multiply [-1.51029f, 2.94375f] 25
CustomOp/linalg::channelwiseQuantMatMul<38>20 contrib.epu.quadric_custom_op [-6.38602f, 7.07606f] 28
/model/layers.5/mlp/Mul multiply [-18.2442f, 18.1234f] 25
/model/layers.5/mlp/down_proj/MatMul_smooth_mul multiply [-3.21112f, 2.15665f] 26
CustomOp/linalg::channelwiseQuantMatMul<38>22 contrib.epu.quadric_custom_op [-37.2974f, 18.5171f] 25
/model/layers.5/Add_1 add [-18.6574f, 63.1892f] 24
/model/layers.6/input_layernorm/Mul_1 contrib.epu.rms_norm [-1.99346f, 7.23117f] 27
/model/layers.6/self_attn/q_proj/MatMul_smooth_mul multiply [-0.737696f, 4.47784f] 27
/model/layers.6/self_attn/k_proj/MatMul_smooth_mul multiply [-0.58721f, 1.42655f] 27
/model/layers.6/self_attn/v_proj/MatMul_smooth_mul multiply [-0.422668f, 1.02092f] 27
CustomOp/linalg::channelwiseQuantMatMul<40>27 contrib.epu.quadric_custom_op [-31.7815f, 8.99221f] 26
/model/layers.6/Add add [-12.6824f, 38.3116f] 24
/model/layers.6/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-9.84406f, 199.177f] 23
/model/layers.6/mlp/gate_proj/MatMul_smooth_mul multiply [-0.969524f, 14.0399f] 22
CustomOp/nn::gateProj<34, 24>25 contrib.epu.quadric_custom_op [-0.278465f, 65.6915f] 24
/model/layers.6/mlp/up_proj/MatMul_smooth_mul multiply [-0.875245f, 14.5852f] 22
CustomOp/linalg::channelwiseQuantMatMul<34>24 contrib.epu.quadric_custom_op [-41.8197f, 74.1344f] 24
/model/layers.6/mlp/Mul multiply [-1282.21f, 4841.01f] 18
/model/layers.6/mlp/down_proj/MatMul_smooth_mul multiply [-112.681f, 254.839f] 18
CustomOp/linalg::channelwiseQuantMatMul<27>26 contrib.epu.quadric_custom_op [-2118.69f, 9638.07f] 17
/model/layers.6/Add_1 add [-2121.44f, 9654.1f] 17
/model/layers.7/input_layernorm/Mul_1 contrib.epu.rms_norm [-5.1855f, 10.441f] 27
/model/layers.7/self_attn/q_proj/MatMul_smooth_mul multiply [-1.18851f, 2.07018f] 27
/model/layers.7/self_attn/k_proj/MatMul_smooth_mul multiply [-1.36347f, 1.4916f] 26
/model/layers.7/self_attn/v_proj/MatMul_smooth_mul multiply [-0.655511f, 1.0774f] 27
CustomOp/linalg::channelwiseQuantMatMul<41>31 contrib.epu.quadric_custom_op [-4.27446f, 3.6406f] 28
/model/layers.7/Add add [-2121.23f, 9654.21f] 17
/model/layers.7/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-6.89828f, 54.4274f] 25
/model/layers.7/mlp/gate_proj/MatMul_smooth_mul multiply [-1.00739f, 4.84292f] 24
CustomOp/nn::gateProj<38, 27>29 contrib.epu.quadric_custom_op [-0.278465f, 11.2674f] 27
/model/layers.7/mlp/up_proj/MatMul_smooth_mul multiply [-1.13376f, 1.88627f] 24
CustomOp/linalg::channelwiseQuantMatMul<40>28 contrib.epu.quadric_custom_op [-13.397f, 10.9477f] 27
/model/layers.7/mlp/Mul multiply [-16.9021f, 17.2346f] 23
/model/layers.7/mlp/down_proj/MatMul_smooth_mul multiply [-2.6115f, 2.85751f] 26
CustomOp/linalg::channelwiseQuantMatMul<37>30 contrib.epu.quadric_custom_op [-6.76926f, 12.1386f] 27
/model/layers.7/Add_1 add [-2121.18f, 9654.34f] 17
/model/layers.8/input_layernorm/Mul_1 contrib.epu.rms_norm [-4.59068f, 9.9222f] 27
/model/layers.8/self_attn/q_proj/MatMul_smooth_mul multiply [-1.28958f, 1.87046f] 27
/model/layers.8/self_attn/k_proj/MatMul_smooth_mul multiply [-0.814643f, 1.79062f] 27
/model/layers.8/self_attn/v_proj/MatMul_smooth_mul multiply [-0.422026f, 0.896034f] 27
CustomOp/linalg::channelwiseQuantMatMul<41>35 contrib.epu.quadric_custom_op [-3.77026f, 7.3769f] 28
/model/layers.8/Add add [-2120.87f, 9653.89f] 17
/model/layers.8/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-7.8127f, 6.98587f] 28
/model/layers.8/mlp/gate_proj/MatMul_smooth_mul multiply [-1.26424f, 1.58606f] 27
CustomOp/nn::gateProj<39, 27>33 contrib.epu.quadric_custom_op [-0.278465f, 6.3098f] 28
/model/layers.8/mlp/up_proj/MatMul_smooth_mul multiply [-1.4333f, 1.11036f] 27
CustomOp/linalg::channelwiseQuantMatMul<40>32 contrib.epu.quadric_custom_op [-6.76552f, 5.97511f] 28
/model/layers.8/mlp/Mul multiply [-15.8356f, 18.0968f] 25
/model/layers.8/mlp/down_proj/MatMul_smooth_mul multiply [-2.78291f, 3.87402f] 27
CustomOp/linalg::channelwiseQuantMatMul<38>34 contrib.epu.quadric_custom_op [-10.3458f, 12.5126f] 27
/model/layers.8/Add_1 add [-2120.62f, 9654.73f] 17
/model/layers.9/input_layernorm/Mul_1 contrib.epu.rms_norm [-4.52299f, 10.3721f] 27
/model/layers.9/self_attn/q_proj/MatMul_smooth_mul multiply [-1.02019f, 1.58785f] 26
/model/layers.9/self_attn/k_proj/MatMul_smooth_mul multiply [-1.23027f, 1.70964f] 26
/model/layers.9/self_attn/v_proj/MatMul_smooth_mul multiply [-0.594754f, 0.902843f] 26
CustomOp/linalg::channelwiseQuantMatMul<39>39 contrib.epu.quadric_custom_op [-7.31632f, 7.46158f] 28
/model/layers.9/Add add [-2120.34f, 9654.1f] 17
/model/layers.9/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-8.45564f, 8.75462f] 27
/model/layers.9/mlp/gate_proj/MatMul_smooth_mul multiply [-1.24338f, 1.8245f] 27
CustomOp/nn::gateProj<39, 27>37 contrib.epu.quadric_custom_op [-0.278465f, 9.10514f] 27
/model/layers.9/mlp/up_proj/MatMul_smooth_mul multiply [-1.7809f, 1.67158f] 27
CustomOp/linalg::channelwiseQuantMatMul<39>36 contrib.epu.quadric_custom_op [-7.51536f, 6.51499f] 28
/model/layers.9/mlp/Mul multiply [-15.1261f, 14.7165f] 24
/model/layers.9/mlp/down_proj/MatMul_smooth_mul multiply [-2.19853f, 3.02141f] 27
CustomOp/linalg::channelwiseQuantMatMul<38>38 contrib.epu.quadric_custom_op [-9.86182f, 8.2895f] 27
/model/layers.9/Add_1 add [-2120.34f, 9656.53f] 17
/model/layers.10/input_layernorm/Mul_1 contrib.epu.rms_norm [-7.60531f, 17.5566f] 26
/model/layers.10/self_attn/q_proj/MatMul_smooth_mul multiply [-1.31638f, 2.45222f] 26
/model/layers.10/self_attn/k_proj/MatMul_smooth_mul multiply [-1.14123f, 2.85072f] 26
/model/layers.10/self_attn/v_proj/MatMul_smooth_mul multiply [-0.635305f, 0.964866f] 26
CustomOp/linalg::channelwiseQuantMatMul<40>43 contrib.epu.quadric_custom_op [-7.399f, 5.84284f] 28
/model/layers.10/Add add [-2119.36f, 9655.58f] 17
/model/layers.10/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-8.33987f, 9.41487f] 27
/model/layers.10/mlp/gate_proj/MatMul_smooth_mul multiply [-1.41946f, 1.68869f] 26
CustomOp/nn::gateProj<39, 27>41 contrib.epu.quadric_custom_op [-0.278465f, 8.82315f] 27
/model/layers.10/mlp/up_proj/MatMul_smooth_mul multiply [-1.41946f, 1.19373f] 27
CustomOp/linalg::channelwiseQuantMatMul<40>40 contrib.epu.quadric_custom_op [-5.76626f, 8.25113f] 27
/model/layers.10/mlp/Mul multiply [-15.7846f, 20.8058f] 24
/model/layers.10/mlp/down_proj/MatMul_smooth_mul multiply [-5.70319f, 2.03262f] 26
CustomOp/linalg::channelwiseQuantMatMul<37>42 contrib.epu.quadric_custom_op [-5.03255f, 18.6078f] 26
/model/layers.10/Add_1 add [-2119.79f, 9656.97f] 17
/model/layers.11/input_layernorm/Mul_1 contrib.epu.rms_norm [-5.77417f, 10.4802f] 27
/model/layers.11/self_attn/q_proj/MatMul_smooth_mul multiply [-1.24383f, 1.49914f] 28
/model/layers.11/self_attn/k_proj/MatMul_smooth_mul multiply [-0.911936f, 1.83978f] 28
/model/layers.11/self_attn/v_proj/MatMul_smooth_mul multiply [-0.494405f, 0.873254f] 28
CustomOp/linalg::channelwiseQuantMatMul<40>47 contrib.epu.quadric_custom_op [-7.13255f, 7.60828f] 28
/model/layers.11/Add add [-2118.77f, 9655.72f] 17
/model/layers.11/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-9.30534f, 7.62255f] 27
/model/layers.11/mlp/gate_proj/MatMul_smooth_mul multiply [-1.12876f, 1.54745f] 26
CustomOp/nn::gateProj<39, 27>45 contrib.epu.quadric_custom_op [-0.278465f, 12.7414f] 27
/model/layers.11/mlp/up_proj/MatMul_smooth_mul multiply [-1.37915f, 0.865526f] 26
CustomOp/linalg::channelwiseQuantMatMul<40>44 contrib.epu.quadric_custom_op [-8.25134f, 6.12666f] 27
/model/layers.11/mlp/Mul multiply [-12.4377f, 12.5137f] 24
/model/layers.11/mlp/down_proj/MatMul_smooth_mul multiply [-3.31324f, 2.34436f] 27
CustomOp/linalg::channelwiseQuantMatMul<38>46 contrib.epu.quadric_custom_op [-7.59268f, 11.5153f] 27
/model/layers.11/Add_1 add [-2118.11f, 9657.75f] 17
/model/layers.12/input_layernorm/Mul_1 contrib.epu.rms_norm [-6.76928f, 11.1257f] 27
/model/layers.12/self_attn/q_proj/MatMul_smooth_mul multiply [-1.31845f, 1.85694f] 28
/model/layers.12/self_attn/k_proj/MatMul_smooth_mul multiply [-1.1593f, 1.7855f] 27
/model/layers.12/self_attn/v_proj/MatMul_smooth_mul multiply [-0.844806f, 0.934658f] 28
CustomOp/linalg::channelwiseQuantMatMul<40>51 contrib.epu.quadric_custom_op [-7.8656f, 13.7375f] 27
/model/layers.12/Add add [-2117.66f, 9657.01f] 17
/model/layers.12/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-10.1594f, 7.15519f] 27
/model/layers.12/mlp/gate_proj/MatMul_smooth_mul multiply [-1.37687f, 1.39484f] 26
CustomOp/nn::gateProj<40, 27>49 contrib.epu.quadric_custom_op [-0.278465f, 7.39754f] 28
/model/layers.12/mlp/up_proj/MatMul_smooth_mul multiply [-1.92166f, 1.10387f] 26
CustomOp/linalg::channelwiseQuantMatMul<39>48 contrib.epu.quadric_custom_op [-9.58119f, 5.33228f] 27
/model/layers.12/mlp/Mul multiply [-60.1034f, 12.2375f] 24
/model/layers.12/mlp/down_proj/MatMul_smooth_mul multiply [-3.8142f, 3.11107f] 25
CustomOp/linalg::channelwiseQuantMatMul<37>50 contrib.epu.quadric_custom_op [-12.1023f, 17.851f] 26
/model/layers.12/Add_1 add [-2118.72f, 9659.33f] 17
/model/layers.13/input_layernorm/Mul_1 contrib.epu.rms_norm [-5.04587f, 9.31552f] 27
/model/layers.13/self_attn/q_proj/MatMul_smooth_mul multiply [-1.00549f, 1.73688f] 28
/model/layers.13/self_attn/k_proj/MatMul_smooth_mul multiply [-1.76774f, 1.88082f] 28
/model/layers.13/self_attn/v_proj/MatMul_smooth_mul multiply [-0.604113f, 0.910164f] 28
CustomOp/linalg::channelwiseQuantMatMul<40>55 contrib.epu.quadric_custom_op [-5.54142f, 8.70166f] 27
/model/layers.13/Add add [-2117.86f, 9659.14f] 17
/model/layers.13/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-11.9317f, 6.23108f] 27
/model/layers.13/mlp/gate_proj/MatMul_smooth_mul multiply [-1.47657f, 1.26196f] 26
CustomOp/nn::gateProj<40, 26>53 contrib.epu.quadric_custom_op [-0.278465f, 17.2259f] 26
/model/layers.13/mlp/up_proj/MatMul_smooth_mul multiply [-2.24165f, 1.21306f] 26
CustomOp/linalg::channelwiseQuantMatMul<39>52 contrib.epu.quadric_custom_op [-30.4806f, 32.1002f] 25
/model/layers.13/mlp/Mul multiply [-16.9783f, 33.7621f] 21
/model/layers.13/mlp/down_proj/MatMul_smooth_mul multiply [-2.68679f, 2.99732f] 25
CustomOp/linalg::channelwiseQuantMatMul<38>54 contrib.epu.quadric_custom_op [-11.2142f, 11.9456f] 27
/model/layers.13/Add_1 add [-2120.76f, 9664.39f] 17
/model/layers.14/input_layernorm/Mul_1 contrib.epu.rms_norm [-6.68111f, 13.0119f] 27
/model/layers.14/self_attn/q_proj/MatMul_smooth_mul multiply [-1.40054f, 1.87548f] 27
/model/layers.14/self_attn/k_proj/MatMul_smooth_mul multiply [-1.56909f, 1.68693f] 28
/model/layers.14/self_attn/v_proj/MatMul_smooth_mul multiply [-0.623238f, 0.820612f] 28
CustomOp/linalg::channelwiseQuantMatMul<40>59 contrib.epu.quadric_custom_op [-8.29877f, 19.2297f] 26
/model/layers.14/Add add [-2120.07f, 9663.34f] 17
/model/layers.14/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-14.412f, 6.25216f] 27
/model/layers.14/mlp/gate_proj/MatMul_smooth_mul multiply [-1.47538f, 1.42171f] 26
CustomOp/nn::gateProj<39, 26>57 contrib.epu.quadric_custom_op [-0.278465f, 17.3895f] 26
/model/layers.14/mlp/up_proj/MatMul_smooth_mul multiply [-2.12944f, 0.916271f] 26
CustomOp/linalg::channelwiseQuantMatMul<39>56 contrib.epu.quadric_custom_op [-24.6362f, 32.0892f] 25
/model/layers.14/mlp/Mul multiply [-84.1848f, 13.7615f] 21
/model/layers.14/mlp/down_proj/MatMul_smooth_mul multiply [-4.83579f, 2.83515f] 24
CustomOp/linalg::channelwiseQuantMatMul<36>58 contrib.epu.quadric_custom_op [-18.0127f, 15.0041f] 26
/model/layers.14/Add_1 add [-2123.29f, 9666.16f] 17
/model/layers.15/input_layernorm/Mul_1 contrib.epu.rms_norm [-7.40155f, 13.516f] 27
/model/layers.15/self_attn/q_proj/MatMul_smooth_mul multiply [-2.20786f, 2.05356f] 27
/model/layers.15/self_attn/k_proj/MatMul_smooth_mul multiply [-1.312f, 1.55167f] 27
/model/layers.15/self_attn/v_proj/MatMul_smooth_mul multiply [-0.944417f, 1.36199f] 28
CustomOp/linalg::channelwiseQuantMatMul<39>63 contrib.epu.quadric_custom_op [-6.83366f, 13.2874f] 27
/model/layers.15/Add add [-2122.7f, 9664.69f] 17
/model/layers.15/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-16.0488f, 6.0668f] 26
/model/layers.15/mlp/gate_proj/MatMul_smooth_mul multiply [-1.4562f, 1.5061f] 26
CustomOp/nn::gateProj<39, 26>61 contrib.epu.quadric_custom_op [-0.278465f, 19.5172f] 26
/model/layers.15/mlp/up_proj/MatMul_smooth_mul multiply [-3.49456f, 0.898868f] 26
CustomOp/linalg::channelwiseQuantMatMul<38>60 contrib.epu.quadric_custom_op [-16.5025f, 59.7275f] 25
/model/layers.15/mlp/Mul multiply [-27.8356f, 48.4799f] 20
/model/layers.15/mlp/down_proj/MatMul_smooth_mul multiply [-3.85041f, 4.73925f] 25
CustomOp/linalg::channelwiseQuantMatMul<36>62 contrib.epu.quadric_custom_op [-16.8693f, 20.4889f] 26
/model/layers.15/Add_1 add [-2123.21f, 9668.25f] 17
/model/layers.16/input_layernorm/Mul_1 contrib.epu.rms_norm [-8.15026f, 15.6546f] 26
/model/layers.16/self_attn/q_proj/MatMul_smooth_mul multiply [-1.3261f, 2.04648f] 27
/model/layers.16/self_attn/k_proj/MatMul_smooth_mul multiply [-1.65707f, 1.55809f] 27
/model/layers.16/self_attn/v_proj/MatMul_smooth_mul multiply [-0.734148f, 0.882209f] 28
CustomOp/linalg::channelwiseQuantMatMul<38>67 contrib.epu.quadric_custom_op [-39.5976f, 22.6377f] 25
/model/layers.16/Add add [-2122.92f, 9666.15f] 17
/model/layers.16/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-36.8359f, 6.53895f] 25
/model/layers.16/mlp/gate_proj/MatMul_smooth_mul multiply [-4.24899f, 2.25932f] 25
CustomOp/nn::gateProj<37, 23>65 contrib.epu.quadric_custom_op [-0.278465f, 78.4472f] 24
/model/layers.16/mlp/up_proj/MatMul_smooth_mul multiply [-4.45054f, 1.66161f] 25
CustomOp/linalg::channelwiseQuantMatMul<37>64 contrib.epu.quadric_custom_op [-39.9357f, 84.0414f] 24
/model/layers.16/mlp/Mul multiply [-191.33f, 4661.68f] 18
/model/layers.16/mlp/down_proj/MatMul_smooth_mul multiply [-40.6463f, 352.551f] 18
CustomOp/linalg::channelwiseQuantMatMul<27>66 contrib.epu.quadric_custom_op [-1202.39f, 13581.2f] 17
/model/layers.16/Add_1 add [-2130f, 13602.6f] 16
/model/layers.17/input_layernorm/Mul_1 contrib.epu.rms_norm [-7.88683f, 13.0965f] 27
/model/layers.17/self_attn/q_proj/MatMul_smooth_mul multiply [-1.33071f, 1.92274f] 27
/model/layers.17/self_attn/k_proj/MatMul_smooth_mul multiply [-1.34317f, 1.59226f] 27
/model/layers.17/self_attn/v_proj/MatMul_smooth_mul multiply [-0.929569f, 1.09737f] 28
CustomOp/linalg::channelwiseQuantMatMul<37>71 contrib.epu.quadric_custom_op [-14.5229f, 61.5807f] 25
/model/layers.17/Add add [-2129.24f, 13637f] 16
/model/layers.17/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-12.2827f, 6.15562f] 27
/model/layers.17/mlp/gate_proj/MatMul_smooth_mul multiply [-1.52264f, 1.37121f] 26
CustomOp/nn::gateProj<39, 28>69 contrib.epu.quadric_custom_op [-0.278465f, 7.73579f] 28
/model/layers.17/mlp/up_proj/MatMul_smooth_mul multiply [-1.52896f, 0.967658f] 26
CustomOp/linalg::channelwiseQuantMatMul<40>68 contrib.epu.quadric_custom_op [-7.26674f, 35.3441f] 25
/model/layers.17/mlp/Mul multiply [-13.8208f, 9.58101f] 22
/model/layers.17/mlp/down_proj/MatMul_smooth_mul multiply [-3.27512f, 3.71965f] 27
CustomOp/linalg::channelwiseQuantMatMul<38>70 contrib.epu.quadric_custom_op [-21.4484f, 22.413f] 26
/model/layers.17/Add_1 add [-2128.21f, 13636.9f] 16
/model/layers.18/input_layernorm/Mul_1 contrib.epu.rms_norm [-10.6242f, 15.4063f] 26
/model/layers.18/self_attn/q_proj/MatMul_smooth_mul multiply [-1.42803f, 2.10296f] 27
/model/layers.18/self_attn/k_proj/MatMul_smooth_mul multiply [-1.48392f, 1.69128f] 27
/model/layers.18/self_attn/v_proj/MatMul_smooth_mul multiply [-0.998706f, 1.36009f] 28
CustomOp/linalg::channelwiseQuantMatMul<40>75 contrib.epu.quadric_custom_op [-11.2483f, 15.3732f] 27
/model/layers.18/Add add [-2127.25f, 13638.1f] 16
/model/layers.18/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-11.7031f, 6.12549f] 27
/model/layers.18/mlp/gate_proj/MatMul_smooth_mul multiply [-1.18675f, 1.43604f] 26
CustomOp/nn::gateProj<39, 27>73 contrib.epu.quadric_custom_op [-0.278465f, 8.13071f] 27
/model/layers.18/mlp/up_proj/MatMul_smooth_mul multiply [-1.42666f, 1.13167f] 26
CustomOp/linalg::channelwiseQuantMatMul<39>72 contrib.epu.quadric_custom_op [-6.42202f, 15.0935f] 27
/model/layers.18/mlp/Mul multiply [-9.16472f, 64.1296f] 24
/model/layers.18/mlp/down_proj/MatMul_smooth_mul multiply [-3.79416f, 8.76246f] 25
CustomOp/linalg::channelwiseQuantMatMul<35>74 contrib.epu.quadric_custom_op [-27.1561f, 62.6349f] 25
/model/layers.18/Add_1 add [-2127.74f, 13638.1f] 16
/model/layers.19/input_layernorm/Mul_1 contrib.epu.rms_norm [-16.3269f, 23.5058f] 26
/model/layers.19/self_attn/q_proj/MatMul_smooth_mul multiply [-1.87381f, 2.36709f] 27
/model/layers.19/self_attn/k_proj/MatMul_smooth_mul multiply [-1.99903f, 2.0809f] 27
/model/layers.19/self_attn/v_proj/MatMul_smooth_mul multiply [-0.96973f, 1.21001f] 27
CustomOp/linalg::channelwiseQuantMatMul<39>79 contrib.epu.quadric_custom_op [-9.69087f, 19.3656f] 26
/model/layers.19/Add add [-2127.19f, 13637.7f] 16
/model/layers.19/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-11.4451f, 6.57414f] 27
/model/layers.19/mlp/gate_proj/MatMul_smooth_mul multiply [-1.50948f, 1.27079f] 27
CustomOp/nn::gateProj<40, 27>77 contrib.epu.quadric_custom_op [-0.278465f, 8.66148f] 27
/model/layers.19/mlp/up_proj/MatMul_smooth_mul multiply [-1.77521f, 1.54593f] 27
CustomOp/linalg::channelwiseQuantMatMul<39>76 contrib.epu.quadric_custom_op [-6.79348f, 5.92249f] 28
/model/layers.19/mlp/Mul multiply [-10.5208f, 11.7491f] 25
/model/layers.19/mlp/down_proj/MatMul_smooth_mul multiply [-3.32359f, 3.64369f] 27
CustomOp/linalg::channelwiseQuantMatMul<36>78 contrib.epu.quadric_custom_op [-28.7991f, 32.1844f] 25
/model/layers.19/Add_1 add [-2127.2f, 13637.6f] 16
/model/layers.20/input_layernorm/Mul_1 contrib.epu.rms_norm [-17.2382f, 20.6114f] 26
/model/layers.20/self_attn/q_proj/MatMul_smooth_mul multiply [-1.79777f, 2.35375f] 27
/model/layers.20/self_attn/k_proj/MatMul_smooth_mul multiply [-2.07954f, 1.97869f] 27
/model/layers.20/self_attn/v_proj/MatMul_smooth_mul multiply [-0.97567f, 1.51478f] 27
CustomOp/linalg::channelwiseQuantMatMul<40>83 contrib.epu.quadric_custom_op [-11.7166f, 31.5278f] 26
/model/layers.20/Add add [-2126.62f, 13634.6f] 16
/model/layers.20/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-12.2707f, 7.12023f] 27
/model/layers.20/mlp/gate_proj/MatMul_smooth_mul multiply [-1.27321f, 1.48049f] 28
CustomOp/nn::gateProj<40, 27>81 contrib.epu.quadric_custom_op [-0.278465f, 8.94129f] 27
/model/layers.20/mlp/up_proj/MatMul_smooth_mul multiply [-1.25212f, 1.368f] 27
CustomOp/linalg::channelwiseQuantMatMul<40>80 contrib.epu.quadric_custom_op [-6.33998f, 8.02105f] 27
/model/layers.20/mlp/Mul multiply [-18.5987f, 19.8616f] 24
/model/layers.20/mlp/down_proj/MatMul_smooth_mul multiply [-4.02727f, 2.56306f] 26
CustomOp/linalg::channelwiseQuantMatMul<38>82 contrib.epu.quadric_custom_op [-16.8789f, 16.8205f] 26
/model/layers.20/Add_1 add [-2125.87f, 13634.7f] 16
/model/layers.21/input_layernorm/Mul_1 contrib.epu.rms_norm [-23.1988f, 24.3195f] 26
/model/layers.21/self_attn/q_proj/MatMul_smooth_mul multiply [-2.32063f, 2.84108f] 27
/model/layers.21/self_attn/k_proj/MatMul_smooth_mul multiply [-2.10805f, 2.21897f] 27
/model/layers.21/self_attn/v_proj/MatMul_smooth_mul multiply [-1.10305f, 1.53724f] 27
CustomOp/linalg::channelwiseQuantMatMul<39>87 contrib.epu.quadric_custom_op [-8.88642f, 21.276f] 26
/model/layers.21/Add add [-2124.82f, 13636.5f] 16
/model/layers.21/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-10.9728f, 7.28116f] 27
/model/layers.21/mlp/gate_proj/MatMul_smooth_mul multiply [-1.61932f, 1.36054f] 26
CustomOp/nn::gateProj<39, 27>85 contrib.epu.quadric_custom_op [-0.278465f, 9.86578f] 27
/model/layers.21/mlp/up_proj/MatMul_smooth_mul multiply [-1.27146f, 1.53064f] 26
CustomOp/linalg::channelwiseQuantMatMul<39>84 contrib.epu.quadric_custom_op [-10.277f, 10.0276f] 27
/model/layers.21/mlp/Mul multiply [-41.594f, 53.735f] 24
/model/layers.21/mlp/down_proj/MatMul_smooth_mul multiply [-5.86662f, 8.1078f] 25
CustomOp/linalg::channelwiseQuantMatMul<37>86 contrib.epu.quadric_custom_op [-32.024f, 16.017f] 25
/model/layers.21/Add_1 add [-2124.91f, 13636.6f] 16
/model/layers.22/input_layernorm/Mul_1 contrib.epu.rms_norm [-30.8369f, 32.7385f] 25
/model/layers.22/self_attn/q_proj/MatMul_smooth_mul multiply [-2.71083f, 3.39645f] 27
/model/layers.22/self_attn/k_proj/MatMul_smooth_mul multiply [-3.44054f, 3.20511f] 26
/model/layers.22/self_attn/v_proj/MatMul_smooth_mul multiply [-1.28456f, 1.37292f] 27
CustomOp/linalg::channelwiseQuantMatMul<39>91 contrib.epu.quadric_custom_op [-12.5842f, 31.0796f] 26
/model/layers.22/Add add [-2124.1f, 13635.5f] 16
/model/layers.22/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-10.0459f, 8.97793f] 27
/model/layers.22/mlp/gate_proj/MatMul_smooth_mul multiply [-1.56994f, 1.54569f] 27
CustomOp/nn::gateProj<39, 26>89 contrib.epu.quadric_custom_op [-0.278465f, 16.0594f] 26
/model/layers.22/mlp/up_proj/MatMul_smooth_mul multiply [-1.54481f, 1.40509f] 27
CustomOp/linalg::channelwiseQuantMatMul<39>88 contrib.epu.quadric_custom_op [-8.84424f, 9.51616f] 27
/model/layers.22/mlp/Mul multiply [-42.2895f, 37.618f] 23
/model/layers.22/mlp/down_proj/MatMul_smooth_mul multiply [-5.37248f, 5.52488f] 26
CustomOp/linalg::channelwiseQuantMatMul<37>90 contrib.epu.quadric_custom_op [-24.8563f, 17.5565f] 26
/model/layers.22/Add_1 add [-2123.61f, 13635.4f] 16
/model/layers.23/input_layernorm/Mul_1 contrib.epu.rms_norm [-32.5652f, 31.0209f] 25
/model/layers.23/self_attn/q_proj/MatMul_smooth_mul multiply [-3.54566f, 3.57557f] 27
/model/layers.23/self_attn/k_proj/MatMul_smooth_mul multiply [-2.95217f, 3.04757f] 27
/model/layers.23/self_attn/v_proj/MatMul_smooth_mul multiply [-1.23035f, 1.36422f] 27
CustomOp/linalg::channelwiseQuantMatMul<39>95 contrib.epu.quadric_custom_op [-8.13639f, 35.0445f] 25
/model/layers.23/Add add [-2123.06f, 13632.6f] 16
/model/layers.23/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-10.1758f, 10.4433f] 27
/model/layers.23/mlp/gate_proj/MatMul_smooth_mul multiply [-1.73753f, 1.58375f] 27
CustomOp/nn::gateProj<39, 26>93 contrib.epu.quadric_custom_op [-0.278465f, 15.0449f] 27
/model/layers.23/mlp/up_proj/MatMul_smooth_mul multiply [-1.42182f, 1.61803f] 27
CustomOp/linalg::channelwiseQuantMatMul<39>92 contrib.epu.quadric_custom_op [-12.3414f, 10.3189f] 27
/model/layers.23/mlp/Mul multiply [-64.6526f, 60.3269f] 23
/model/layers.23/mlp/down_proj/MatMul_smooth_mul multiply [-5.1301f, 6.31523f] 25
CustomOp/linalg::channelwiseQuantMatMul<37>94 contrib.epu.quadric_custom_op [-24.6908f, 31.6436f] 26
/model/layers.23/Add_1 add [-2122.79f, 13632.6f] 16
/model/layers.24/input_layernorm/Mul_1 contrib.epu.rms_norm [-50.8646f, 39.9522f] 25
/model/layers.24/self_attn/q_proj/MatMul_smooth_mul multiply [-4.28403f, 3.79681f] 26
/model/layers.24/self_attn/k_proj/MatMul_smooth_mul multiply [-3.08274f, 3.52826f] 27
/model/layers.24/self_attn/v_proj/MatMul_smooth_mul multiply [-2.11367f, 2.06183f] 27
CustomOp/linalg::channelwiseQuantMatMul<38>99 contrib.epu.quadric_custom_op [-16.7965f, 31.3655f] 26
/model/layers.24/Add add [-2123.99f, 13635.1f] 16
/model/layers.24/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-10.2601f, 12.5924f] 27
/model/layers.24/mlp/gate_proj/MatMul_smooth_mul multiply [-1.79581f, 1.95177f] 26
CustomOp/nn::gateProj<39, 26>97 contrib.epu.quadric_custom_op [-0.278465f, 17.9184f] 26
/model/layers.24/mlp/up_proj/MatMul_smooth_mul multiply [-1.58951f, 1.69809f] 26
CustomOp/linalg::channelwiseQuantMatMul<39>96 contrib.epu.quadric_custom_op [-11.0467f, 11.354f] 27
/model/layers.24/mlp/Mul multiply [-107.554f, 90.9104f] 23
/model/layers.24/mlp/down_proj/MatMul_smooth_mul multiply [-6.40439f, 6.75044f] 25
CustomOp/linalg::channelwiseQuantMatMul<36>98 contrib.epu.quadric_custom_op [-22.1531f, 38.053f] 25
/model/layers.24/Add_1 add [-2123.99f, 13635.3f] 16
/model/layers.25/input_layernorm/Mul_1 contrib.epu.rms_norm [-40.2525f, 33.3074f] 25
/model/layers.25/self_attn/q_proj/MatMul_smooth_mul multiply [-3.55716f, 3.96355f] 26
/model/layers.25/self_attn/k_proj/MatMul_smooth_mul multiply [-2.68001f, 3.21491f] 27
/model/layers.25/self_attn/v_proj/MatMul_smooth_mul multiply [-1.83399f, 1.64135f] 27
CustomOp/linalg::channelwiseQuantMatMul<39>103 contrib.epu.quadric_custom_op [-10.4602f, 17.5029f] 26
/model/layers.25/Add add [-2123.75f, 13635.5f] 16
/model/layers.25/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-11.309f, 12.7584f] 27
/model/layers.25/mlp/gate_proj/MatMul_smooth_mul multiply [-1.95923f, 2.00956f] 26
CustomOp/nn::gateProj<39, 26>101 contrib.epu.quadric_custom_op [-0.278465f, 20.8095f] 26
/model/layers.25/mlp/up_proj/MatMul_smooth_mul multiply [-1.65599f, 1.83448f] 26
CustomOp/linalg::channelwiseQuantMatMul<39>100 contrib.epu.quadric_custom_op [-16.033f, 12.6045f] 26
/model/layers.25/mlp/Mul multiply [-117.555f, 109.813f] 22
/model/layers.25/mlp/down_proj/MatMul_smooth_mul multiply [-7.04222f, 8.54142f] 24
CustomOp/linalg::channelwiseQuantMatMul<36>102 contrib.epu.quadric_custom_op [-19.0471f, 36.3233f] 25
/model/layers.25/Add_1 add [-2123.73f, 13635.5f] 16
/model/layers.26/input_layernorm/Mul_1 contrib.epu.rms_norm [-50.6021f, 38.8768f] 25
/model/layers.26/self_attn/q_proj/MatMul_smooth_mul multiply [-4.46383f, 4.41033f] 26
/model/layers.26/self_attn/k_proj/MatMul_smooth_mul multiply [-3.09661f, 3.23338f] 27
/model/layers.26/self_attn/v_proj/MatMul_smooth_mul multiply [-1.95177f, 1.62671f] 27
CustomOp/linalg::channelwiseQuantMatMul<40>107 contrib.epu.quadric_custom_op [-8.51294f, 19.8805f] 26
/model/layers.26/Add add [-2124.04f, 13636.4f] 16
/model/layers.26/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-12.5561f, 15.4058f] 26
/model/layers.26/mlp/gate_proj/MatMul_smooth_mul multiply [-2.00442f, 1.59836f] 26
CustomOp/nn::gateProj<39, 26>105 contrib.epu.quadric_custom_op [-0.278465f, 22.6042f] 26
/model/layers.26/mlp/up_proj/MatMul_smooth_mul multiply [-1.56655f, 2.01047f] 26
CustomOp/linalg::channelwiseQuantMatMul<38>104 contrib.epu.quadric_custom_op [-12.8656f, 13.9187f] 27
/model/layers.26/mlp/Mul multiply [-105.01f, 132.743f] 22
/model/layers.26/mlp/down_proj/MatMul_smooth_mul multiply [-6.63407f, 7.95295f] 24
CustomOp/linalg::channelwiseQuantMatMul<36>106 contrib.epu.quadric_custom_op [-21.3917f, 49.6387f] 25
/model/layers.26/Add_1 add [-2124.03f, 13636.6f] 16
/model/layers.27/input_layernorm/Mul_1 contrib.epu.rms_norm [-61.2838f, 48.1027f] 25
/model/layers.27/self_attn/q_proj/MatMul_smooth_mul multiply [-4.33818f, 4.62879f] 26
/model/layers.27/self_attn/k_proj/MatMul_smooth_mul multiply [-2.61422f, 3.56265f] 26
/model/layers.27/self_attn/v_proj/MatMul_smooth_mul multiply [-2.02177f, 2.13091f] 27
CustomOp/linalg::channelwiseQuantMatMul<39>111 contrib.epu.quadric_custom_op [-8.8824f, 19.0919f] 26
/model/layers.27/Add add [-2124.55f, 13638.8f] 16
/model/layers.27/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-14.5542f, 17.2044f] 26
/model/layers.27/mlp/gate_proj/MatMul_smooth_mul multiply [-1.75765f, 1.86848f] 26
CustomOp/nn::gateProj<39, 26>109 contrib.epu.quadric_custom_op [-0.278465f, 19.2789f] 26
/model/layers.27/mlp/up_proj/MatMul_smooth_mul multiply [-3.00212f, 1.82197f] 26
CustomOp/linalg::channelwiseQuantMatMul<38>108 contrib.epu.quadric_custom_op [-13.1627f, 12.5415f] 27
/model/layers.27/mlp/Mul multiply [-137.571f, 122.284f] 23
/model/layers.27/mlp/down_proj/MatMul_smooth_mul multiply [-8.10809f, 8.06381f] 24
CustomOp/linalg::channelwiseQuantMatMul<36>110 contrib.epu.quadric_custom_op [-29.7459f, 42.6272f] 25
/model/layers.27/Add_1 add [-2124.49f, 13639.2f] 16
/model/layers.28/input_layernorm/Mul_1 contrib.epu.rms_norm [-65.7556f, 56.4749f] 24
/model/layers.28/self_attn/q_proj/MatMul_smooth_mul multiply [-4.29473f, 4.18318f] 26
/model/layers.28/self_attn/k_proj/MatMul_smooth_mul multiply [-3.32743f, 3.96202f] 27
/model/layers.28/self_attn/v_proj/MatMul_smooth_mul multiply [-2.12739f, 2.4797f] 27
CustomOp/linalg::channelwiseQuantMatMul<39>115 contrib.epu.quadric_custom_op [-16.1447f, 22.5356f] 26
/model/layers.28/Add add [-2125.41f, 13642.1f] 16
/model/layers.28/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-13.3559f, 17.9234f] 26
/model/layers.28/mlp/gate_proj/MatMul_smooth_mul multiply [-2.34241f, 2.10349f] 23
CustomOp/nn::gateProj<38, 26>113 contrib.epu.quadric_custom_op [-0.278465f, 23.2141f] 26
/model/layers.28/mlp/up_proj/MatMul_smooth_mul multiply [-2.12235f, 1.94853f] 23
CustomOp/linalg::channelwiseQuantMatMul<38>112 contrib.epu.quadric_custom_op [-18.7225f, 14.9992f] 26
/model/layers.28/mlp/Mul multiply [-132.766f, 140.229f] 22
/model/layers.28/mlp/down_proj/MatMul_smooth_mul multiply [-14.6529f, 10.7982f] 24
CustomOp/linalg::channelwiseQuantMatMul<35>114 contrib.epu.quadric_custom_op [-38.7672f, 58.2323f] 25
/model/layers.28/Add_1 add [-2125.36f, 13642.5f] 16
/model/layers.29/input_layernorm/Mul_1 contrib.epu.rms_norm [-87.7913f, 63.9043f] 24
/model/layers.29/self_attn/q_proj/MatMul_smooth_mul multiply [-5.64585f, 4.53467f] 26
/model/layers.29/self_attn/k_proj/MatMul_smooth_mul multiply [-4.66197f, 4.02658f] 26
/model/layers.29/self_attn/v_proj/MatMul_smooth_mul multiply [-3.72409f, 3.62137f] 26
CustomOp/linalg::channelwiseQuantMatMul<39>119 contrib.epu.quadric_custom_op [-15.1006f, 23.0286f] 26
/model/layers.29/Add add [-2126.23f, 13643.9f] 16
/model/layers.29/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-14.5663f, 19.9309f] 26
/model/layers.29/mlp/gate_proj/MatMul_smooth_mul multiply [-1.95651f, 1.86312f] 25
CustomOp/nn::gateProj<39, 26>117 contrib.epu.quadric_custom_op [-0.278465f, 19.1744f] 26
/model/layers.29/mlp/up_proj/MatMul_smooth_mul multiply [-2.62913f, 2.32513f] 25
CustomOp/linalg::channelwiseQuantMatMul<38>116 contrib.epu.quadric_custom_op [-16.4508f, 15.0669f] 26
/model/layers.29/mlp/Mul multiply [-150.54f, 123.012f] 22
/model/layers.29/mlp/down_proj/MatMul_smooth_mul multiply [-11.6399f, 11.8174f] 24
CustomOp/linalg::channelwiseQuantMatMul<35>118 contrib.epu.quadric_custom_op [-49.9533f, 76.5029f] 24
/model/layers.29/Add_1 add [-2126.19f, 13644f] 16
/model/layers.30/input_layernorm/Mul_1 contrib.epu.rms_norm [-97.8058f, 70.9147f] 24
/model/layers.30/self_attn/q_proj/MatMul_smooth_mul multiply [-8.32574f, 6.67203f] 26
/model/layers.30/self_attn/k_proj/MatMul_smooth_mul multiply [-5.20679f, 4.87224f] 26
/model/layers.30/self_attn/v_proj/MatMul_smooth_mul multiply [-3.51157f, 3.44255f] 26
CustomOp/linalg::channelwiseQuantMatMul<38>123 contrib.epu.quadric_custom_op [-26.3882f, 42.82f] 25
/model/layers.30/Add add [-2126.46f, 13648.6f] 16
/model/layers.30/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-16.1268f, 19.8625f] 26
/model/layers.30/mlp/gate_proj/MatMul_smooth_mul multiply [-2.32769f, 1.89138f] 27
CustomOp/nn::gateProj<38, 26>121 contrib.epu.quadric_custom_op [-0.278465f, 20.8345f] 26
/model/layers.30/mlp/up_proj/MatMul_smooth_mul multiply [-2.93709f, 3.14719f] 27
CustomOp/linalg::channelwiseQuantMatMul<38>120 contrib.epu.quadric_custom_op [-22.3626f, 17.1751f] 26
/model/layers.30/mlp/Mul multiply [-147.928f, 141.815f] 22
/model/layers.30/mlp/down_proj/MatMul_smooth_mul multiply [-15.4673f, 11.2169f] 24
CustomOp/linalg::channelwiseQuantMatMul<35>122 contrib.epu.quadric_custom_op [-53.484f, 106.991f] 24
/model/layers.30/Add_1 add [-2126.44f, 13648.7f] 16
/model/layers.31/input_layernorm/Mul_1 contrib.epu.rms_norm [-119.533f, 93.6401f] 24
/model/layers.31/self_attn/q_proj/MatMul_smooth_mul multiply [-7.37432f, 6.46369f] 26
/model/layers.31/self_attn/k_proj/MatMul_smooth_mul multiply [-5.50964f, 5.44922f] 25
/model/layers.31/self_attn/v_proj/MatMul_smooth_mul multiply [-3.79376f, 3.57184f] 26
CustomOp/linalg::channelwiseQuantMatMul<38>127 contrib.epu.quadric_custom_op [-17.2974f, 50.1237f] 25
/model/layers.31/Add add [-2126.83f, 13653.5f] 16
/model/layers.31/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-17.2516f, 18.0006f] 26
/model/layers.31/mlp/gate_proj/MatMul_smooth_mul multiply [-2.36857f, 2.28309f] 27
CustomOp/nn::gateProj<38, 26>125 contrib.epu.quadric_custom_op [-0.278465f, 21.2043f] 26
/model/layers.31/mlp/up_proj/MatMul_smooth_mul multiply [-2.60498f, 3.19835f] 27
CustomOp/linalg::channelwiseQuantMatMul<37>124 contrib.epu.quadric_custom_op [-18.9059f, 19.1897f] 26
/model/layers.31/mlp/Mul multiply [-147.492f, 171.108f] 22
/model/layers.31/mlp/down_proj/MatMul_smooth_mul multiply [-18.9471f, 14.1054f] 24
CustomOp/linalg::channelwiseQuantMatMul<34>126 contrib.epu.quadric_custom_op [-42.7217f, 101.298f] 24
/model/layers.31/Add_1 add [-2126.65f, 13653.4f] 16
/model/layers.32/input_layernorm/Mul_1 contrib.epu.rms_norm [-129.345f, 108.642f] 23
/model/layers.32/self_attn/q_proj/MatMul_smooth_mul multiply [-7.50556f, 7.79626f] 26
/model/layers.32/self_attn/k_proj/MatMul_smooth_mul multiply [-6.04744f, 5.6172f] 26
/model/layers.32/self_attn/v_proj/MatMul_smooth_mul multiply [-5.33153f, 4.74853f] 26
CustomOp/linalg::channelwiseQuantMatMul<37>131 contrib.epu.quadric_custom_op [-26.9792f, 52.8894f] 25
/model/layers.32/Add add [-2125.55f, 13676.5f] 16
/model/layers.32/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-17.899f, 23.7326f] 26
/model/layers.32/mlp/gate_proj/MatMul_smooth_mul multiply [-2.44445f, 3.21304f] 26
CustomOp/nn::gateProj<38, 26>129 contrib.epu.quadric_custom_op [-0.278465f, 25.1264f] 26
/model/layers.32/mlp/up_proj/MatMul_smooth_mul multiply [-4.51652f, 3.91687f] 26
CustomOp/linalg::channelwiseQuantMatMul<37>128 contrib.epu.quadric_custom_op [-18.7255f, 22.3183f] 26
/model/layers.32/mlp/Mul multiply [-190.096f, 285.781f] 21
/model/layers.32/mlp/down_proj/MatMul_smooth_mul multiply [-16.2074f, 13.687f] 24
CustomOp/linalg::channelwiseQuantMatMul<34>130 contrib.epu.quadric_custom_op [-67.0631f, 111.421f] 24
/model/layers.32/Add_1 add [-2125.27f, 13672.9f] 16
/model/layers.33/input_layernorm/Mul_1 contrib.epu.rms_norm [-187.593f, 150.813f] 23
/model/layers.33/self_attn/q_proj/MatMul_smooth_mul multiply [-8.17666f, 7.40637f] 25
/model/layers.33/self_attn/k_proj/MatMul_smooth_mul multiply [-7.68112f, 7.72248f] 25
/model/layers.33/self_attn/v_proj/MatMul_smooth_mul multiply [-6.35676f, 6.26976f] 25
CustomOp/linalg::channelwiseQuantMatMul<37>135 contrib.epu.quadric_custom_op [-22.3657f, 115.159f] 24
/model/layers.33/Add add [-2125.11f, 13717f] 16
/model/layers.33/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-29.27f, 27.8766f] 26
/model/layers.33/mlp/gate_proj/MatMul_smooth_mul multiply [-4.70675f, 3.16855f] 26
CustomOp/nn::gateProj<37, 26>133 contrib.epu.quadric_custom_op [-0.278465f, 20.9414f] 26
/model/layers.33/mlp/up_proj/MatMul_smooth_mul multiply [-7.67804f, 4.66365f] 27
CustomOp/linalg::channelwiseQuantMatMul<36>132 contrib.epu.quadric_custom_op [-21.9644f, 28.9324f] 26
/model/layers.33/mlp/Mul multiply [-191.84f, 180.474f] 21
/model/layers.33/mlp/down_proj/MatMul_smooth_mul multiply [-16.8717f, 15.8574f] 24
CustomOp/linalg::channelwiseQuantMatMul<34>134 contrib.epu.quadric_custom_op [-224.43f, 154.382f] 23
/model/layers.33/Add_1 add [-2126.49f, 13736f] 16
/model/layers.34/input_layernorm/Mul_1 contrib.epu.rms_norm [-190.924f, 150.511f] 23
/model/layers.34/self_attn/q_proj/MatMul_smooth_mul multiply [-9.52248f, 9.91184f] 25
/model/layers.34/self_attn/k_proj/MatMul_smooth_mul multiply [-7.38449f, 7.55077f] 26
/model/layers.34/self_attn/v_proj/MatMul_smooth_mul multiply [-5.32747f, 5.83561f] 26
CustomOp/linalg::channelwiseQuantMatMul<36>139 contrib.epu.quadric_custom_op [-134.891f, 252.592f] 23
/model/layers.34/Add add [-2127.57f, 13665.9f] 16
/model/layers.34/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-102.457f, 31.8715f] 24
/model/layers.34/mlp/gate_proj/MatMul_smooth_mul multiply [-11.9706f, 3.27663f] 24
CustomOp/nn::gateProj<35, 25>137 contrib.epu.quadric_custom_op [-0.278465f, 43.9184f] 25
/model/layers.34/mlp/up_proj/MatMul_smooth_mul multiply [-15.5429f, 4.71439f] 24
CustomOp/linalg::channelwiseQuantMatMul<34>136 contrib.epu.quadric_custom_op [-33.824f, 68.3458f] 24
/model/layers.34/mlp/Mul multiply [-1174.96f, 792.102f] 19
/model/layers.34/mlp/down_proj/MatMul_smooth_mul multiply [-110.529f, 78.4391f] 22
CustomOp/linalg::channelwiseQuantMatMul<31>138 contrib.epu.quadric_custom_op [-18980.2f, 553.5f] 16
/model/layers.34/Add_1 add [-7343.94f, 5351.69f] 16
/model/layers.35/input_layernorm/Mul_1 contrib.epu.rms_norm [-298.728f, 121.485f] 22
/model/layers.35/self_attn/q_proj/MatMul_smooth_mul multiply [-40.8404f, 9.94293f] 24
/model/layers.35/self_attn/k_proj/MatMul_smooth_mul multiply [-34.7046f, 8.17391f] 25
/model/layers.35/self_attn/v_proj/MatMul_smooth_mul multiply [-6.21749f, 4.66202f] 25
CustomOp/linalg::channelwiseQuantMatMul<36>143 contrib.epu.quadric_custom_op [-282.141f, 824.661f] 21
/model/layers.35/Add add [-7101.39f, 5072.41f] 16
/model/layers.35/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-601.076f, 47.2325f] 21
/model/layers.35/mlp/gate_proj/MatMul_smooth_mul multiply [-42.4844f, 4.64554f] 22
CustomOp/nn::gateProj<34, 24>141 contrib.epu.quadric_custom_op [-0.278465f, 109.277f] 24
/model/layers.35/mlp/up_proj/MatMul_smooth_mul multiply [-58.8737f, 4.981f] 22
CustomOp/linalg::channelwiseQuantMatMul<33>140 contrib.epu.quadric_custom_op [-159.195f, 162.484f] 23
/model/layers.35/mlp/Mul multiply [-1002.33f, 1755.35f] 16
/model/layers.35/mlp/down_proj/MatMul_smooth_mul multiply [-73.9084f, 97.6708f] 21
CustomOp/linalg::channelwiseQuantMatMul<29>142 contrib.epu.quadric_custom_op [-4812.36f, 1990.75f] 18
/model/layers.35/Add_1 add [-6990.34f, 2581.91f] 16
/model/norm/Mul_1 contrib.epu.rms_norm [-139.062f, 130.451f] 23
/Gather take [-93.9597f, 66.4554f] 23
/lm_head/MatMul_smooth_mul multiply [-5.11149f, 4.01418f] 24
CustomOp/linalg::channelwiseQuantMatMul<36>144 contrib.epu.quadric_custom_op [-15.0257f, 24.2741f] 26
2026-09-22 03:29 - INFO - epu - codegen - START====================build_cpu_runnable_fx_relay
2026-09-22 03:29 - INFO - epu - codegen - START=======================quantize_to_chimera_fx
2026-09-22 03:32 - INFO - epu - codegen - START=================================relay_to_tir
2026-09-22 03:32 - INFO - epu - codegen - START===========================relay_to_epu_relay
2026-09-22 03:32 - INFO - epu - codegen - START==============================adapt_and_order
2026-09-22 03:33 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-09-22 03:33 - INFO - epu - codegen - START=============================plan_lrm_virtual
2026-09-22 03:36 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-09-22 03:36 - INFO - epu - codegen - START===============================lrm_alloc_loop
2026-09-22 03:41 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-09-22 03:41 - INFO - epu - codegen - START================================lrm_splitting
2026-09-22 03:48 - INFO - epu - codegen - START==============================ext_split_relay
2026-09-22 03:54 - INFO - epu - codegen - START====================================build_tir
2026-09-22 03:54 - INFO - epu - chimera_job - Compilation of qwen3_custom_ops_seq1024_QC_P_1d7_2MB_4kB_128GBps_128GBps_16_OFF_x1_x1 successful
╒═════════════════════╤════════════════════════════════════════════════════════════════════════╕
│ Module Name │ qwen3_custom_ops_seq1024_QC_P_1d7_2MB_4kB_128GBps_128GBps_16_OFF_x1_x1 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────┤
│ ONNX File │ qwen3_custom_ops_seq1024.onnx │
├─────────────────────┼────────────────────────────────────────────────────────────────────────┤
│ Product Target │ QC-P │
├─────────────────────┼────────────────────────────────────────────────────────────────────────┤
│ Number of Cores │ 1 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────┤
│ ISS Clock Frequency │ 1.700 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────┤
│ L2M Size │ 2MB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────┤
│ LRM Size │ 4kB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────┤
│ External Read BW │ 128GBps │
├─────────────────────┼────────────────────────────────────────────────────────────────────────┤
│ External Write BW │ 128GBps │
├─────────────────────┼────────────────────────────────────────────────────────────────────────┤
│ MACS per PE │ 16 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────┤
│ Max L2M │ 0.172MB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────┤
│ Max LRM │ 0.609kB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────┤
│ Max Temp Ext Bytes │ 0.583MB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────┤
│ Network GMACs │ 7.870 │
╘═════════════════════╧════════════════════════════════════════════════════════════════════════╛
╒═════╤════════╤══════════════════════════╤═══════════════════╤══════════════════════════╤═══════╕
│ │ Type │ Name │ shape │ type │ mse │
╞═════╪════════╪══════════════════════════╪═══════════════════╪══════════════════════════╪═══════╡
│ 0 │ Input │ input_ids │ [1, 1] │ tensor[int32] │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 1 │ Input │ attention_mask │ [1, 1, 1024] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 2 │ Input │ sin │ [1, 1, 128] │ tensor[FixedPoint32<30>] │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 3 │ Input │ cos │ [1, 1, 128] │ tensor[FixedPoint32<30>] │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 4 │ Input │ past_key_values.0.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 5 │ Input │ past_key_values.0.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 6 │ Input │ past_key_values.1.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 7 │ Input │ past_key_values.1.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 8 │ Input │ past_key_values.2.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 9 │ Input │ past_key_values.2.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 10 │ Input │ past_key_values.3.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 11 │ Input │ past_key_values.3.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 12 │ Input │ past_key_values.4.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 13 │ Input │ past_key_values.4.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 14 │ Input │ past_key_values.5.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 15 │ Input │ past_key_values.5.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 16 │ Input │ past_key_values.6.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 17 │ Input │ past_key_values.6.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 18 │ Input │ past_key_values.7.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 19 │ Input │ past_key_values.7.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 20 │ Input │ past_key_values.8.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 21 │ Input │ past_key_values.8.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 22 │ Input │ past_key_values.9.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 23 │ Input │ past_key_values.9.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 24 │ Input │ past_key_values.10.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 25 │ Input │ past_key_values.10.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 26 │ Input │ past_key_values.11.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 27 │ Input │ past_key_values.11.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 28 │ Input │ past_key_values.12.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 29 │ Input │ past_key_values.12.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 30 │ Input │ past_key_values.13.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 31 │ Input │ past_key_values.13.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 32 │ Input │ past_key_values.14.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 33 │ Input │ past_key_values.14.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 34 │ Input │ past_key_values.15.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 35 │ Input │ past_key_values.15.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 36 │ Input │ past_key_values.16.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 37 │ Input │ past_key_values.16.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 38 │ Input │ past_key_values.17.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 39 │ Input │ past_key_values.17.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 40 │ Input │ past_key_values.18.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 41 │ Input │ past_key_values.18.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 42 │ Input │ past_key_values.19.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 43 │ Input │ past_key_values.19.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 44 │ Input │ past_key_values.20.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 45 │ Input │ past_key_values.20.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 46 │ Input │ past_key_values.21.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 47 │ Input │ past_key_values.21.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 48 │ Input │ past_key_values.22.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 49 │ Input │ past_key_values.22.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 50 │ Input │ past_key_values.23.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 51 │ Input │ past_key_values.23.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 52 │ Input │ past_key_values.24.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 53 │ Input │ past_key_values.24.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 54 │ Input │ past_key_values.25.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 55 │ Input │ past_key_values.25.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 56 │ Input │ past_key_values.26.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 57 │ Input │ past_key_values.26.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 58 │ Input │ past_key_values.27.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 59 │ Input │ past_key_values.27.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 60 │ Input │ past_key_values.28.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 61 │ Input │ past_key_values.28.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 62 │ Input │ past_key_values.29.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 63 │ Input │ past_key_values.29.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 64 │ Input │ past_key_values.30.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 65 │ Input │ past_key_values.30.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 66 │ Input │ past_key_values.31.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 67 │ Input │ past_key_values.31.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 68 │ Input │ past_key_values.32.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 69 │ Input │ past_key_values.32.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 70 │ Input │ past_key_values.33.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 71 │ Input │ past_key_values.33.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 72 │ Input │ past_key_values.34.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 73 │ Input │ past_key_values.34.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 74 │ Input │ past_key_values.35.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 75 │ Input │ past_key_values.35.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 76 │ Output │ logits │ [1, 151936] │ tensor[FixedPoint32<26>] │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 77 │ Output │ present.0.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 78 │ Output │ present.0.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 79 │ Output │ present.1.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 80 │ Output │ present.1.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 81 │ Output │ present.2.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 82 │ Output │ present.2.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 83 │ Output │ present.3.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 84 │ Output │ present.3.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 85 │ Output │ present.4.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 86 │ Output │ present.4.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 87 │ Output │ present.5.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 88 │ Output │ present.5.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 89 │ Output │ present.6.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 90 │ Output │ present.6.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 91 │ Output │ present.7.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 92 │ Output │ present.7.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 93 │ Output │ present.8.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 94 │ Output │ present.8.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 95 │ Output │ present.9.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 96 │ Output │ present.9.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 97 │ Output │ present.10.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 98 │ Output │ present.10.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 99 │ Output │ present.11.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 100 │ Output │ present.11.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 101 │ Output │ present.12.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 102 │ Output │ present.12.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 103 │ Output │ present.13.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 104 │ Output │ present.13.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 105 │ Output │ present.14.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 106 │ Output │ present.14.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 107 │ Output │ present.15.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 108 │ Output │ present.15.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 109 │ Output │ present.16.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 110 │ Output │ present.16.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 111 │ Output │ present.17.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 112 │ Output │ present.17.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 113 │ Output │ present.18.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 114 │ Output │ present.18.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 115 │ Output │ present.19.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 116 │ Output │ present.19.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 117 │ Output │ present.20.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 118 │ Output │ present.20.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 119 │ Output │ present.21.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 120 │ Output │ present.21.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 121 │ Output │ present.22.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 122 │ Output │ present.22.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 123 │ Output │ present.23.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 124 │ Output │ present.23.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 125 │ Output │ present.24.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 126 │ Output │ present.24.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 127 │ Output │ present.25.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 128 │ Output │ present.25.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 129 │ Output │ present.26.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 130 │ Output │ present.26.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 131 │ Output │ present.27.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 132 │ Output │ present.27.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 133 │ Output │ present.28.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 134 │ Output │ present.28.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 135 │ Output │ present.29.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 136 │ Output │ present.29.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 137 │ Output │ present.30.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 138 │ Output │ present.30.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 139 │ Output │ present.31.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 140 │ Output │ present.31.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 141 │ Output │ present.32.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 142 │ Output │ present.32.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 143 │ Output │ present.33.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 144 │ Output │ present.33.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 145 │ Output │ present.34.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 146 │ Output │ present.34.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 147 │ Output │ present.35.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 148 │ Output │ present.35.value │ [1, 8, 1024, 128] │ n/a │ n/a │
╘═════╧════════╧══════════════════════════╧═══════════════════╧══════════════════════════╧═══════╛
6. Prepare the Prompt
The tokenizer runs on the host, in the examples environment (transformers 4.51 or newer, which is where Qwen3 landed). prepare_inputs.py writes the four files the runner reads: input_logits.bin (token ids), sin.bin / cos.bin (RoPE tables for every position in the context) and prompt_len.bin.
The prompt is deliberately short — "The tallest" is two tokens — because every token costs a full pass through 8 billion parameters on the ISS.
!python3 prepare_inputs.py --prompt "The tallest" --new_tokens 1
2026-09-22 03:54:05.435270: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2026-09-22 03:54:05.436365: I external/local_tsl/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2026-09-22 03:54:05.471947: I external/local_tsl/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2026-09-22 03:54:05.587421: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-09-22 03:54:06.157394: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
[ 785 81617]
Prompt length: 2
7. Run Autoregressive Decode on the ISS
qwen3.cpp is the host-side runner around the CGC-generated kernel. It loads the packed weights, then loops over the prompt and the generated tokens: one kernel launch per token, the KV cache appended on-chip, the next token chosen with an on-chip top-1 over the 151,936 logits — prompt_len + new_tokens − 1 passes in total, all inside a single ISS session. Each generated token is streamed back to the host through a kernel parameter register as soon as it is picked.
sdk source compiles the runner and the kernel with Quadric LLVM for the Chimera GPNPU backend and executes them on the ISS with the 4-core QC-P configuration from section 1. The weights CGC emitted first move next to the runner, where it looks for them.
Compile: ~10 minutes · Run: ~30 minutes
stage_const_tensor("ccl_build/qwen3_custom_ops_seq1024*")
Staged const_tensor_data.bin (6.29 GB of INT4/INT8 weights) for the runner
PosixPath('const_tensor_data.bin')
%%bash
set -euo pipefail
sdk source -v qwen3.cpp \
--include-cgc-headers \
--target QC-P \
--num-cores 4 \
--ocm-size 2MB \
--macs-per-pe 16 \
--clock-freq-ghz 1 \
--ext-read-bw 24GBps \
--ext-write-bw 24GBps \
--ddr-axi-width 256 \
--quiet
2026-09-22 03:54 - DEBUG - sdk - cli - Executing command: cmake CMakeLists.txt -B /quadric/sdk-cli/examples/models/qwen/qwen3_8b/qwen3_QC-P_1d0_2MB_4kB_24GBps_24GBps_16_OFF_x1_x4/build -DNUM_GPNPUS=4 -DNUM_CORES=16 -DNUM_BORDERS=2 -DEPU_VERSION=2.0.0 -DQLLVM_ROOT_PATH=/quadric/llvm -DOCM_SIZE_KIBIBYTES=2048 -DNUM_PE_MACS=16 -DASSERT_MLS_WIDTH_LINE_ALIGN=ON -DSTACKOVERFLOW_ERROR=OFF -DHARDWARE_TARGET=OFF
2026-09-22 03:54 - DEBUG - sdk - cli - Executing command: make -j8
[SDK-CLI] : Executing on QC-P simulator
2026-09-22 03:57 - DEBUG - sdk - cli - Executing command: ./qwen3_host -c --ddrRdBwTotal 196608.0 --ddrWrBwTotal 196608.0 --ddrAxiWidth 256 --instMemDepth 1310720 --ocmSize 2097152 --cycleTimeNS 1.0 --no-check --ddrRdAvgPct 100 --ddrRdMaxPct 100 --ddrWrAvgPct 100 --ddrWrMaxPct 100 --postKernelFlowTimeoutCycles 4000000 --clusterSize 4 --numClusters 1
2026-09-22 04:06 - WARNING - epu - core - No profile.json found. Not able to show performance results.
2026-09-22 04:06 - INFO - epu - core - If you would like to see performance results, add profiling statements to your source code.
[SDK-CLI] : Execution completed.
8. Decode the Output and Read the Profile
output_seq.bin holds the generated token ids; the tokenizer turns them back into text. The ISS profile then gives the cycle breakdown of core 0 — compute, MAC, data movement, stalls — and, from the total across all four cores, the per-token throughput. The run above made two passes through the network (a two-token prompt plus one generated token), so the cycles per token are the total divided by two.
!python3 decode_outputs.py
Token ids: [4752]
Generated text: building
plot_decode_profile("qwen3_QC-P*/**/profile_core0.json", clock_freq_ghz=1.0)
[SDK-CLI] : TotalCycles: 100,556,349
[SDK-CLI] : Executions/second: 10
compute : ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 18.15M
data_array : ▇▇▇▇▇▇▇ 7.121M
mac : ▇ 1.077M
data_ocm : ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 44.984M
data_external: ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 29.222M
'qwen3_QC-P_1d0_2MB_4kB_24GBps_24GBps_16_OFF_x1_x4/output/profile_core0.json'
prompt_tokens = int(np.fromfile("prompt_len.bin", dtype=np.int32)[0])
report_decode_throughput(
"qwen3_QC-P*/**/profile_core*.json",
passes=prompt_tokens + NEW_TOKENS - 1,
clock_freq_ghz=1.0,
silicon_clock_ghz=1.7,
)
Total cycles : 100,556,369 (2 passes, 4 core(s))
Cycles per token : 50,278,184
@ 1.0 GHz ISS clock : 50.3 ms/token -> 19.9 tokens/s
@ 1.7 GHz silicon clock: 29.6 ms/token -> 33.8 tokens/s
50278184.5
9. Compare Decode Against ONNX Runtime
A generated token that reads well is not a correctness check. The reference is the same shape-fixed graph before custom-op replacement, run in ONNX Runtime on the host: the prompt is replayed token by token with the same left-padded KV cache and the same per-position RoPE tables the runner consumed, and the logits of the last prompt pass are compared with the ones the runner wrote to output_logits.bin. The gate: the top-1 token must match, at least three of the top-5 tokens must be shared, and the two logit vectors must correlate with Pearson r ≥ 0.7 — so a divergence anywhere in the decoder stack surfaces here, not in a sentence someone has to judge by eye. Fixed-point custom kernels do not reproduce ONNX Runtime's fp32 execution of the quantized graph bit for bit, which is why the correlation threshold, not equality, is the gate.
compare_decode_with_ort(decode_seq_onnx)
Prompt tokens : [785, 81617]
ONNX Runtime top-5 : [4752, 315, 11, 883, 825]
ISS top-5 : [4752, 16301, 883, 11, 323]
Top-1 match : True top-5 overlap: 3/5
Logit Pearson r : 0.7655 max |diff|: 12.054
PASS: ISS decode matches the ONNX Runtime reference
{'ort_top5': [4752, 315, 11, 883, 825],
'iss_top5': [4752, 16301, 883, 11, 323],
'pearson': 0.7654599343142197,
'top5_overlap': 3}
10. Prefill: Validate One Decoder Layer on the ISS
Prefill pushes the whole 1024-token prompt through the network at once, so a single decoder layer already costs on the order of a billion ISS cycles. Rather than run all 36 layers, the pipeline in prefill/ validates the network the way it is built — one decoder at a time, each an independent subgraph with well-defined inputs and outputs:
- Cut decoder N out of the prefill graph (hidden state, attention mask,
cos,sinand KV cache in → hidden state and KV cache out) - Replace its ops with the prefill custom ops —
nn::qwen3PrefillAttention,nn::gateUpProjection,linalg::channelwiseQuantMatMul - Compile with CGC for the same 4-core QC-P target
- Run on the ISS through
qwen3_prefill_runner.cpp - Compare against ONNX Runtime with a tolerance derived from the layer's tensor ranges (
atol + rtol · |ref|), plus the Pearson correlation and a plot of the two output distributions
Feeding each decoder ONNX Runtime's output for the previous layer (isolated mode) exercises the kernels with a clean input, so any mismatch points at that layer alone. The same pipeline also runs in chained mode — the GPNPU's own output fed forward, gated on per-token cosine similarity — to measure how quantization noise accumulates with depth.
The prefill graph is downloaded pre-built: the same W4A8 export, shape-fixed with mode="prefill", with calibrated tensor ranges that set each decoder's fixed-point input and output formats.
prefill_onnx, _prefill_weights, prefill_tranges = map(str, download_prefill_model(PREFILL_DIR))
Downloading qwen3_8b_prefill_1024_context.onnx...
Downloaded qwen3_8b_prefill_1024_context.onnx (0.01 GB)
Downloading 27fe33ec-1773-11f1-b0a3-ea22fe300241...
Downloaded 27fe33ec-1773-11f1-b0a3-ea22fe300241 (6.27 GB)
Downloading qwen_0_36_calibrated.onnx.tranges...
Downloaded qwen_0_36_calibrated.onnx.tranges (0.00 GB)
Prefill tokenizes a real 1024-token prompt (a WikiText passage) rather than two words. The token ids, the RoPE tables and the prompt length land in prefill/, where the per-decoder runner reads them; decoder 0 also gets its own copy of the token ids since it consumes ids rather than a hidden state.
prepare_prefill_prompt(PREFILL_DIR / "prompts" / "0.txt", PREFILL_DIR)
Tokenized 0.txt -> prefill/{input_logits, sin, cos, prompt_len}.bin
Every decoder is cut on the same six input edges and three output edges. Decoder 0 takes the token ids; every later decoder takes the previous layer's residual-stream output:
source = (PREFILL_DIR / "cut_model.py").read_text()
edges = re.search(
r"def get_decoder_io_edges.*?return input_edges, output_edges\n", source, re.DOTALL
)
print(edges.group(0))
def get_decoder_io_edges(decoder_number):
"""Return (input_edges, output_edges) for cutting decoder N from Qwen3 prefill."""
hidden_input = (
"input_ids" if decoder_number == 0 else f"/model/layers.{decoder_number - 1}/Add_1_output_0"
)
input_edges = [
hidden_input,
"attention_mask",
"cos",
"sin",
f"past_key_values.{decoder_number}.key",
f"past_key_values.{decoder_number}.value",
]
output_edges = [
f"/model/layers.{decoder_number}/Add_1_output_0",
f"present.{decoder_number}.key",
f"present.{decoder_number}.value",
]
return input_edges, output_edges
Decoder 2 is the layer under test — a representative middle layer; decoders 1–34 share one compiled kernel. Its isolated-mode input is ONNX Runtime's output for decoders 0 → 1, so the bootstrap first runs decoders 0–2 through ONNX Runtime on the host, cutting each from the prefill graph and chaining the outputs forward as fixed-point inputs.
Runtime: a few minutes per decoder
VALIDATE_DECODER = 2
for decoder in range(VALIDATE_DECODER + 1):
run_pipeline(
decoder=decoder,
ort_only=True,
input_model=prefill_onnx,
tranges=prefill_tranges,
data_dir=str(PREFILL_DIR),
)
=== Decoder 0 pipeline (mode=isolated, cores=4) ===
════════════════════════════════════════════════════════════════
── Step 1: Cut decoder 0 from prefill/qwen3_8b_prefill_1024_context.onnx ──
Cutting decoder 0:
Input edges: ['input_ids', 'attention_mask', 'cos', 'sin', 'past_key_values.0.key', 'past_key_values.0.value']
Output edges: ['/model/layers.0/Add_1_output_0', 'present.0.key', 'present.0.value']
Successfully saved to /quadric/sdk-cli/examples/models/qwen/qwen3_8b/prefill/decoder_0/decoder_0.onnx
── Step 6: Run ORT (--ort-only) ──
✓ ORT complete → /quadric/sdk-cli/examples/models/qwen/qwen3_8b/prefill/decoder_0/onnx_runtime_outputs/output_0.npy
── Step 9: Chain output to decoder 1 ──
✓ Chained ORT 0 → 1 (FixedPoint32<26>, 16.0 MB)
✓ Chained: decoder_0/onnx_runtime_outputs/output_0.npy → decoder_1/input_logits_ort_float.bin
════════════════════════════════════════════════════════════════
Decoder 0 ort-only bootstrap complete
════════════════════════════════════════════════════════════════
=== Decoder 1 pipeline (mode=isolated, cores=4) ===
════════════════════════════════════════════════════════════════
── Step 1: Cut decoder 1 from prefill/qwen3_8b_prefill_1024_context.onnx ──
Cutting decoder 1:
Input edges: ['/model/layers.0/Add_1_output_0', 'attention_mask', 'cos', 'sin', 'past_key_values.1.key', 'past_key_values.1.value']
Output edges: ['/model/layers.1/Add_1_output_0', 'present.1.key', 'present.1.value']
Successfully saved to /quadric/sdk-cli/examples/models/qwen/qwen3_8b/prefill/decoder_1/decoder_1.onnx
── Step 6: Run ORT (--ort-only) ──
✓ ORT complete → /quadric/sdk-cli/examples/models/qwen/qwen3_8b/prefill/decoder_1/onnx_runtime_outputs/output_0.npy
── Step 9: Chain output to decoder 2 ──
✓ Chained ORT 1 → 2 (FixedPoint32<25>, 16.0 MB)
✓ Chained: decoder_1/onnx_runtime_outputs/output_0.npy → decoder_2/input_logits_ort_float.bin
════════════════════════════════════════════════════════════════
Decoder 1 ort-only bootstrap complete
════════════════════════════════════════════════════════════════
=== Decoder 2 pipeline (mode=isolated, cores=4) ===
════════════════════════════════════════════════════════════════
── Step 1: Cut decoder 2 from prefill/qwen3_8b_prefill_1024_context.onnx ──
Cutting decoder 2:
Input edges: ['/model/layers.1/Add_1_output_0', 'attention_mask', 'cos', 'sin', 'past_key_values.2.key', 'past_key_values.2.value']
Output edges: ['/model/layers.2/Add_1_output_0', 'present.2.key', 'present.2.value']
Successfully saved to /quadric/sdk-cli/examples/models/qwen/qwen3_8b/prefill/decoder_2/decoder_2.onnx
── Step 6: Run ORT (--ort-only) ──
✓ ORT complete → /quadric/sdk-cli/examples/models/qwen/qwen3_8b/prefill/decoder_2/onnx_runtime_outputs/output_0.npy
── Step 9: Chain output to decoder 3 ──
✓ Chained ORT 2 → 3 (FixedPoint32<24>, 16.0 MB)
✓ Chained: decoder_2/onnx_runtime_outputs/output_0.npy → decoder_3/input_logits_ort_float.bin
════════════════════════════════════════════════════════════════
Decoder 2 ort-only bootstrap complete
════════════════════════════════════════════════════════════════
Now the GPNPU side: custom ops, CGC compile, sdk source on the ISS, and the comparison. The gate passes when at most 50 of the 4.19 million output elements fall outside atol + rtol · |ref|, with atol derived from the layer's output range and rtol = 1 %; the printout also reports the mean and maximum absolute difference and the Pearson correlation (typically ≥ 0.998).
Compile: ~10 minutes · Run: ~30 minutes
run_pipeline(
decoder=VALIDATE_DECODER,
mode="isolated",
skip_cut=True, # the bootstrap already cut decoder_2.onnx
tranges=prefill_tranges,
rtol=1e-2,
max_failures=50,
data_dir=str(PREFILL_DIR),
)
=== Decoder 2 pipeline (mode=isolated, cores=4) ===
── Step 1: Skipping cut (--skip-cut) ──
── Step 2: Custom op matching ──
Warning: /model/layers.2/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Processing matmul ops for decoder 2...
Decoder 2 gate_up frac bits:
gate_combined_scale=5.992835e-02 frac_bits=35
up_combined_scale=5.992835e-02 frac_bits=35
gate_im_frac_bits=27
up_im_frac_bits=27
gate_output_frac_bits=24
up_output_frac_bits=24
Successfully saved modified model to /quadric/sdk-cli/examples/models/qwen/qwen3_8b/prefill/decoder_2/custom_op_decoder_2.onnx
✓ Custom op complete → /quadric/sdk-cli/examples/models/qwen/qwen3_8b/prefill/decoder_2/custom_op_decoder_2.onnx
── Step 3: CGC compile ──
Starting CGC compilation...
2026-09-22 04:11 - INFO - epu - chimera_job - START==================================onnx_ingest
2026-09-22 04:11 - INFO - epu - chimera_job - Numerical ranges provided
2026-09-22 04:11 - INFO - epu - codegen - START===============================optimize_relay
2026-09-22 04:11 - INFO - epu - codegen - START====================quantize_to_cpu_runnable_fx
2026-09-22 04:11 - INFO - epu - fx -
Source name Op Output 0 Range Output 0 Frac Bits
------------------------------------------------- ----------------------------- ---------------------- --------------------
/model/layers.2/input_layernorm/Mul_1 contrib.epu.rms_norm [-0.862898f, 1.48399f] 29
CustomOp/linalg::channelwiseQuantMatMul<42>2 contrib.epu.quadric_custom_op [-1.03252f, 1.31911f] 29
CustomOp/add<MultiCoreMode::All>2 contrib.epu.quadric_custom_op [-18.1085f, 57.4384f] 25
/model/layers.2/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-82.5232f, 81.3135f] 24
CustomOp/nn::gateUpProjection<27,27,24,24,35,35>0 contrib.epu.quadric_custom_op [-26.306f, 24.5606f] 26
CustomOp/linalg::channelwiseQuantMatMul<37>1 contrib.epu.quadric_custom_op [-10.8904f, 26.5761f] 26
CustomOp/add<MultiCoreMode::All>1 contrib.epu.quadric_custom_op [-23.2706f, 81.9635f] 24
2026-09-22 04:11 - INFO - epu - codegen - START====================build_cpu_runnable_fx_relay
2026-09-22 04:11 - INFO - epu - codegen - START=======================quantize_to_chimera_fx
2026-09-22 04:11 - INFO - epu - codegen - START=================================relay_to_tir
2026-09-22 04:11 - INFO - epu - codegen - START===========================relay_to_epu_relay
2026-09-22 04:11 - INFO - epu - codegen - START==============================adapt_and_order
2026-09-22 04:11 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-09-22 04:11 - INFO - epu - codegen - START=============================plan_lrm_virtual
2026-09-22 04:11 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-09-22 04:11 - INFO - epu - codegen - START===============================lrm_alloc_loop
2026-09-22 04:11 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-09-22 04:11 - INFO - epu - codegen - START================================lrm_splitting
2026-09-22 04:11 - INFO - epu - codegen - START==============================ext_split_relay
2026-09-22 04:11 - INFO - epu - codegen - START====================================build_tir
2026-09-22 04:11 - INFO - epu - chimera_job - Compilation of custom_op_decoder_2_QC_P_1d7_2MB_4kB_128GBps_128GBps_16_OFF_x1_x4 successful
Compilation complete!
╒═════════════════════╤═══════════════════════════════════════════════════════════════════════════════════════════╕
│ Module Name │ custom_op_decoder_2_QC_P_1d7_2MB_4kB_128GBps_128GBps_16_OFF_x1_x4 │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ ONNX File │ /quadric/sdk-cli/examples/models/qwen/qwen3_8b/prefill/decoder_2/custom_op_decoder_2.onnx │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ Product Target │ QC-P │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ Number of Cores │ 4 │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ ISS Clock Frequency │ 1.700 │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ L2M Size │ 2MB │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ LRM Size │ 4kB │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ External Read BW │ 128GBps │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ External Write BW │ 128GBps │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ MACS per PE │ 16 │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ Max L2M │ 0.000MB │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ Max LRM │ 0.000kB │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ Max Temp Ext Bytes │ 96.000MB │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ Network GMACs │ 206.158 │
╘═════════════════════╧═══════════════════════════════════════════════════════════════════════════════════════════╛
╒════╤════════╤════════════════════════════════╤═══════════════════╤══════════════════════════╤═══════╕
│ │ Type │ Name │ shape │ type │ mse │
╞════╪════════╪════════════════════════════════╪═══════════════════╪══════════════════════════╪═══════╡
│ 0 │ Input │ /model/layers.1/Add_1_output_0 │ [1, 1024, 4096] │ tensor[FixedPoint32<25>] │ n/a │
├────┼────────┼────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 1 │ Input │ attention_mask │ [1, 1024, 1024] │ n/a │ n/a │
├────┼────────┼────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 2 │ Input │ cos │ [1, 1024, 128] │ tensor[FixedPoint32<30>] │ n/a │
├────┼────────┼────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 3 │ Input │ sin │ [1, 1024, 128] │ tensor[FixedPoint32<30>] │ n/a │
├────┼────────┼────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 4 │ Input │ past_key_values.2.key │ [1, 8, 0, 128] │ n/a │ n/a │
├────┼────────┼────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 5 │ Input │ past_key_values.2.value │ [1, 8, 0, 128] │ n/a │ n/a │
├────┼────────┼────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 6 │ Output │ /model/layers.2/Add_1_output_0 │ [1, 1024, 4096] │ tensor[FixedPoint32<24>] │ n/a │
├────┼────────┼────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 7 │ Output │ present.2.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├────┼────────┼────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 8 │ Output │ present.2.value │ [1, 8, 1024, 128] │ n/a │ n/a │
╘════╧════════╧════════════════════════════════╧═══════════════════╧══════════════════════════╧═══════╛
✓ CGC compile complete
── Step 4: Generate decoder_config.hpp + runner symlink ──
✓ Config + symlink ready → /quadric/sdk-cli/examples/models/qwen/qwen3_8b/prefill/decoder_2/qwen3_prefill_runner.cpp
── Step 5: SDK compile & ISS run ──
$ sdk source --quiet /quadric/sdk-cli/examples/models/qwen/qwen3_8b/prefill/decoder_2/qwen3_prefill_runner.cpp --include-cgc-headers --target QC-P --num-cores 4 --ocm-size 2MB --macs-per-pe 16 --clock-freq-ghz 1 --ext-read-bw 24GBps --ext-write-bw 24GBps --ddr-axi-width 256
✓ SDK compile complete → /quadric/sdk-cli/examples/models/qwen/qwen3_8b/prefill/decoder_2/epu_outputs/output_isolated.bin
── Step 6: Validate against ONNX Runtime ──
auto-atol: p_flip=0.187% auto=0.2252 base=0.6067 -> atol=0.6067
GPNPU vs ORT: max|d|=0.9108 mean|d|=0.0115 | atol=0.607 rtol=0.01 fail=0/4194304 (0.000%) [PASS]
✓ Validation complete
── Step 7: Plot output distributions ──
GPNPU min=-20.0026 max=62.1398 std=0.3093
ORT min=-20.0026 max=62.1398 std=0.3092
Scale factor (GPNPU std / ORT std): 1.00x
Pearson r (raw): 0.9986
Pearson r (scaled): 0.9986
Saved → decoder_2/output_comparison.png
✓ Plot saved → /quadric/sdk-cli/examples/models/qwen/qwen3_8b/prefill/decoder_2/output_comparison.png
── Step 8: Update decoder report ──
Warning: tranges lookup failed (No module named 'build_decoder_config'), using wrapper C++ frac bits
Updated /quadric/sdk-cli/examples/models/qwen/qwen3_8b/prefill/decoder_report.md
decoder 2: in_fb=25, out_fb=24, mean_diff=0.0115, max_diff=0.9108, combined%=0.000%, result=PASS, scale=1.000x, r=0.9986
✓ Report updated → decoder_report.md
── Step 9: Chain output to decoder 3 (mode: isolated) ──
✓ Chained ORT 2 → 3 (FixedPoint32<24>, 16.0 MB)
✓ Chained: /quadric/sdk-cli/examples/models/qwen/qwen3_8b/prefill/decoder_2/onnx_runtime_outputs/output_ort.npy → decoder_3/input_logits_ort.bin (FixedPoint32<24>)
mean_diff : 0.0115
max_diff : 0.9108
n_fail : 0 (0.000%)
pearson_r : 0.9986
result : PASS
── Profile Summary (from iss_run_log.txt) ──
Regions × Cores : 8 × 4
Total cycles : 102,162,105 (Σ regions, max across cores)
MAC cycles : 2,765,876 (2.7% of total)
The pipeline saves a distribution overlap, scatter and error histograms for the two outputs — a quick visual check that the GPNPU's fixed-point output sits on the ONNX Runtime reference across the full value range, not just on average:
display(Image(filename=str(PREFILL_DIR / f"decoder_{VALIDATE_DECODER}" / "output_comparison.png")))
11. Prefill: Compile All 36 Decoders
With one layer validated, the same custom-op replacement runs over the complete prefill graph — all 36 decoders plus the LM head projection (include_lm_head=True) — and CGC compiles the result into the full prefill program. This is the artifact a deployment ships alongside the decode kernel: prompt in, first-token logits and a filled KV cache out. Like the decode compile, this step uses the export's tensor ranges.
The measured decoder from section 10 also gives a first read on time to first token. Decoders 1–34 share one compiled kernel, so 36 × one layer's cycles bounds the prefill latency from above — a fused 36-layer build amortizes the constant-tensor loads across layer boundaries and comes in at or below this sum. Attention makes prefill roughly quadratic in prompt length, so the numbers are specific to 1024 tokens.
Memory: ~10 GB peak · Runtime: under a minute for the custom ops, ~10 minutes for CGC
prefill_custom_onnx = str(PREFILL_DIR / "qwen3_custom_op_all_dec.onnx")
qwen3_prefill_custom_op_replacer(
prefill_onnx,
prefill_custom_onnx,
tranges_path,
num_heads=NUM_KV_HEADS,
embed_dim=EMBED_DIM,
seq_length=SEQ_LEN,
num_decoders=NUM_DECODERS,
include_lm_head=True,
)
Warning: /model/layers.0/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.1/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.2/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.3/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.4/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.5/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.6/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.7/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.8/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.9/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.10/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.11/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.12/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.13/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.14/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.15/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.16/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.17/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.18/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.19/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.20/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.21/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.22/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.23/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.24/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.25/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.26/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.27/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.28/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.29/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.30/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.31/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.32/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.33/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.34/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.35/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Processing matmul ops for decoder 0...
Decoder 0 gate_up frac bits:
gate_combined_scale=5.489234e-04 frac_bits=41
up_combined_scale=3.307317e-04 frac_bits=42
gate_im_frac_bits=31
up_im_frac_bits=31
gate_output_frac_bits=28
up_output_frac_bits=28
Processing matmul ops for decoder 1...
Decoder 1 gate_up frac bits:
gate_combined_scale=7.408716e-03 frac_bits=38
up_combined_scale=3.896641e-03 frac_bits=39
gate_im_frac_bits=28
up_im_frac_bits=29
gate_output_frac_bits=27
up_output_frac_bits=26
Processing matmul ops for decoder 2...
Decoder 2 gate_up frac bits:
gate_combined_scale=5.992835e-02 frac_bits=35
up_combined_scale=5.992835e-02 frac_bits=35
gate_im_frac_bits=27
up_im_frac_bits=27
gate_output_frac_bits=24
up_output_frac_bits=24
Processing matmul ops for decoder 3...
Decoder 3 gate_up frac bits:
gate_combined_scale=1.153846e-02 frac_bits=37
up_combined_scale=9.374998e-03 frac_bits=37
gate_im_frac_bits=28
up_im_frac_bits=28
gate_output_frac_bits=26
up_output_frac_bits=26
Processing matmul ops for decoder 4...
Decoder 4 gate_up frac bits:
gate_combined_scale=1.012771e-02 frac_bits=37
up_combined_scale=1.079110e-02 frac_bits=37
gate_im_frac_bits=28
up_im_frac_bits=28
gate_output_frac_bits=27
up_output_frac_bits=27
Processing matmul ops for decoder 5...
Decoder 5 gate_up frac bits:
gate_combined_scale=1.041709e-02 frac_bits=37
up_combined_scale=5.855937e-03 frac_bits=38
gate_im_frac_bits=28
up_im_frac_bits=29
gate_output_frac_bits=27
up_output_frac_bits=27
Processing matmul ops for decoder 6...
Decoder 6 gate_up frac bits:
gate_combined_scale=6.929483e-02 frac_bits=34
up_combined_scale=7.354605e-02 frac_bits=34
gate_im_frac_bits=27
up_im_frac_bits=27
gate_output_frac_bits=24
up_output_frac_bits=24
Processing matmul ops for decoder 7...
Decoder 7 gate_up frac bits:
gate_combined_scale=7.519060e-03 frac_bits=38
up_combined_scale=1.755872e-03 frac_bits=40
gate_im_frac_bits=28
up_im_frac_bits=29
gate_output_frac_bits=27
up_output_frac_bits=27
Processing matmul ops for decoder 8...
Decoder 8 gate_up frac bits:
gate_combined_scale=2.171312e-03 frac_bits=39
up_combined_scale=1.607593e-03 frac_bits=40
gate_im_frac_bits=29
up_im_frac_bits=30
gate_output_frac_bits=27
up_output_frac_bits=28
Processing matmul ops for decoder 9...
Decoder 9 gate_up frac bits:
gate_combined_scale=2.659734e-03 frac_bits=39
up_combined_scale=2.695962e-03 frac_bits=39
gate_im_frac_bits=29
up_im_frac_bits=30
gate_output_frac_bits=27
up_output_frac_bits=28
Processing matmul ops for decoder 10...
Decoder 10 gate_up frac bits:
gate_combined_scale=2.187623e-03 frac_bits=39
up_combined_scale=1.581833e-03 frac_bits=40
gate_im_frac_bits=29
up_im_frac_bits=30
gate_output_frac_bits=27
up_output_frac_bits=27
Processing matmul ops for decoder 11...
Decoder 11 gate_up frac bits:
gate_combined_scale=1.958575e-03 frac_bits=39
up_combined_scale=1.510549e-03 frac_bits=40
gate_im_frac_bits=29
up_im_frac_bits=30
gate_output_frac_bits=27
up_output_frac_bits=27
Processing matmul ops for decoder 12...
Decoder 12 gate_up frac bits:
gate_combined_scale=1.847970e-03 frac_bits=40
up_combined_scale=2.534580e-03 frac_bits=39
gate_im_frac_bits=29
up_im_frac_bits=30
gate_output_frac_bits=27
up_output_frac_bits=27
Processing matmul ops for decoder 13...
Decoder 13 gate_up frac bits:
gate_combined_scale=1.677463e-03 frac_bits=40
up_combined_scale=3.220730e-03 frac_bits=39
gate_im_frac_bits=30
up_im_frac_bits=29
gate_output_frac_bits=26
up_output_frac_bits=25
Processing matmul ops for decoder 14...
Decoder 14 gate_up frac bits:
gate_combined_scale=2.226458e-03 frac_bits=39
up_combined_scale=2.970458e-03 frac_bits=39
gate_im_frac_bits=30
up_im_frac_bits=29
gate_output_frac_bits=26
up_output_frac_bits=25
Processing matmul ops for decoder 15...
Decoder 15 gate_up frac bits:
gate_combined_scale=2.298211e-03 frac_bits=39
up_combined_scale=6.424773e-03 frac_bits=38
gate_im_frac_bits=29
up_im_frac_bits=29
gate_output_frac_bits=26
up_output_frac_bits=25
Processing matmul ops for decoder 16...
Decoder 16 gate_up frac bits:
gate_combined_scale=8.712117e-03 frac_bits=37
up_combined_scale=9.366387e-03 frac_bits=37
gate_im_frac_bits=28
up_im_frac_bits=28
gate_output_frac_bits=23
up_output_frac_bits=24
Processing matmul ops for decoder 17...
Decoder 17 gate_up frac bits:
gate_combined_scale=2.006680e-03 frac_bits=39
up_combined_scale=1.808745e-03 frac_bits=40
gate_im_frac_bits=30
up_im_frac_bits=30
gate_output_frac_bits=27
up_output_frac_bits=25
Processing matmul ops for decoder 18...
Decoder 18 gate_up frac bits:
gate_combined_scale=2.155281e-03 frac_bits=39
up_combined_scale=1.964622e-03 frac_bits=39
gate_im_frac_bits=29
up_im_frac_bits=30
gate_output_frac_bits=27
up_output_frac_bits=26
Processing matmul ops for decoder 19...
Decoder 19 gate_up frac bits:
gate_combined_scale=1.855934e-03 frac_bits=40
up_combined_scale=2.282373e-03 frac_bits=39
gate_im_frac_bits=30
up_im_frac_bits=30
gate_output_frac_bits=27
up_output_frac_bits=28
Processing matmul ops for decoder 20...
Decoder 20 gate_up frac bits:
gate_combined_scale=1.904722e-03 frac_bits=40
up_combined_scale=1.601911e-03 frac_bits=40
gate_im_frac_bits=29
up_im_frac_bits=29
gate_output_frac_bits=27
up_output_frac_bits=27
Processing matmul ops for decoder 21...
Decoder 21 gate_up frac bits:
gate_combined_scale=2.035758e-03 frac_bits=39
up_combined_scale=2.340634e-03 frac_bits=39
gate_im_frac_bits=30
up_im_frac_bits=29
gate_output_frac_bits=27
up_output_frac_bits=27
Processing matmul ops for decoder 22...
Decoder 22 gate_up frac bits:
gate_combined_scale=2.405553e-03 frac_bits=39
up_combined_scale=2.320086e-03 frac_bits=39
gate_im_frac_bits=30
up_im_frac_bits=30
gate_output_frac_bits=26
up_output_frac_bits=27
Processing matmul ops for decoder 23...
Decoder 23 gate_up frac bits:
gate_combined_scale=2.210377e-03 frac_bits=39
up_combined_scale=2.788087e-03 frac_bits=39
gate_im_frac_bits=30
up_im_frac_bits=29
gate_output_frac_bits=26
up_output_frac_bits=27
Processing matmul ops for decoder 24...
Decoder 24 gate_up frac bits:
gate_combined_scale=3.017297e-03 frac_bits=39
up_combined_scale=3.114498e-03 frac_bits=39
gate_im_frac_bits=29
up_im_frac_bits=29
gate_output_frac_bits=26
up_output_frac_bits=27
Processing matmul ops for decoder 25...
Decoder 25 gate_up frac bits:
gate_combined_scale=3.542291e-03 frac_bits=39
up_combined_scale=3.030597e-03 frac_bits=39
gate_im_frac_bits=29
up_im_frac_bits=29
gate_output_frac_bits=26
up_output_frac_bits=26
Processing matmul ops for decoder 26...
Decoder 26 gate_up frac bits:
gate_combined_scale=2.672773e-03 frac_bits=39
up_combined_scale=4.103271e-03 frac_bits=38
gate_im_frac_bits=29
up_im_frac_bits=29
gate_output_frac_bits=26
up_output_frac_bits=27
Processing matmul ops for decoder 27...
Decoder 27 gate_up frac bits:
gate_combined_scale=3.190623e-03 frac_bits=39
up_combined_scale=4.202626e-03 frac_bits=38
gate_im_frac_bits=29
up_im_frac_bits=29
gate_output_frac_bits=26
up_output_frac_bits=27
Processing matmul ops for decoder 28...
Decoder 28 gate_up frac bits:
gate_combined_scale=4.199372e-03 frac_bits=38
up_combined_scale=4.026828e-03 frac_bits=38
gate_im_frac_bits=29
up_im_frac_bits=29
gate_output_frac_bits=26
up_output_frac_bits=26
Processing matmul ops for decoder 29...
Decoder 29 gate_up frac bits:
gate_combined_scale=3.686063e-03 frac_bits=39
up_combined_scale=4.187949e-03 frac_bits=38
gate_im_frac_bits=30
up_im_frac_bits=29
gate_output_frac_bits=26
up_output_frac_bits=26
Processing matmul ops for decoder 30...
Decoder 30 gate_up frac bits:
gate_combined_scale=4.159406e-03 frac_bits=38
up_combined_scale=7.111935e-03 frac_bits=38
gate_im_frac_bits=29
up_im_frac_bits=28
gate_output_frac_bits=26
up_output_frac_bits=26
Processing matmul ops for decoder 31...
Decoder 31 gate_up frac bits:
gate_combined_scale=4.884809e-03 frac_bits=38
up_combined_scale=8.509487e-03 frac_bits=37
gate_im_frac_bits=29
up_im_frac_bits=28
gate_output_frac_bits=26
up_output_frac_bits=26
Processing matmul ops for decoder 32...
Decoder 32 gate_up frac bits:
gate_combined_scale=5.781837e-03 frac_bits=38
up_combined_scale=7.929652e-03 frac_bits=37
gate_im_frac_bits=28
up_im_frac_bits=28
gate_output_frac_bits=26
up_output_frac_bits=26
Processing matmul ops for decoder 33...
Decoder 33 gate_up frac bits:
gate_combined_scale=9.199777e-03 frac_bits=37
up_combined_scale=2.238982e-02 frac_bits=36
gate_im_frac_bits=28
up_im_frac_bits=28
gate_output_frac_bits=26
up_output_frac_bits=26
Processing matmul ops for decoder 34...
Decoder 34 gate_up frac bits:
gate_combined_scale=4.481895e-02 frac_bits=35
up_combined_scale=6.740215e-02 frac_bits=34
gate_im_frac_bits=27
up_im_frac_bits=27
gate_output_frac_bits=25
up_output_frac_bits=24
Processing matmul ops for decoder 35...
Decoder 35 gate_up frac bits:
gate_combined_scale=1.067153e-01 frac_bits=34
up_combined_scale=1.776727e-01 frac_bits=33
gate_im_frac_bits=25
up_im_frac_bits=25
gate_output_frac_bits=24
up_output_frac_bits=23
Successfully saved modified model to /quadric/sdk-cli/examples/models/qwen/qwen3_8b/prefill/qwen3_custom_op_all_dec.onnx
prefill_job = ChimeraJob(
prefill_custom_onnx,
hw_config=HW_CONFIG,
trange_file=tranges_path,
target_lang="QIL",
io_to_ignore=kv_cache_io_names(NUM_DECODERS),
)
prefill_job.compile()
print(prefill_job)
2026-09-22 04:20 - INFO - epu - chimera_job - START==================================onnx_ingest
2026-09-22 04:20 - INFO - epu - chimera_job - Numerical ranges provided
/usr/local/lib/python3.10/dist-packages/tvm/relay/frontend/onnx.py:6272: UserWarning: This protobuf of onnx model is too large (>2GB). Call check_model with model path instead.
warnings.warn(str(e))
2026-09-22 04:21 - INFO - epu - codegen - START===============================optimize_relay
2026-09-22 04:21 - INFO - epu - codegen - START====================quantize_to_cpu_runnable_fx
2026-09-22 04:21 - INFO - epu - fx -
Source name Op Output 0 Range Output 0 Frac Bits
--------------------------------------------------- ----------------------------- ---------------------- --------------------
/model/embed_tokens/Gather contrib.epu.embedding [-0.628906f, 0.8125f] 31
/model/layers.0/input_layernorm/Mul_1 contrib.epu.rms_norm [-0.359118f, 0.37561f] 31
CustomOp/linalg::channelwiseQuantMatMul<43>2 contrib.epu.quadric_custom_op [-2.25614f, 3.85482f] 28
CustomOp/add<MultiCoreMode::All>2 contrib.epu.quadric_custom_op [-2.30253f, 4.25466f] 28
/model/layers.0/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-1.23024f, 1.59211f] 29
CustomOp/nn::gateUpProjection<31,31,28,28,41,42>0 contrib.epu.quadric_custom_op [-11.3828f, 9.95375f] 27
CustomOp/linalg::channelwiseQuantMatMul<38>1 contrib.epu.quadric_custom_op [-5.18762f, 16.7897f] 26
CustomOp/add<MultiCoreMode::All>1 contrib.epu.quadric_custom_op [-6.67269f, 19.2419f] 26
/model/layers.1/input_layernorm/Mul_1 contrib.epu.rms_norm [-1.30106f, 1.14032f] 30
CustomOp/linalg::channelwiseQuantMatMul<43>5 contrib.epu.quadric_custom_op [-1.41003f, 1.53298f] 29
CustomOp/add<MultiCoreMode::All>5 contrib.epu.quadric_custom_op [-6.78994f, 19.2634f] 26
/model/layers.1/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-51.933f, 47.6116f] 25
CustomOp/nn::gateUpProjection<28,29,27,26,38,39>3 contrib.epu.quadric_custom_op [-22.4947f, 19.6069f] 26
CustomOp/linalg::channelwiseQuantMatMul<37>4 contrib.epu.quadric_custom_op [-12.4679f, 38.1379f] 25
CustomOp/add<MultiCoreMode::All>4 contrib.epu.quadric_custom_op [-18.1108f, 57.4013f] 25
/model/layers.2/input_layernorm/Mul_1 contrib.epu.rms_norm [-0.862898f, 1.28555f] 29
CustomOp/linalg::channelwiseQuantMatMul<42>8 contrib.epu.quadric_custom_op [-1.03252f, 1.31911f] 29
CustomOp/add<MultiCoreMode::All>8 contrib.epu.quadric_custom_op [-18.1085f, 57.4384f] 25
/model/layers.2/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-82.5232f, 81.3135f] 24
CustomOp/nn::gateUpProjection<27,27,24,24,35,35>6 contrib.epu.quadric_custom_op [-26.306f, 24.5606f] 26
CustomOp/linalg::channelwiseQuantMatMul<37>7 contrib.epu.quadric_custom_op [-10.8904f, 26.5761f] 26
CustomOp/add<MultiCoreMode::All>7 contrib.epu.quadric_custom_op [-23.2706f, 81.9635f] 24
/model/layers.3/input_layernorm/Mul_1 contrib.epu.rms_norm [-1.24747f, 3.00854f] 28
CustomOp/linalg::channelwiseQuantMatMul<42>11 contrib.epu.quadric_custom_op [-1.84038f, 1.97427f] 29
CustomOp/add<MultiCoreMode::All>11 contrib.epu.quadric_custom_op [-23.1594f, 81.7177f] 24
/model/layers.3/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-53.3995f, 62.6125f] 25
CustomOp/nn::gateUpProjection<28,28,26,26,37,37>9 contrib.epu.quadric_custom_op [-23.0629f, 23.7062f] 26
CustomOp/linalg::channelwiseQuantMatMul<39>10 contrib.epu.quadric_custom_op [-7.28584f, 7.4731f] 27
CustomOp/add<MultiCoreMode::All>10 contrib.epu.quadric_custom_op [-23.3718f, 85.3067f] 24
/model/layers.4/input_layernorm/Mul_1 contrib.epu.rms_norm [-1.80589f, 2.00268f] 29
CustomOp/linalg::channelwiseQuantMatMul<42>14 contrib.epu.quadric_custom_op [-3.53964f, 2.62418f] 29
CustomOp/add<MultiCoreMode::All>14 contrib.epu.quadric_custom_op [-23.2038f, 85.5677f] 24
/model/layers.4/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-32.953f, 41.0647f] 25
CustomOp/nn::gateUpProjection<28,28,27,27,37,37>12 contrib.epu.quadric_custom_op [-20.2387f, 20.039f] 26
CustomOp/linalg::channelwiseQuantMatMul<39>13 contrib.epu.quadric_custom_op [-6.69227f, 12.1684f] 27
CustomOp/add<MultiCoreMode::All>13 contrib.epu.quadric_custom_op [-22.4392f, 87.0669f] 24
/model/layers.5/input_layernorm/Mul_1 contrib.epu.rms_norm [-2.14681f, 3.19879f] 28
CustomOp/linalg::channelwiseQuantMatMul<42>17 contrib.epu.quadric_custom_op [-6.97809f, 4.76181f] 28
CustomOp/add<MultiCoreMode::All>17 contrib.epu.quadric_custom_op [-21.9713f, 84.54f] 24
/model/layers.5/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-24.6416f, 19.9332f] 26
CustomOp/nn::gateUpProjection<28,29,27,27,37,38>15 contrib.epu.quadric_custom_op [-18.2442f, 18.1234f] 26
CustomOp/linalg::channelwiseQuantMatMul<38>16 contrib.epu.quadric_custom_op [-37.2974f, 18.5171f] 25
CustomOp/add<MultiCoreMode::All>16 contrib.epu.quadric_custom_op [-18.6574f, 63.1892f] 24
/model/layers.6/input_layernorm/Mul_1 contrib.epu.rms_norm [-1.99346f, 7.23117f] 27
CustomOp/linalg::channelwiseQuantMatMul<40>20 contrib.epu.quadric_custom_op [-31.7815f, 8.99221f] 26
CustomOp/add<MultiCoreMode::All>20 contrib.epu.quadric_custom_op [-12.6824f, 38.3116f] 25
/model/layers.6/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-9.84406f, 199.177f] 23
CustomOp/nn::gateUpProjection<27,27,24,24,34,34>18 contrib.epu.quadric_custom_op [-1282.21f, 4841.01f] 18
CustomOp/linalg::channelwiseQuantMatMul<27>19 contrib.epu.quadric_custom_op [-2118.69f, 9638.07f] 17
CustomOp/add<MultiCoreMode::All>19 contrib.epu.quadric_custom_op [-2121.44f, 9654.1f] 17
/model/layers.7/input_layernorm/Mul_1 contrib.epu.rms_norm [-5.1855f, 10.441f] 27
CustomOp/linalg::channelwiseQuantMatMul<41>23 contrib.epu.quadric_custom_op [-4.27446f, 3.6406f] 28
CustomOp/add<MultiCoreMode::All>23 contrib.epu.quadric_custom_op [-2121.23f, 9654.21f] 17
/model/layers.7/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-6.89828f, 54.4274f] 25
CustomOp/nn::gateUpProjection<28,29,27,27,38,40>21 contrib.epu.quadric_custom_op [-16.9021f, 17.2346f] 26
CustomOp/linalg::channelwiseQuantMatMul<37>22 contrib.epu.quadric_custom_op [-6.76926f, 12.1386f] 27
CustomOp/add<MultiCoreMode::All>22 contrib.epu.quadric_custom_op [-2121.18f, 9654.34f] 17
/model/layers.8/input_layernorm/Mul_1 contrib.epu.rms_norm [-4.59068f, 9.9222f] 27
CustomOp/linalg::channelwiseQuantMatMul<41>26 contrib.epu.quadric_custom_op [-3.77026f, 7.3769f] 27
CustomOp/add<MultiCoreMode::All>26 contrib.epu.quadric_custom_op [-2120.87f, 9653.89f] 17
/model/layers.8/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-7.8127f, 6.98587f] 28
CustomOp/nn::gateUpProjection<29,30,27,28,39,40>24 contrib.epu.quadric_custom_op [-15.8356f, 18.0968f] 26
CustomOp/linalg::channelwiseQuantMatMul<38>25 contrib.epu.quadric_custom_op [-10.3458f, 12.5126f] 27
CustomOp/add<MultiCoreMode::All>25 contrib.epu.quadric_custom_op [-2120.62f, 9654.73f] 17
/model/layers.9/input_layernorm/Mul_1 contrib.epu.rms_norm [-4.52299f, 10.3721f] 27
CustomOp/linalg::channelwiseQuantMatMul<39>29 contrib.epu.quadric_custom_op [-7.31632f, 7.46158f] 27
CustomOp/add<MultiCoreMode::All>29 contrib.epu.quadric_custom_op [-2120.34f, 9654.1f] 17
/model/layers.9/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-8.45564f, 8.75462f] 27
CustomOp/nn::gateUpProjection<29,30,27,28,39,39>27 contrib.epu.quadric_custom_op [-15.1261f, 14.7165f] 27
CustomOp/linalg::channelwiseQuantMatMul<38>28 contrib.epu.quadric_custom_op [-9.86182f, 8.2895f] 27
CustomOp/add<MultiCoreMode::All>28 contrib.epu.quadric_custom_op [-2120.34f, 9656.53f] 17
/model/layers.10/input_layernorm/Mul_1 contrib.epu.rms_norm [-7.60531f, 17.5566f] 26
CustomOp/linalg::channelwiseQuantMatMul<40>32 contrib.epu.quadric_custom_op [-7.399f, 5.84284f] 28
CustomOp/add<MultiCoreMode::All>32 contrib.epu.quadric_custom_op [-2119.36f, 9655.58f] 17
/model/layers.10/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-8.33987f, 9.41487f] 27
CustomOp/nn::gateUpProjection<29,30,27,27,39,40>30 contrib.epu.quadric_custom_op [-15.7846f, 20.8058f] 26
CustomOp/linalg::channelwiseQuantMatMul<37>31 contrib.epu.quadric_custom_op [-5.03255f, 18.6078f] 26
CustomOp/add<MultiCoreMode::All>31 contrib.epu.quadric_custom_op [-2119.79f, 9656.97f] 17
/model/layers.11/input_layernorm/Mul_1 contrib.epu.rms_norm [-5.77417f, 10.4802f] 27
CustomOp/linalg::channelwiseQuantMatMul<40>35 contrib.epu.quadric_custom_op [-7.13255f, 7.60828f] 27
CustomOp/add<MultiCoreMode::All>35 contrib.epu.quadric_custom_op [-2118.77f, 9655.72f] 17
/model/layers.11/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-9.30534f, 7.62255f] 27
CustomOp/nn::gateUpProjection<29,30,27,27,39,40>33 contrib.epu.quadric_custom_op [-12.4377f, 12.5137f] 27
CustomOp/linalg::channelwiseQuantMatMul<38>34 contrib.epu.quadric_custom_op [-7.59268f, 11.5153f] 27
CustomOp/add<MultiCoreMode::All>34 contrib.epu.quadric_custom_op [-2118.11f, 9657.75f] 17
/model/layers.12/input_layernorm/Mul_1 contrib.epu.rms_norm [-6.76928f, 11.1257f] 27
CustomOp/linalg::channelwiseQuantMatMul<40>38 contrib.epu.quadric_custom_op [-7.8656f, 13.7375f] 27
CustomOp/add<MultiCoreMode::All>38 contrib.epu.quadric_custom_op [-2117.66f, 9657.01f] 17
/model/layers.12/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-10.1594f, 7.15519f] 27
CustomOp/nn::gateUpProjection<29,30,27,27,40,39>36 contrib.epu.quadric_custom_op [-60.1034f, 12.2375f] 25
CustomOp/linalg::channelwiseQuantMatMul<37>37 contrib.epu.quadric_custom_op [-12.1023f, 17.851f] 26
CustomOp/add<MultiCoreMode::All>37 contrib.epu.quadric_custom_op [-2118.72f, 9659.33f] 17
/model/layers.13/input_layernorm/Mul_1 contrib.epu.rms_norm [-5.04587f, 9.31552f] 27
CustomOp/linalg::channelwiseQuantMatMul<40>41 contrib.epu.quadric_custom_op [-5.54142f, 8.70166f] 27
CustomOp/add<MultiCoreMode::All>41 contrib.epu.quadric_custom_op [-2117.86f, 9659.14f] 17
/model/layers.13/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-11.9317f, 6.23108f] 27
CustomOp/nn::gateUpProjection<30,29,26,25,40,39>39 contrib.epu.quadric_custom_op [-16.9783f, 33.7621f] 25
CustomOp/linalg::channelwiseQuantMatMul<38>40 contrib.epu.quadric_custom_op [-11.2142f, 11.9456f] 27
CustomOp/add<MultiCoreMode::All>40 contrib.epu.quadric_custom_op [-2120.76f, 9664.39f] 17
/model/layers.14/input_layernorm/Mul_1 contrib.epu.rms_norm [-6.68111f, 13.0119f] 27
CustomOp/linalg::channelwiseQuantMatMul<40>44 contrib.epu.quadric_custom_op [-8.29877f, 19.2297f] 26
CustomOp/add<MultiCoreMode::All>44 contrib.epu.quadric_custom_op [-2120.07f, 9663.34f] 17
/model/layers.14/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-14.412f, 6.25216f] 27
CustomOp/nn::gateUpProjection<30,29,26,25,39,39>42 contrib.epu.quadric_custom_op [-84.1848f, 13.7615f] 24
CustomOp/linalg::channelwiseQuantMatMul<36>43 contrib.epu.quadric_custom_op [-18.0127f, 15.0041f] 26
CustomOp/add<MultiCoreMode::All>43 contrib.epu.quadric_custom_op [-2123.29f, 9666.16f] 17
/model/layers.15/input_layernorm/Mul_1 contrib.epu.rms_norm [-7.40155f, 13.516f] 27
CustomOp/linalg::channelwiseQuantMatMul<39>47 contrib.epu.quadric_custom_op [-6.83366f, 13.2874f] 27
CustomOp/add<MultiCoreMode::All>47 contrib.epu.quadric_custom_op [-2122.7f, 9664.69f] 17
/model/layers.15/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-16.0488f, 6.0668f] 26
CustomOp/nn::gateUpProjection<29,29,26,25,39,38>45 contrib.epu.quadric_custom_op [-27.8356f, 48.4799f] 25
CustomOp/linalg::channelwiseQuantMatMul<36>46 contrib.epu.quadric_custom_op [-16.8693f, 20.4889f] 26
CustomOp/add<MultiCoreMode::All>46 contrib.epu.quadric_custom_op [-2123.21f, 9668.25f] 17
/model/layers.16/input_layernorm/Mul_1 contrib.epu.rms_norm [-8.15026f, 15.6546f] 26
CustomOp/linalg::channelwiseQuantMatMul<38>50 contrib.epu.quadric_custom_op [-39.5976f, 22.6377f] 25
CustomOp/add<MultiCoreMode::All>50 contrib.epu.quadric_custom_op [-2122.92f, 9666.15f] 17
/model/layers.16/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-36.8359f, 6.53895f] 25
CustomOp/nn::gateUpProjection<28,28,23,24,37,37>48 contrib.epu.quadric_custom_op [-191.33f, 4661.68f] 18
CustomOp/linalg::channelwiseQuantMatMul<27>49 contrib.epu.quadric_custom_op [-1202.39f, 13581.2f] 17
CustomOp/add<MultiCoreMode::All>49 contrib.epu.quadric_custom_op [-2130f, 13602.6f] 17
/model/layers.17/input_layernorm/Mul_1 contrib.epu.rms_norm [-7.88683f, 13.0965f] 27
CustomOp/linalg::channelwiseQuantMatMul<37>53 contrib.epu.quadric_custom_op [-14.5229f, 61.5807f] 25
CustomOp/add<MultiCoreMode::All>53 contrib.epu.quadric_custom_op [-2129.24f, 13637f] 17
/model/layers.17/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-12.2827f, 6.15562f] 27
CustomOp/nn::gateUpProjection<30,30,27,25,39,40>51 contrib.epu.quadric_custom_op [-13.8208f, 9.58101f] 27
CustomOp/linalg::channelwiseQuantMatMul<38>52 contrib.epu.quadric_custom_op [-21.4484f, 22.413f] 26
CustomOp/add<MultiCoreMode::All>52 contrib.epu.quadric_custom_op [-2128.21f, 13636.9f] 17
/model/layers.18/input_layernorm/Mul_1 contrib.epu.rms_norm [-10.6242f, 15.4063f] 26
CustomOp/linalg::channelwiseQuantMatMul<40>56 contrib.epu.quadric_custom_op [-11.2483f, 15.3732f] 26
CustomOp/add<MultiCoreMode::All>56 contrib.epu.quadric_custom_op [-2127.25f, 13638.1f] 17
/model/layers.18/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-11.7031f, 6.12549f] 27
CustomOp/nn::gateUpProjection<29,30,27,26,39,39>54 contrib.epu.quadric_custom_op [-9.16472f, 64.1296f] 24
CustomOp/linalg::channelwiseQuantMatMul<35>55 contrib.epu.quadric_custom_op [-27.1561f, 62.6349f] 25
CustomOp/add<MultiCoreMode::All>55 contrib.epu.quadric_custom_op [-2127.74f, 13638.1f] 17
/model/layers.19/input_layernorm/Mul_1 contrib.epu.rms_norm [-16.3269f, 23.5058f] 26
CustomOp/linalg::channelwiseQuantMatMul<39>59 contrib.epu.quadric_custom_op [-9.69087f, 19.3656f] 26
CustomOp/add<MultiCoreMode::All>59 contrib.epu.quadric_custom_op [-2127.19f, 13637.7f] 17
/model/layers.19/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-11.4451f, 6.57414f] 27
CustomOp/nn::gateUpProjection<30,30,27,28,40,39>57 contrib.epu.quadric_custom_op [-10.5208f, 11.7491f] 27
CustomOp/linalg::channelwiseQuantMatMul<36>58 contrib.epu.quadric_custom_op [-28.7991f, 32.1844f] 25
CustomOp/add<MultiCoreMode::All>58 contrib.epu.quadric_custom_op [-2127.2f, 13637.6f] 17
/model/layers.20/input_layernorm/Mul_1 contrib.epu.rms_norm [-17.2382f, 20.6114f] 26
CustomOp/linalg::channelwiseQuantMatMul<40>62 contrib.epu.quadric_custom_op [-11.7166f, 31.5278f] 25
CustomOp/add<MultiCoreMode::All>62 contrib.epu.quadric_custom_op [-2126.62f, 13634.6f] 17
/model/layers.20/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-12.2707f, 7.12023f] 27
CustomOp/nn::gateUpProjection<29,29,27,27,40,40>60 contrib.epu.quadric_custom_op [-18.5987f, 19.8616f] 26
CustomOp/linalg::channelwiseQuantMatMul<38>61 contrib.epu.quadric_custom_op [-16.8789f, 16.8205f] 26
CustomOp/add<MultiCoreMode::All>61 contrib.epu.quadric_custom_op [-2125.87f, 13634.7f] 17
/model/layers.21/input_layernorm/Mul_1 contrib.epu.rms_norm [-23.1988f, 24.3195f] 26
CustomOp/linalg::channelwiseQuantMatMul<39>65 contrib.epu.quadric_custom_op [-8.88642f, 21.276f] 26
CustomOp/add<MultiCoreMode::All>65 contrib.epu.quadric_custom_op [-2124.82f, 13636.5f] 17
/model/layers.21/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-10.9728f, 7.28116f] 27
CustomOp/nn::gateUpProjection<30,29,27,27,39,39>63 contrib.epu.quadric_custom_op [-41.594f, 53.735f] 25
CustomOp/linalg::channelwiseQuantMatMul<37>64 contrib.epu.quadric_custom_op [-32.024f, 16.017f] 25
CustomOp/add<MultiCoreMode::All>64 contrib.epu.quadric_custom_op [-2124.91f, 13636.6f] 17
/model/layers.22/input_layernorm/Mul_1 contrib.epu.rms_norm [-30.8369f, 32.7385f] 25
CustomOp/linalg::channelwiseQuantMatMul<39>68 contrib.epu.quadric_custom_op [-12.5842f, 31.0796f] 25
CustomOp/add<MultiCoreMode::All>68 contrib.epu.quadric_custom_op [-2124.1f, 13635.5f] 17
/model/layers.22/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-10.0459f, 8.97793f] 27
CustomOp/nn::gateUpProjection<30,30,26,27,39,39>66 contrib.epu.quadric_custom_op [-42.2895f, 37.618f] 25
CustomOp/linalg::channelwiseQuantMatMul<37>67 contrib.epu.quadric_custom_op [-24.8563f, 17.5565f] 26
CustomOp/add<MultiCoreMode::All>67 contrib.epu.quadric_custom_op [-2123.61f, 13635.4f] 17
/model/layers.23/input_layernorm/Mul_1 contrib.epu.rms_norm [-32.5652f, 31.0209f] 25
CustomOp/linalg::channelwiseQuantMatMul<39>71 contrib.epu.quadric_custom_op [-8.13639f, 35.0445f] 25
CustomOp/add<MultiCoreMode::All>71 contrib.epu.quadric_custom_op [-2123.06f, 13632.6f] 17
/model/layers.23/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-10.1758f, 10.4433f] 27
CustomOp/nn::gateUpProjection<30,29,26,27,39,39>69 contrib.epu.quadric_custom_op [-64.6526f, 60.3269f] 24
CustomOp/linalg::channelwiseQuantMatMul<37>70 contrib.epu.quadric_custom_op [-24.6908f, 31.6436f] 25
CustomOp/add<MultiCoreMode::All>70 contrib.epu.quadric_custom_op [-2122.79f, 13632.6f] 17
/model/layers.24/input_layernorm/Mul_1 contrib.epu.rms_norm [-50.8646f, 39.9522f] 25
CustomOp/linalg::channelwiseQuantMatMul<38>74 contrib.epu.quadric_custom_op [-16.7965f, 31.3655f] 25
CustomOp/add<MultiCoreMode::All>74 contrib.epu.quadric_custom_op [-2123.99f, 13635.1f] 17
/model/layers.24/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-10.2601f, 12.5924f] 27
CustomOp/nn::gateUpProjection<29,29,26,27,39,39>72 contrib.epu.quadric_custom_op [-107.554f, 90.9104f] 24
CustomOp/linalg::channelwiseQuantMatMul<36>73 contrib.epu.quadric_custom_op [-22.1531f, 38.053f] 25
CustomOp/add<MultiCoreMode::All>73 contrib.epu.quadric_custom_op [-2123.99f, 13635.3f] 17
/model/layers.25/input_layernorm/Mul_1 contrib.epu.rms_norm [-40.2525f, 33.3074f] 25
CustomOp/linalg::channelwiseQuantMatMul<39>77 contrib.epu.quadric_custom_op [-10.4602f, 17.5029f] 26
CustomOp/add<MultiCoreMode::All>77 contrib.epu.quadric_custom_op [-2123.75f, 13635.5f] 17
/model/layers.25/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-11.309f, 12.7584f] 27
CustomOp/nn::gateUpProjection<29,29,26,26,39,39>75 contrib.epu.quadric_custom_op [-117.555f, 109.813f] 24
CustomOp/linalg::channelwiseQuantMatMul<36>76 contrib.epu.quadric_custom_op [-19.0471f, 36.3233f] 25
CustomOp/add<MultiCoreMode::All>76 contrib.epu.quadric_custom_op [-2123.73f, 13635.5f] 17
/model/layers.26/input_layernorm/Mul_1 contrib.epu.rms_norm [-50.6021f, 38.8768f] 25
CustomOp/linalg::channelwiseQuantMatMul<40>80 contrib.epu.quadric_custom_op [-8.51294f, 19.8805f] 26
CustomOp/add<MultiCoreMode::All>80 contrib.epu.quadric_custom_op [-2124.04f, 13636.4f] 17
/model/layers.26/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-12.5561f, 15.4058f] 26
CustomOp/nn::gateUpProjection<29,29,26,27,39,38>78 contrib.epu.quadric_custom_op [-105.01f, 132.743f] 23
CustomOp/linalg::channelwiseQuantMatMul<36>79 contrib.epu.quadric_custom_op [-21.3917f, 49.6387f] 25
CustomOp/add<MultiCoreMode::All>79 contrib.epu.quadric_custom_op [-2124.03f, 13636.6f] 17
/model/layers.27/input_layernorm/Mul_1 contrib.epu.rms_norm [-61.2838f, 48.1027f] 25
CustomOp/linalg::channelwiseQuantMatMul<39>83 contrib.epu.quadric_custom_op [-8.8824f, 19.0919f] 26
CustomOp/add<MultiCoreMode::All>83 contrib.epu.quadric_custom_op [-2124.55f, 13638.8f] 17
/model/layers.27/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-14.5542f, 17.2044f] 26
CustomOp/nn::gateUpProjection<29,29,26,27,39,38>81 contrib.epu.quadric_custom_op [-137.571f, 122.284f] 23
CustomOp/linalg::channelwiseQuantMatMul<36>82 contrib.epu.quadric_custom_op [-29.7459f, 42.6272f] 25
CustomOp/add<MultiCoreMode::All>82 contrib.epu.quadric_custom_op [-2124.49f, 13639.2f] 17
/model/layers.28/input_layernorm/Mul_1 contrib.epu.rms_norm [-65.7556f, 56.4749f] 24
CustomOp/linalg::channelwiseQuantMatMul<39>86 contrib.epu.quadric_custom_op [-16.1447f, 22.5356f] 26
CustomOp/add<MultiCoreMode::All>86 contrib.epu.quadric_custom_op [-2125.41f, 13642.1f] 17
/model/layers.28/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-13.3559f, 17.9234f] 26
CustomOp/nn::gateUpProjection<29,29,26,26,38,38>84 contrib.epu.quadric_custom_op [-132.766f, 140.229f] 23
CustomOp/linalg::channelwiseQuantMatMul<35>85 contrib.epu.quadric_custom_op [-38.7672f, 58.2323f] 25
CustomOp/add<MultiCoreMode::All>85 contrib.epu.quadric_custom_op [-2125.36f, 13642.5f] 17
/model/layers.29/input_layernorm/Mul_1 contrib.epu.rms_norm [-87.7913f, 63.9043f] 24
CustomOp/linalg::channelwiseQuantMatMul<39>89 contrib.epu.quadric_custom_op [-15.1006f, 23.0286f] 26
CustomOp/add<MultiCoreMode::All>89 contrib.epu.quadric_custom_op [-2126.23f, 13643.9f] 17
/model/layers.29/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-14.5663f, 19.9309f] 26
CustomOp/nn::gateUpProjection<30,29,26,26,39,38>87 contrib.epu.quadric_custom_op [-150.54f, 123.012f] 23
CustomOp/linalg::channelwiseQuantMatMul<35>88 contrib.epu.quadric_custom_op [-49.9533f, 76.5029f] 24
CustomOp/add<MultiCoreMode::All>88 contrib.epu.quadric_custom_op [-2126.19f, 13644f] 17
/model/layers.30/input_layernorm/Mul_1 contrib.epu.rms_norm [-97.8058f, 70.9147f] 24
CustomOp/linalg::channelwiseQuantMatMul<38>92 contrib.epu.quadric_custom_op [-26.3882f, 42.82f] 25
CustomOp/add<MultiCoreMode::All>92 contrib.epu.quadric_custom_op [-2126.46f, 13648.6f] 17
/model/layers.30/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-16.1268f, 19.8625f] 26
CustomOp/nn::gateUpProjection<29,28,26,26,38,38>90 contrib.epu.quadric_custom_op [-147.928f, 141.815f] 23
CustomOp/linalg::channelwiseQuantMatMul<35>91 contrib.epu.quadric_custom_op [-53.484f, 106.991f] 24
CustomOp/add<MultiCoreMode::All>91 contrib.epu.quadric_custom_op [-2126.44f, 13648.7f] 17
/model/layers.31/input_layernorm/Mul_1 contrib.epu.rms_norm [-119.533f, 93.6401f] 24
CustomOp/linalg::channelwiseQuantMatMul<38>95 contrib.epu.quadric_custom_op [-17.2974f, 50.1237f] 25
CustomOp/add<MultiCoreMode::All>95 contrib.epu.quadric_custom_op [-2126.83f, 13653.5f] 17
/model/layers.31/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-17.2516f, 18.0006f] 26
CustomOp/nn::gateUpProjection<29,28,26,26,38,37>93 contrib.epu.quadric_custom_op [-147.492f, 171.108f] 23
CustomOp/linalg::channelwiseQuantMatMul<34>94 contrib.epu.quadric_custom_op [-42.7217f, 101.298f] 24
CustomOp/add<MultiCoreMode::All>94 contrib.epu.quadric_custom_op [-2126.65f, 13653.4f] 17
/model/layers.32/input_layernorm/Mul_1 contrib.epu.rms_norm [-129.345f, 108.642f] 23
CustomOp/linalg::channelwiseQuantMatMul<37>98 contrib.epu.quadric_custom_op [-26.9792f, 52.8894f] 25
CustomOp/add<MultiCoreMode::All>98 contrib.epu.quadric_custom_op [-2125.55f, 13676.5f] 17
/model/layers.32/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-17.899f, 23.7326f] 26
CustomOp/nn::gateUpProjection<28,28,26,26,38,37>96 contrib.epu.quadric_custom_op [-190.096f, 285.781f] 22
CustomOp/linalg::channelwiseQuantMatMul<34>97 contrib.epu.quadric_custom_op [-67.0631f, 111.421f] 24
CustomOp/add<MultiCoreMode::All>97 contrib.epu.quadric_custom_op [-2125.27f, 13672.9f] 17
/model/layers.33/input_layernorm/Mul_1 contrib.epu.rms_norm [-187.593f, 150.813f] 23
CustomOp/linalg::channelwiseQuantMatMul<37>101 contrib.epu.quadric_custom_op [-22.3657f, 115.159f] 24
CustomOp/add<MultiCoreMode::All>101 contrib.epu.quadric_custom_op [-2125.11f, 13717f] 17
/model/layers.33/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-29.27f, 27.8766f] 26
CustomOp/nn::gateUpProjection<28,28,26,26,37,36>99 contrib.epu.quadric_custom_op [-191.84f, 180.474f] 23
CustomOp/linalg::channelwiseQuantMatMul<34>100 contrib.epu.quadric_custom_op [-224.43f, 154.382f] 23
CustomOp/add<MultiCoreMode::All>100 contrib.epu.quadric_custom_op [-2126.49f, 13736f] 17
/model/layers.34/input_layernorm/Mul_1 contrib.epu.rms_norm [-190.924f, 150.511f] 23
CustomOp/linalg::channelwiseQuantMatMul<36>104 contrib.epu.quadric_custom_op [-134.891f, 252.592f] 23
CustomOp/add<MultiCoreMode::All>104 contrib.epu.quadric_custom_op [-2127.57f, 13665.9f] 17
/model/layers.34/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-102.457f, 31.8715f] 24
CustomOp/nn::gateUpProjection<27,27,25,24,35,34>102 contrib.epu.quadric_custom_op [-1174.96f, 792.102f] 20
CustomOp/linalg::channelwiseQuantMatMul<31>103 contrib.epu.quadric_custom_op [-18980.2f, 553.5f] 16
CustomOp/add<MultiCoreMode::All>103 contrib.epu.quadric_custom_op [-7343.94f, 5351.69f] 18
/model/layers.35/input_layernorm/Mul_1 contrib.epu.rms_norm [-298.728f, 121.485f] 22
CustomOp/linalg::channelwiseQuantMatMul<36>107 contrib.epu.quadric_custom_op [-282.141f, 824.661f] 21
CustomOp/add<MultiCoreMode::All>107 contrib.epu.quadric_custom_op [-7101.39f, 5072.41f] 18
/model/layers.35/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-601.076f, 47.2325f] 21
CustomOp/nn::gateUpProjection<25,25,24,23,34,33>105 contrib.epu.quadric_custom_op [-1002.33f, 1755.35f] 20
CustomOp/linalg::channelwiseQuantMatMul<29>106 contrib.epu.quadric_custom_op [-4812.36f, 1990.75f] 18
CustomOp/add<MultiCoreMode::All>106 contrib.epu.quadric_custom_op [-6990.34f, 2581.91f] 18
/model/norm/Mul_1 contrib.epu.rms_norm [-139.062f, 130.451f] 23
CustomOp/nn::gather<Direction::Height>108 contrib.epu.quadric_custom_op [-93.9597f, 66.4554f] 24
CustomOp/linalg::channelwiseQuantMatMul<36>108 contrib.epu.quadric_custom_op [-15.0257f, 24.2741f] 26
2026-09-22 04:21 - INFO - epu - codegen - START====================build_cpu_runnable_fx_relay
2026-09-22 04:21 - INFO - epu - codegen - START=======================quantize_to_chimera_fx
2026-09-22 04:21 - INFO - epu - codegen - START=================================relay_to_tir
2026-09-22 04:21 - INFO - epu - codegen - START===========================relay_to_epu_relay
2026-09-22 04:21 - INFO - epu - codegen - START==============================adapt_and_order
2026-09-22 04:22 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-09-22 04:22 - INFO - epu - codegen - START=============================plan_lrm_virtual
2026-09-22 04:24 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-09-22 04:24 - INFO - epu - codegen - START===============================lrm_alloc_loop
2026-09-22 04:25 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-09-22 04:26 - INFO - epu - codegen - START================================lrm_splitting
2026-09-22 04:29 - INFO - epu - codegen - START==============================ext_split_relay
2026-09-22 04:31 - INFO - epu - codegen - START====================================build_tir
2026-09-22 04:31 - INFO - epu - chimera_job - Compilation of qwen3_custom_op_all_dec_QC_P_1d7_2MB_4kB_128GBps_128GBps_16_OFF_x1_x1 successful
╒═════════════════════╤═══════════════════════════════════════════════════════════════════════╕
│ Module Name │ qwen3_custom_op_all_dec_QC_P_1d7_2MB_4kB_128GBps_128GBps_16_OFF_x1_x1 │
├─────────────────────┼───────────────────────────────────────────────────────────────────────┤
│ ONNX File │ prefill/qwen3_custom_op_all_dec.onnx │
├─────────────────────┼───────────────────────────────────────────────────────────────────────┤
│ Product Target │ QC-P │
├─────────────────────┼───────────────────────────────────────────────────────────────────────┤
│ Number of Cores │ 1 │
├─────────────────────┼───────────────────────────────────────────────────────────────────────┤
│ ISS Clock Frequency │ 1.700 │
├─────────────────────┼───────────────────────────────────────────────────────────────────────┤
│ L2M Size │ 2MB │
├─────────────────────┼───────────────────────────────────────────────────────────────────────┤
│ LRM Size │ 4kB │
├─────────────────────┼───────────────────────────────────────────────────────────────────────┤
│ External Read BW │ 128GBps │
├─────────────────────┼───────────────────────────────────────────────────────────────────────┤
│ External Write BW │ 128GBps │
├─────────────────────┼───────────────────────────────────────────────────────────────────────┤
│ MACS per PE │ 16 │
├─────────────────────┼───────────────────────────────────────────────────────────────────────┤
│ Max L2M │ 0.000MB │
├─────────────────────┼───────────────────────────────────────────────────────────────────────┤
│ Max LRM │ 0.000kB │
├─────────────────────┼───────────────────────────────────────────────────────────────────────┤
│ Max Temp Ext Bytes │ 128.000MB │
├─────────────────────┼───────────────────────────────────────────────────────────────────────┤
│ Network GMACs │ │
╘═════════════════════╧═══════════════════════════════════════════════════════════════════════╛
╒═════╤════════╤══════════════════════════╤═══════════════════╤══════════════════════════╤═══════╕
│ │ Type │ Name │ shape │ type │ mse │
╞═════╪════════╪══════════════════════════╪═══════════════════╪══════════════════════════╪═══════╡
│ 0 │ Input │ input_ids │ [1, 1024] │ tensor[int32] │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 1 │ Input │ attention_mask │ [1, 1024, 1024] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 2 │ Input │ sin │ [1, 1024, 128] │ tensor[FixedPoint32<30>] │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 3 │ Input │ cos │ [1, 1024, 128] │ tensor[FixedPoint32<30>] │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 4 │ Input │ past_key_values.0.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 5 │ Input │ past_key_values.0.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 6 │ Input │ past_key_values.1.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 7 │ Input │ past_key_values.1.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 8 │ Input │ past_key_values.2.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 9 │ Input │ past_key_values.2.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 10 │ Input │ past_key_values.3.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 11 │ Input │ past_key_values.3.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 12 │ Input │ past_key_values.4.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 13 │ Input │ past_key_values.4.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 14 │ Input │ past_key_values.5.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 15 │ Input │ past_key_values.5.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 16 │ Input │ past_key_values.6.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 17 │ Input │ past_key_values.6.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 18 │ Input │ past_key_values.7.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 19 │ Input │ past_key_values.7.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 20 │ Input │ past_key_values.8.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 21 │ Input │ past_key_values.8.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 22 │ Input │ past_key_values.9.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 23 │ Input │ past_key_values.9.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 24 │ Input │ past_key_values.10.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 25 │ Input │ past_key_values.10.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 26 │ Input │ past_key_values.11.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 27 │ Input │ past_key_values.11.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 28 │ Input │ past_key_values.12.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 29 │ Input │ past_key_values.12.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 30 │ Input │ past_key_values.13.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 31 │ Input │ past_key_values.13.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 32 │ Input │ past_key_values.14.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 33 │ Input │ past_key_values.14.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 34 │ Input │ past_key_values.15.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 35 │ Input │ past_key_values.15.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 36 │ Input │ past_key_values.16.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 37 │ Input │ past_key_values.16.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 38 │ Input │ past_key_values.17.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 39 │ Input │ past_key_values.17.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 40 │ Input │ past_key_values.18.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 41 │ Input │ past_key_values.18.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 42 │ Input │ past_key_values.19.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 43 │ Input │ past_key_values.19.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 44 │ Input │ past_key_values.20.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 45 │ Input │ past_key_values.20.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 46 │ Input │ past_key_values.21.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 47 │ Input │ past_key_values.21.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 48 │ Input │ past_key_values.22.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 49 │ Input │ past_key_values.22.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 50 │ Input │ past_key_values.23.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 51 │ Input │ past_key_values.23.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 52 │ Input │ past_key_values.24.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 53 │ Input │ past_key_values.24.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 54 │ Input │ past_key_values.25.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 55 │ Input │ past_key_values.25.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 56 │ Input │ past_key_values.26.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 57 │ Input │ past_key_values.26.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 58 │ Input │ past_key_values.27.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 59 │ Input │ past_key_values.27.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 60 │ Input │ past_key_values.28.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 61 │ Input │ past_key_values.28.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 62 │ Input │ past_key_values.29.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 63 │ Input │ past_key_values.29.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 64 │ Input │ past_key_values.30.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 65 │ Input │ past_key_values.30.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 66 │ Input │ past_key_values.31.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 67 │ Input │ past_key_values.31.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 68 │ Input │ past_key_values.32.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 69 │ Input │ past_key_values.32.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 70 │ Input │ past_key_values.33.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 71 │ Input │ past_key_values.33.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 72 │ Input │ past_key_values.34.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 73 │ Input │ past_key_values.34.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 74 │ Input │ past_key_values.35.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 75 │ Input │ past_key_values.35.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 76 │ Output │ logits │ [1, 151936] │ tensor[FixedPoint32<26>] │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 77 │ Output │ present.0.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 78 │ Output │ present.0.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 79 │ Output │ present.1.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 80 │ Output │ present.1.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 81 │ Output │ present.2.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 82 │ Output │ present.2.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 83 │ Output │ present.3.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 84 │ Output │ present.3.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 85 │ Output │ present.4.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 86 │ Output │ present.4.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 87 │ Output │ present.5.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 88 │ Output │ present.5.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 89 │ Output │ present.6.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 90 │ Output │ present.6.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 91 │ Output │ present.7.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 92 │ Output │ present.7.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 93 │ Output │ present.8.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 94 │ Output │ present.8.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 95 │ Output │ present.9.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 96 │ Output │ present.9.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 97 │ Output │ present.10.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 98 │ Output │ present.10.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 99 │ Output │ present.11.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 100 │ Output │ present.11.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 101 │ Output │ present.12.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 102 │ Output │ present.12.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 103 │ Output │ present.13.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 104 │ Output │ present.13.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 105 │ Output │ present.14.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 106 │ Output │ present.14.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 107 │ Output │ present.15.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 108 │ Output │ present.15.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 109 │ Output │ present.16.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 110 │ Output │ present.16.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 111 │ Output │ present.17.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 112 │ Output │ present.17.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 113 │ Output │ present.18.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 114 │ Output │ present.18.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 115 │ Output │ present.19.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 116 │ Output │ present.19.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 117 │ Output │ present.20.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 118 │ Output │ present.20.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 119 │ Output │ present.21.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 120 │ Output │ present.21.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 121 │ Output │ present.22.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 122 │ Output │ present.22.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 123 │ Output │ present.23.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 124 │ Output │ present.23.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 125 │ Output │ present.24.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 126 │ Output │ present.24.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 127 │ Output │ present.25.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 128 │ Output │ present.25.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 129 │ Output │ present.26.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 130 │ Output │ present.26.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 131 │ Output │ present.27.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 132 │ Output │ present.27.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 133 │ Output │ present.28.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 134 │ Output │ present.28.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 135 │ Output │ present.29.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 136 │ Output │ present.29.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 137 │ Output │ present.30.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 138 │ Output │ present.30.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 139 │ Output │ present.31.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 140 │ Output │ present.31.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 141 │ Output │ present.32.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 142 │ Output │ present.32.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 143 │ Output │ present.33.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 144 │ Output │ present.33.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 145 │ Output │ present.34.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 146 │ Output │ present.34.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 147 │ Output │ present.35.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 148 │ Output │ present.35.value │ [1, 8, 1024, 128] │ n/a │ n/a │
╘═════╧════════╧══════════════════════════╧═══════════════════╧══════════════════════════╧═══════╛
project_prefill_latency(
PREFILL_DIR / f"decoder_{VALIDATE_DECODER}",
num_decoders=NUM_DECODERS,
seq_len=SEQ_LEN,
clock_ghz=1.7,
)
Decoder cycles (measured) : 102,162,105 MAC util 2.7%
Full prefill (36 decoders) : 3,677,835,780 cycles
Time to first token : 2,163 ms @ 1.7 GHz
Prefill throughput : 473 tokens/s
{'total_cycles': 102162105,
'total_mac_cycles': 2765876,
'mac_util': 2.7073404566203876,
'n_regions': 8,
'iss_log_path': 'prefill/decoder_2/qwen3_prefill_runner_QC-P_1d0_2MB_4kB_24GBps_24GBps_16_OFF_x1_x4/output/iss_run_log.txt'}
Summary
| Model | Qwen3-8B — 36 layers, hidden size 4096, 32 query / 8 KV heads, 151,936-token vocabulary |
| Quantization | W4A8 SmoothQuant (α = 0.36): INT4 weights packed v8i4, INT8 activations |
| Target | QC-P × 4 cores, 2 MB OCM per core, 16 MACs/PE, 1 GHz, 24 GB/s DDR per core |
| Decode | Full autoregressive loop on the ISS — ~18–19 tokens/s average, 20–21 peak; logits match ONNX Runtime (top-1, top-5, Pearson r) |
| Prefill | Decoder 2 validated against ONNX Runtime on the ISS; all 36 decoders + LM head compiled through CGC |
| Custom Ops | nn::qwen3Attention, nn::gateProj, linalg::channelwiseQuantMatMul (decode); nn::qwen3PrefillAttention, nn::gateUpProjection (prefill) |
Key takeaways
- One export, two graphs. Decode and prefill differ only in their fixed shapes; weights, tensor ranges and custom ops are shared, so the prefill program is one shape-fix away from the decode kernel.
- Fused attention keeps the KV cache on-chip.
nn::qwen3Attentionappends to the cache and attends over it inside OCM, so the runner never round-trips cache state through the host between tokens. - INT4 weights are a bandwidth play. Packing eight weights per 32-bit word halves the DDR traffic of an INT8 model per token — the quantity that sets decode speed on a memory-bound 8B model.
- Every GPNPU result is checked against a host reference. Decode logits are compared with an ONNX Runtime replay of the same prompt; prefill compares one cut decoder against ONNX Runtime element by element. Validate a decoder, not the network: Cutting one layer, compiling it and comparing it against ONNX Runtime pins any numerical drift to a single layer, and its measured cycles project the full-prefill time to first token.
sdk sourcecloses the loop. One command compiles the CGC output and a hand-written host runner with Quadric LLVM and executes them on the cycle-accurate multicore ISS.
Citation
@article{qwen3technicalreport,
title = {Qwen3 Technical Report},
author = {Qwen Team},
journal = {arXiv preprint arXiv:2505.09388},
year = {2025}
}
