NOTE: The Jupyter Notebook below is included in the Chimera SDK and can be run interactively by running the following CLI command:
$ quadric sdk notebook
From the Jupyter Notebook window in your browser, select the notebook named /quadric/sdk-cli/examples/models/qwen/qwen3_8b/qwen3_8b.ipynb.
Qwen3-8B Decode on Chimera GPNPU
This notebook compiles Qwen3-8B with CGC (Chimera Graph Compiler) and runs the autoregressive decode loop on the ISS (Instruction Set Simulator) in a 4-core QC-P configuration: one token in, one token out, the KV cache carried on-chip between steps. Roughly 18–19 tokens/s average (20–21 peak) at 1 GHz.
Its companion, qwen3_8b_prefill.ipynb, covers the other half — prefill, the whole 1024-token prompt through all 36 layers in one pass, which fills the KV cache this loop then consumes. Both are cut from the same W4A8 SmoothQuant export and share its tensor ranges and custom operators; only the shape-fixing step and the host runner differ. They are kept as two notebooks so that each phase runs in its own kernel.
Pipeline
Model: Qwen3-8B (8.2B parameters, 36 decoder layers, hidden size 4096, 32 query / 8 KV heads, 151,936-token vocabulary), W4A8 SmoothQuant (α = 0.36), 1024-token context
| Phase | Graph shape | On the ISS | Where |
|---|---|---|---|
| Decode | 1 token in, 1023 cached | The full autoregressive loop on 4× QC-P, checked against ONNX Runtime | sections 3–9 below |
| Prefill | 1024 tokens in, empty cache | One decoder layer validated against ONNX Runtime; all 36 compiled | qwen3_8b_prefill.ipynb |
1. Setup
Imports, the model's architecture constants and the hardware target. The QC-P configuration below is shared by every compile and ISS run in this notebook: 4 cores, 2 MB OCM (On-Chip Memory) per core, 16 MACs per PE, 1 GHz, 24 GB/s of DDR bandwidth per core over a 256-bit AXI interface. CGC needs a deep stack to schedule an 8-billion-parameter graph, so the limit is raised once here.
import re
from pathlib import Path
import numpy as np
from tvm.contrib.epu.chimera_job.chimera_job import ChimeraJob
from tvm.contrib.epu.chimera_job.hw_config import HWConfig
from examples.models.qwen.qwen_helpers import (
compare_decode_with_ort,
kv_cache_io_names,
plot_decode_profile,
raise_stack_limit,
report_decode_throughput,
stage_const_tensor,
)
from custom_op_match import qwen3_custom_op_replacer
from fix_shapes import fix_shapes
from qwen3_8b_helpers import download_decode_model
%matplotlib inline
NUM_DECODERS = 36
EMBED_DIM = 4096
NUM_KV_HEADS = 8 # 32 query heads share 8 key/value heads (grouped-query attention)
SEQ_LEN = 1024 # context window: prompt + generated tokens
NEW_TOKENS = 1
HW_CONFIG = HWConfig(product="QC-P", ocm_size="2MB", macs_per_pe=16)
raise_stack_limit()
Stack limit raised to 32 MB for CGC compilation
2. Download the W4A8 Checkpoint
Qwen3-8B was quantized offline to W4A8 — 4-bit per-channel weights, 8-bit activations — with SmoothQuant (α = 0.36) migrating activation outliers into the weights before quantization. The export also lifts the RoPE sin/cos tables out as graph inputs, so the runner supplies the rotary angle for each position, and carries an attention mask shaped for both the prefill and the decode stage. Three files make up the decode graph:
| File | Contents |
|---|---|
qwen_0_36.onnx | The quantized graph with dynamic batch and sequence dimensions |
9b9822ba-….data | External weight data (~6 GB) |
qwen_0_36.onnx.tranges | Per-tensor value ranges that parameterize the fixed-point custom ops |
decode_onnx, _decode_weights, tranges_path = map(str, download_decode_model())
Downloading qwen_0_36.onnx...
Downloaded qwen_0_36.onnx (0.02 GB)
Downloading 9b9822ba-a73f-11f0-9995-10ffe060dbe7.data...
Downloaded 9b9822ba-a73f-11f0-9995-10ffe060dbe7.data (6.27 GB)
Downloading qwen_0_36.onnx.tranges...
Downloaded qwen_0_36.onnx.tranges (0.00 GB)
3. Fix Shapes for Autoregressive Decode
CGC compiles static graphs. The downloaded model still has symbolic batch_size, sequence_length and past_sequence_length dimensions, so fix_shapes pins them for the decode step — one new token attending to a 1023-entry KV cache — and runs ONNX Runtime's quantization pre-processing (shape inference and constant folding) so every tensor has a known shape before custom-op matching. The same helper produces the prefill layout with mode="prefill" (1024 tokens in, empty cache); section 10 downloads that graph pre-built.
Runtime: ~1 minute
decode_seq_onnx = "qwen3_seq1024.onnx"
fix_shapes(decode_onnx, decode_seq_onnx, seq_len=SEQ_LEN, mode="autoregressive")
Loading model from qwen_0_36.onnx...
Fixing shapes for autoregressive mode: seq_len=1024
Saving intermediate fixed model to fixed.onnx...
Running quantization pre-processing...
Shape fixing complete! Output saved to qwen3_seq1024.onnx
4. Replace Attention, Projections and the LM Head with Custom Ops
CGC compiles most of the graph on its own. Three patterns are swapped for CCL (Chimera Compute Language) kernels from the SDK's neural-network block library, because a fused kernel keeps the whole operation inside OCM and reads each weight exactly once per token:
| ONNX subgraph | Custom op | What it buys |
|---|---|---|
| Q/K/V projections → QK-RMSNorm → RoPE → grouped-query attention over the KV cache | nn::qwen3Attention | One kernel per layer appends to the cache and computes attention in OCM |
| Gate and up projections + SiLU | nn::gateProj | The SwiGLU pair in a single pass over the hidden state |
| Every INT4-weight MatMul, including the 4096 × 151,936 LM head | linalg::channelwiseQuantMatMul | INT8 activations against INT4 weights packed eight per 32-bit word (v8i4), per-channel scales |
qwen3_custom_op_replacer pattern-matches these subgraphs across all 36 layers, turns the tensor ranges into the fixed-point fractional bits each kernel expects as template arguments, and packs the INT4 weights — the packing is what takes the time.
Runtime: ~30–45 minutes
decode_custom_onnx = "qwen3_custom_ops_seq1024.onnx"
qwen3_custom_op_replacer(
decode_seq_onnx,
decode_custom_onnx,
tranges_path,
num_heads=NUM_KV_HEADS,
embed_dim=EMBED_DIM,
seq_length=SEQ_LEN,
num_decoders=NUM_DECODERS,
)
Successfully saved modified model to qwen3_custom_ops_seq1024.onnx
The weight packing is the one piece worth seeing in full. Eight signed 4-bit values are stacked along the input dimension into one int32, so a [K, N] weight becomes [K/8, N] words that the kernel unpacks on the fly — the same layout the v8i4 vector type uses on the GPNPU:
source = Path("custom_op_match.py").read_text()
print(re.search(r"def pack_row_v8i4.*?\n return output\n", source, re.DOTALL).group(0))
def pack_row_v8i4(input_array):
"""Pack 8 rows of 4-bit values into 32-bit integers.
Parameters
----------
input_array : numpy.ndarray
The input 2D array with shape (height, width) to pack.
Returns
-------
numpy.ndarray
The packed array with shape (ceil(height/8), width).
"""
pack_factor = 8
height, width = input_array.shape
output_height = math.ceil(height / pack_factor)
shift_granularity = 32 // pack_factor
mask = (1 << shift_granularity) - 1
# Pad input if height is not divisible by pack_factor
if height % pack_factor != 0:
padding = pack_factor - (height % pack_factor)
input_array = np.pad(
input_array, ((0, padding), (0, 0)), mode="constant", constant_values=0
)
# Reshape to group rows by pack_factor
reshaped = input_array.reshape(output_height, pack_factor, width)
# Convert to int32 and mask to get unsigned values
unsigned_vals = reshaped.astype(np.int32) & mask
# Create shift amounts for each position in the pack
shifts = np.arange(pack_factor) * shift_granularity
shifts = shifts.reshape(pack_factor, 1)
# Apply shifts to each group
shifted = unsigned_vals * (1 << shifts)
# Combine all values in each group using bitwise OR (sum works since non-overlapping bits)
output = shifted.sum(axis=1, dtype=np.int32)
return output
5. Compile the Decode Graph with CGC
CGC turns the custom-op graph into C++ for the QC-P target: it schedules the 36 layers, tiles every tensor to fit the 2 MB OCM, and emits the kernel plus a const_tensor_data.bin holding the packed weights. The KV cache tensors and the attention mask are listed in io_to_ignore — the autoregressive runner in section 7 owns them across token steps, so CGC leaves them out of the kernel's I/O.
Memory: ~55 GB peak · Runtime: ~25 minutes
decode_job = ChimeraJob(
decode_custom_onnx,
hw_config=HW_CONFIG,
trange_file=tranges_path,
target_lang="QIL", # Quadric Intermediate Language
io_to_ignore=kv_cache_io_names(NUM_DECODERS),
)
decode_job.compile()
print(decode_job)
2026-10-23 01:03 - INFO - epu - chimera_job - START==================================onnx_ingest
2026-10-23 01:03 - INFO - epu - chimera_job - Numerical ranges provided
2026-10-23 01:04 - INFO - epu - util - START===============================optimize_relay
2026-10-23 01:04 - INFO - epu - util - START====================quantize_to_cpu_runnable_fx
2026-10-23 01:04 - INFO - epu - fx -
Source name Op Output 0 Range Output 0 Frac Bits
--------------------------------------------------- ----------------------------- ----------------------- --------------------
/model/embed_tokens/Gather contrib.epu.embedding [-0.628906f, 0.8125f] 31
/model/layers.0/input_layernorm/Mul_1 contrib.epu.rms_norm [-0.359118f, 0.37561f] 31
/model/layers.0/self_attn/q_proj/MatMul_smooth_mul multiply [-0.287793f, 0.307279f] 31
/model/layers.0/self_attn/k_proj/MatMul_smooth_mul multiply [-0.247679f, 0.223768f] 31
/model/layers.0/self_attn/v_proj/MatMul_smooth_mul multiply [-0.100749f, 0.107016f] 31
CustomOp/linalg::channelwiseQuantMatMul<43>3 contrib.epu.quadric_custom_op [-2.25614f, 3.85482f] 29
/model/layers.0/Add add [-2.30253f, 4.25466f] 28
/model/layers.0/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-1.23024f, 1.59211f] 29
/model/layers.0/mlp/gate_proj/MatMul_smooth_mul multiply [-0.682836f, 0.548619f] 28
CustomOp/nn::gateProj<41, 29>1 contrib.epu.quadric_custom_op [-0.278465f, 3.85478f] 29
/model/layers.0/mlp/up_proj/MatMul_smooth_mul multiply [-0.421256f, 0.474667f] 29
CustomOp/linalg::channelwiseQuantMatMul<42>0 contrib.epu.quadric_custom_op [-3.34433f, 3.38188f] 29
/model/layers.0/mlp/Mul multiply [-11.3828f, 9.95375f] 27
/model/layers.0/mlp/down_proj/MatMul_smooth_mul multiply [-2.19349f, 2.29969f] 26
CustomOp/linalg::channelwiseQuantMatMul<38>2 contrib.epu.quadric_custom_op [-5.18762f, 16.7897f] 26
/model/layers.0/Add_1 add [-6.67269f, 19.2419f] 26
/model/layers.1/input_layernorm/Mul_1 contrib.epu.rms_norm [-1.30106f, 1.14032f] 30
/model/layers.1/self_attn/q_proj/MatMul_smooth_mul multiply [-0.549389f, 0.618498f] 28
/model/layers.1/self_attn/k_proj/MatMul_smooth_mul multiply [-0.547089f, 0.290293f] 28
/model/layers.1/self_attn/v_proj/MatMul_smooth_mul multiply [-0.18635f, 0.285787f] 28
CustomOp/linalg::channelwiseQuantMatMul<43>7 contrib.epu.quadric_custom_op [-1.41003f, 1.53298f] 30
/model/layers.1/Add add [-6.78994f, 19.2634f] 26
/model/layers.1/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-51.933f, 47.6116f] 25
/model/layers.1/mlp/gate_proj/MatMul_smooth_mul multiply [-3.17403f, 3.52375f] 22
CustomOp/nn::gateProj<38, 27>5 contrib.epu.quadric_custom_op [-0.278465f, 7.77009f] 28
/model/layers.1/mlp/up_proj/MatMul_smooth_mul multiply [-2.03129f, 2.22485f] 22
CustomOp/linalg::channelwiseQuantMatMul<39>4 contrib.epu.quadric_custom_op [-13.6112f, 16.4607f] 26
/model/layers.1/mlp/Mul multiply [-22.4947f, 19.6069f] 23
/model/layers.1/mlp/down_proj/MatMul_smooth_mul multiply [-3.95853f, 6.61841f] 24
CustomOp/linalg::channelwiseQuantMatMul<37>6 contrib.epu.quadric_custom_op [-12.4679f, 38.1379f] 25
/model/layers.1/Add_1 add [-18.1108f, 57.4013f] 25
/model/layers.2/input_layernorm/Mul_1 contrib.epu.rms_norm [-0.862898f, 1.28555f] 29
/model/layers.2/self_attn/q_proj/MatMul_smooth_mul multiply [-0.369643f, 0.667969f] 28
/model/layers.2/self_attn/k_proj/MatMul_smooth_mul multiply [-0.560969f, 0.593967f] 28
/model/layers.2/self_attn/v_proj/MatMul_smooth_mul multiply [-0.209467f, 0.313265f] 28
CustomOp/linalg::channelwiseQuantMatMul<42>11 contrib.epu.quadric_custom_op [-1.03252f, 1.31911f] 30
/model/layers.2/Add add [-18.1085f, 57.4384f] 25
/model/layers.2/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-82.5232f, 81.3135f] 24
/model/layers.2/mlp/gate_proj/MatMul_smooth_mul multiply [-12.8995f, 14.1477f] 25
CustomOp/nn::gateProj<35, 24>9 contrib.epu.quadric_custom_op [-0.278465f, 7.80494f] 28
/model/layers.2/mlp/up_proj/MatMul_smooth_mul multiply [-12.5991f, 14.1477f] 25
CustomOp/linalg::channelwiseQuantMatMul<35>8 contrib.epu.quadric_custom_op [-122.194f, 103.963f] 24
/model/layers.2/mlp/Mul multiply [-26.306f, 24.5606f] 21
/model/layers.2/mlp/down_proj/MatMul_smooth_mul multiply [-5.16373f, 2.29959f] 22
CustomOp/linalg::channelwiseQuantMatMul<37>10 contrib.epu.quadric_custom_op [-10.8904f, 26.5761f] 26
/model/layers.2/Add_1 add [-23.2706f, 81.9635f] 24
/model/layers.3/input_layernorm/Mul_1 contrib.epu.rms_norm [-1.24747f, 3.00854f] 28
/model/layers.3/self_attn/q_proj/MatMul_smooth_mul multiply [-0.43238f, 0.730476f] 27
/model/layers.3/self_attn/k_proj/MatMul_smooth_mul multiply [-0.371823f, 1.19244f] 27
/model/layers.3/self_attn/v_proj/MatMul_smooth_mul multiply [-0.246321f, 0.362407f] 27
CustomOp/linalg::channelwiseQuantMatMul<42>15 contrib.epu.quadric_custom_op [-1.84038f, 1.97427f] 30
/model/layers.3/Add add [-23.1594f, 81.7177f] 24
/model/layers.3/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-53.3995f, 62.6125f] 25
/model/layers.3/mlp/gate_proj/MatMul_smooth_mul multiply [-3.54248f, 4.5413f] 22
CustomOp/nn::gateProj<37, 26>13 contrib.epu.quadric_custom_op [-0.278465f, 8.11204f] 27
/model/layers.3/mlp/up_proj/MatMul_smooth_mul multiply [-2.58039f, 4.31592f] 22
CustomOp/linalg::channelwiseQuantMatMul<37>12 contrib.epu.quadric_custom_op [-17.4614f, 10.6391f] 26
/model/layers.3/mlp/Mul multiply [-23.0629f, 23.7062f] 23
/model/layers.3/mlp/down_proj/MatMul_smooth_mul multiply [-3.35231f, 3.48879f] 25
CustomOp/linalg::channelwiseQuantMatMul<39>14 contrib.epu.quadric_custom_op [-7.28584f, 7.4731f] 28
/model/layers.3/Add_1 add [-23.3718f, 85.3067f] 24
/model/layers.4/input_layernorm/Mul_1 contrib.epu.rms_norm [-1.80589f, 2.00268f] 29
/model/layers.4/self_attn/q_proj/MatMul_smooth_mul multiply [-0.898383f, 0.659137f] 28
/model/layers.4/self_attn/k_proj/MatMul_smooth_mul multiply [-0.707868f, 1.08651f] 28
/model/layers.4/self_attn/v_proj/MatMul_smooth_mul multiply [-0.439939f, 0.234896f] 28
CustomOp/linalg::channelwiseQuantMatMul<42>19 contrib.epu.quadric_custom_op [-3.53964f, 2.62418f] 29
/model/layers.4/Add add [-23.2038f, 85.5677f] 24
/model/layers.4/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-32.953f, 41.0647f] 25
/model/layers.4/mlp/gate_proj/MatMul_smooth_mul multiply [-3.60791f, 4.06511f] 25
CustomOp/nn::gateProj<37, 27>17 contrib.epu.quadric_custom_op [-0.278465f, 9.32974f] 27
/model/layers.4/mlp/up_proj/MatMul_smooth_mul multiply [-2.50043f, 4.23358f] 25
CustomOp/linalg::channelwiseQuantMatMul<37>16 contrib.epu.quadric_custom_op [-8.54877f, 10.1004f] 27
/model/layers.4/mlp/Mul multiply [-20.2387f, 20.039f] 24
/model/layers.4/mlp/down_proj/MatMul_smooth_mul multiply [-2.69153f, 2.8874f] 26
CustomOp/linalg::channelwiseQuantMatMul<39>18 contrib.epu.quadric_custom_op [-6.69227f, 12.1684f] 27
/model/layers.4/Add_1 add [-22.4392f, 87.0669f] 24
/model/layers.5/input_layernorm/Mul_1 contrib.epu.rms_norm [-2.14681f, 3.19879f] 28
/model/layers.5/self_attn/q_proj/MatMul_smooth_mul multiply [-1.45607f, 0.922822f] 28
/model/layers.5/self_attn/k_proj/MatMul_smooth_mul multiply [-1.13808f, 1.1609f] 28
/model/layers.5/self_attn/v_proj/MatMul_smooth_mul multiply [-0.502765f, 0.449179f] 28
CustomOp/linalg::channelwiseQuantMatMul<42>23 contrib.epu.quadric_custom_op [-6.97809f, 4.76181f] 28
/model/layers.5/Add add [-21.9713f, 84.54f] 24
/model/layers.5/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-24.6416f, 19.9332f] 26
/model/layers.5/mlp/gate_proj/MatMul_smooth_mul multiply [-3.30293f, 4.25595f] 25
CustomOp/nn::gateProj<37, 27>21 contrib.epu.quadric_custom_op [-0.278465f, 7.52237f] 28
/model/layers.5/mlp/up_proj/MatMul_smooth_mul multiply [-1.51029f, 2.94375f] 25
CustomOp/linalg::channelwiseQuantMatMul<38>20 contrib.epu.quadric_custom_op [-6.38602f, 7.07606f] 28
/model/layers.5/mlp/Mul multiply [-18.2442f, 18.1234f] 25
/model/layers.5/mlp/down_proj/MatMul_smooth_mul multiply [-3.21112f, 2.15665f] 26
CustomOp/linalg::channelwiseQuantMatMul<38>22 contrib.epu.quadric_custom_op [-37.2974f, 18.5171f] 25
/model/layers.5/Add_1 add [-18.6574f, 63.1892f] 24
/model/layers.6/input_layernorm/Mul_1 contrib.epu.rms_norm [-1.99346f, 7.23117f] 27
/model/layers.6/self_attn/q_proj/MatMul_smooth_mul multiply [-0.737696f, 4.47784f] 27
/model/layers.6/self_attn/k_proj/MatMul_smooth_mul multiply [-0.58721f, 1.42655f] 27
/model/layers.6/self_attn/v_proj/MatMul_smooth_mul multiply [-0.422668f, 1.02092f] 27
CustomOp/linalg::channelwiseQuantMatMul<40>27 contrib.epu.quadric_custom_op [-31.7815f, 8.99221f] 26
/model/layers.6/Add add [-12.6824f, 38.3116f] 24
/model/layers.6/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-9.84406f, 199.177f] 23
/model/layers.6/mlp/gate_proj/MatMul_smooth_mul multiply [-0.969524f, 14.0399f] 22
CustomOp/nn::gateProj<34, 24>25 contrib.epu.quadric_custom_op [-0.278465f, 65.6915f] 24
/model/layers.6/mlp/up_proj/MatMul_smooth_mul multiply [-0.875245f, 14.5852f] 22
CustomOp/linalg::channelwiseQuantMatMul<34>24 contrib.epu.quadric_custom_op [-41.8197f, 74.1344f] 24
/model/layers.6/mlp/Mul multiply [-1282.21f, 4841.01f] 18
/model/layers.6/mlp/down_proj/MatMul_smooth_mul multiply [-112.681f, 254.839f] 18
CustomOp/linalg::channelwiseQuantMatMul<27>26 contrib.epu.quadric_custom_op [-2118.69f, 9638.07f] 17
/model/layers.6/Add_1 add [-2121.44f, 9654.1f] 17
/model/layers.7/input_layernorm/Mul_1 contrib.epu.rms_norm [-5.1855f, 10.441f] 27
/model/layers.7/self_attn/q_proj/MatMul_smooth_mul multiply [-1.18851f, 2.07018f] 27
/model/layers.7/self_attn/k_proj/MatMul_smooth_mul multiply [-1.36347f, 1.4916f] 26
/model/layers.7/self_attn/v_proj/MatMul_smooth_mul multiply [-0.655511f, 1.0774f] 27
CustomOp/linalg::channelwiseQuantMatMul<41>31 contrib.epu.quadric_custom_op [-4.27446f, 3.6406f] 28
/model/layers.7/Add add [-2121.23f, 9654.21f] 17
/model/layers.7/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-6.89828f, 54.4274f] 25
/model/layers.7/mlp/gate_proj/MatMul_smooth_mul multiply [-1.00739f, 4.84292f] 24
CustomOp/nn::gateProj<38, 27>29 contrib.epu.quadric_custom_op [-0.278465f, 11.2674f] 27
/model/layers.7/mlp/up_proj/MatMul_smooth_mul multiply [-1.13376f, 1.88627f] 24
CustomOp/linalg::channelwiseQuantMatMul<40>28 contrib.epu.quadric_custom_op [-13.397f, 10.9477f] 27
/model/layers.7/mlp/Mul multiply [-16.9021f, 17.2346f] 23
/model/layers.7/mlp/down_proj/MatMul_smooth_mul multiply [-2.6115f, 2.85751f] 26
CustomOp/linalg::channelwiseQuantMatMul<37>30 contrib.epu.quadric_custom_op [-6.76926f, 12.1386f] 27
/model/layers.7/Add_1 add [-2121.18f, 9654.34f] 17
/model/layers.8/input_layernorm/Mul_1 contrib.epu.rms_norm [-4.59068f, 9.9222f] 27
/model/layers.8/self_attn/q_proj/MatMul_smooth_mul multiply [-1.28958f, 1.87046f] 27
/model/layers.8/self_attn/k_proj/MatMul_smooth_mul multiply [-0.814643f, 1.79062f] 27
/model/layers.8/self_attn/v_proj/MatMul_smooth_mul multiply [-0.422026f, 0.896034f] 27
CustomOp/linalg::channelwiseQuantMatMul<41>35 contrib.epu.quadric_custom_op [-3.77026f, 7.3769f] 28
/model/layers.8/Add add [-2120.87f, 9653.89f] 17
/model/layers.8/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-7.8127f, 6.98587f] 28
/model/layers.8/mlp/gate_proj/MatMul_smooth_mul multiply [-1.26424f, 1.58606f] 27
CustomOp/nn::gateProj<39, 27>33 contrib.epu.quadric_custom_op [-0.278465f, 6.3098f] 28
/model/layers.8/mlp/up_proj/MatMul_smooth_mul multiply [-1.4333f, 1.11036f] 27
CustomOp/linalg::channelwiseQuantMatMul<40>32 contrib.epu.quadric_custom_op [-6.76552f, 5.97511f] 28
/model/layers.8/mlp/Mul multiply [-15.8356f, 18.0968f] 25
/model/layers.8/mlp/down_proj/MatMul_smooth_mul multiply [-2.78291f, 3.87402f] 27
CustomOp/linalg::channelwiseQuantMatMul<38>34 contrib.epu.quadric_custom_op [-10.3458f, 12.5126f] 27
/model/layers.8/Add_1 add [-2120.62f, 9654.73f] 17
/model/layers.9/input_layernorm/Mul_1 contrib.epu.rms_norm [-4.52299f, 10.3721f] 27
/model/layers.9/self_attn/q_proj/MatMul_smooth_mul multiply [-1.02019f, 1.58785f] 26
/model/layers.9/self_attn/k_proj/MatMul_smooth_mul multiply [-1.23027f, 1.70964f] 26
/model/layers.9/self_attn/v_proj/MatMul_smooth_mul multiply [-0.594754f, 0.902843f] 26
CustomOp/linalg::channelwiseQuantMatMul<39>39 contrib.epu.quadric_custom_op [-7.31632f, 7.46158f] 28
/model/layers.9/Add add [-2120.34f, 9654.1f] 17
/model/layers.9/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-8.45564f, 8.75462f] 27
/model/layers.9/mlp/gate_proj/MatMul_smooth_mul multiply [-1.24338f, 1.8245f] 27
CustomOp/nn::gateProj<39, 27>37 contrib.epu.quadric_custom_op [-0.278465f, 9.10514f] 27
/model/layers.9/mlp/up_proj/MatMul_smooth_mul multiply [-1.7809f, 1.67158f] 27
CustomOp/linalg::channelwiseQuantMatMul<39>36 contrib.epu.quadric_custom_op [-7.51536f, 6.51499f] 28
/model/layers.9/mlp/Mul multiply [-15.1261f, 14.7165f] 24
/model/layers.9/mlp/down_proj/MatMul_smooth_mul multiply [-2.19853f, 3.02141f] 27
CustomOp/linalg::channelwiseQuantMatMul<38>38 contrib.epu.quadric_custom_op [-9.86182f, 8.2895f] 27
/model/layers.9/Add_1 add [-2120.34f, 9656.53f] 17
/model/layers.10/input_layernorm/Mul_1 contrib.epu.rms_norm [-7.60531f, 17.5566f] 26
/model/layers.10/self_attn/q_proj/MatMul_smooth_mul multiply [-1.31638f, 2.45222f] 26
/model/layers.10/self_attn/k_proj/MatMul_smooth_mul multiply [-1.14123f, 2.85072f] 26
/model/layers.10/self_attn/v_proj/MatMul_smooth_mul multiply [-0.635305f, 0.964866f] 26
CustomOp/linalg::channelwiseQuantMatMul<40>43 contrib.epu.quadric_custom_op [-7.399f, 5.84284f] 28
/model/layers.10/Add add [-2119.36f, 9655.58f] 17
/model/layers.10/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-8.33987f, 9.41487f] 27
/model/layers.10/mlp/gate_proj/MatMul_smooth_mul multiply [-1.41946f, 1.68869f] 26
CustomOp/nn::gateProj<39, 27>41 contrib.epu.quadric_custom_op [-0.278465f, 8.82315f] 27
/model/layers.10/mlp/up_proj/MatMul_smooth_mul multiply [-1.41946f, 1.19373f] 27
CustomOp/linalg::channelwiseQuantMatMul<40>40 contrib.epu.quadric_custom_op [-5.76626f, 8.25113f] 27
/model/layers.10/mlp/Mul multiply [-15.7846f, 20.8058f] 24
/model/layers.10/mlp/down_proj/MatMul_smooth_mul multiply [-5.70319f, 2.03262f] 26
CustomOp/linalg::channelwiseQuantMatMul<37>42 contrib.epu.quadric_custom_op [-5.03255f, 18.6078f] 26
/model/layers.10/Add_1 add [-2119.79f, 9656.97f] 17
/model/layers.11/input_layernorm/Mul_1 contrib.epu.rms_norm [-5.77417f, 10.4802f] 27
/model/layers.11/self_attn/q_proj/MatMul_smooth_mul multiply [-1.24383f, 1.49914f] 28
/model/layers.11/self_attn/k_proj/MatMul_smooth_mul multiply [-0.911936f, 1.83978f] 28
/model/layers.11/self_attn/v_proj/MatMul_smooth_mul multiply [-0.494405f, 0.873254f] 28
CustomOp/linalg::channelwiseQuantMatMul<40>47 contrib.epu.quadric_custom_op [-7.13255f, 7.60828f] 28
/model/layers.11/Add add [-2118.77f, 9655.72f] 17
/model/layers.11/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-9.30534f, 7.62255f] 27
/model/layers.11/mlp/gate_proj/MatMul_smooth_mul multiply [-1.12876f, 1.54745f] 26
CustomOp/nn::gateProj<39, 27>45 contrib.epu.quadric_custom_op [-0.278465f, 12.7414f] 27
/model/layers.11/mlp/up_proj/MatMul_smooth_mul multiply [-1.37915f, 0.865526f] 26
CustomOp/linalg::channelwiseQuantMatMul<40>44 contrib.epu.quadric_custom_op [-8.25134f, 6.12666f] 27
/model/layers.11/mlp/Mul multiply [-12.4377f, 12.5137f] 24
/model/layers.11/mlp/down_proj/MatMul_smooth_mul multiply [-3.31324f, 2.34436f] 27
CustomOp/linalg::channelwiseQuantMatMul<38>46 contrib.epu.quadric_custom_op [-7.59268f, 11.5153f] 27
/model/layers.11/Add_1 add [-2118.11f, 9657.75f] 17
/model/layers.12/input_layernorm/Mul_1 contrib.epu.rms_norm [-6.76928f, 11.1257f] 27
/model/layers.12/self_attn/q_proj/MatMul_smooth_mul multiply [-1.31845f, 1.85694f] 28
/model/layers.12/self_attn/k_proj/MatMul_smooth_mul multiply [-1.1593f, 1.7855f] 27
/model/layers.12/self_attn/v_proj/MatMul_smooth_mul multiply [-0.844806f, 0.934658f] 28
CustomOp/linalg::channelwiseQuantMatMul<40>51 contrib.epu.quadric_custom_op [-7.8656f, 13.7375f] 27
/model/layers.12/Add add [-2117.66f, 9657.01f] 17
/model/layers.12/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-10.1594f, 7.15519f] 27
/model/layers.12/mlp/gate_proj/MatMul_smooth_mul multiply [-1.37687f, 1.39484f] 26
CustomOp/nn::gateProj<40, 27>49 contrib.epu.quadric_custom_op [-0.278465f, 7.39754f] 28
/model/layers.12/mlp/up_proj/MatMul_smooth_mul multiply [-1.92166f, 1.10387f] 26
CustomOp/linalg::channelwiseQuantMatMul<39>48 contrib.epu.quadric_custom_op [-9.58119f, 5.33228f] 27
/model/layers.12/mlp/Mul multiply [-60.1034f, 12.2375f] 24
/model/layers.12/mlp/down_proj/MatMul_smooth_mul multiply [-3.8142f, 3.11107f] 25
CustomOp/linalg::channelwiseQuantMatMul<37>50 contrib.epu.quadric_custom_op [-12.1023f, 17.851f] 26
/model/layers.12/Add_1 add [-2118.72f, 9659.33f] 17
/model/layers.13/input_layernorm/Mul_1 contrib.epu.rms_norm [-5.04587f, 9.31552f] 27
/model/layers.13/self_attn/q_proj/MatMul_smooth_mul multiply [-1.00549f, 1.73688f] 28
/model/layers.13/self_attn/k_proj/MatMul_smooth_mul multiply [-1.76774f, 1.88082f] 28
/model/layers.13/self_attn/v_proj/MatMul_smooth_mul multiply [-0.604113f, 0.910164f] 28
CustomOp/linalg::channelwiseQuantMatMul<40>55 contrib.epu.quadric_custom_op [-5.54142f, 8.70166f] 27
/model/layers.13/Add add [-2117.86f, 9659.14f] 17
/model/layers.13/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-11.9317f, 6.23108f] 27
/model/layers.13/mlp/gate_proj/MatMul_smooth_mul multiply [-1.47657f, 1.26196f] 26
CustomOp/nn::gateProj<40, 26>53 contrib.epu.quadric_custom_op [-0.278465f, 17.2259f] 26
/model/layers.13/mlp/up_proj/MatMul_smooth_mul multiply [-2.24165f, 1.21306f] 26
CustomOp/linalg::channelwiseQuantMatMul<39>52 contrib.epu.quadric_custom_op [-30.4806f, 32.1002f] 25
/model/layers.13/mlp/Mul multiply [-16.9783f, 33.7621f] 21
/model/layers.13/mlp/down_proj/MatMul_smooth_mul multiply [-2.68679f, 2.99732f] 25
CustomOp/linalg::channelwiseQuantMatMul<38>54 contrib.epu.quadric_custom_op [-11.2142f, 11.9456f] 27
/model/layers.13/Add_1 add [-2120.76f, 9664.39f] 17
/model/layers.14/input_layernorm/Mul_1 contrib.epu.rms_norm [-6.68111f, 13.0119f] 27
/model/layers.14/self_attn/q_proj/MatMul_smooth_mul multiply [-1.40054f, 1.87548f] 27
/model/layers.14/self_attn/k_proj/MatMul_smooth_mul multiply [-1.56909f, 1.68693f] 28
/model/layers.14/self_attn/v_proj/MatMul_smooth_mul multiply [-0.623238f, 0.820612f] 28
CustomOp/linalg::channelwiseQuantMatMul<40>59 contrib.epu.quadric_custom_op [-8.29877f, 19.2297f] 26
/model/layers.14/Add add [-2120.07f, 9663.34f] 17
/model/layers.14/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-14.412f, 6.25216f] 27
/model/layers.14/mlp/gate_proj/MatMul_smooth_mul multiply [-1.47538f, 1.42171f] 26
CustomOp/nn::gateProj<39, 26>57 contrib.epu.quadric_custom_op [-0.278465f, 17.3895f] 26
/model/layers.14/mlp/up_proj/MatMul_smooth_mul multiply [-2.12944f, 0.916271f] 26
CustomOp/linalg::channelwiseQuantMatMul<39>56 contrib.epu.quadric_custom_op [-24.6362f, 32.0892f] 25
/model/layers.14/mlp/Mul multiply [-84.1848f, 13.7615f] 21
/model/layers.14/mlp/down_proj/MatMul_smooth_mul multiply [-4.83579f, 2.83515f] 24
CustomOp/linalg::channelwiseQuantMatMul<36>58 contrib.epu.quadric_custom_op [-18.0127f, 15.0041f] 26
/model/layers.14/Add_1 add [-2123.29f, 9666.16f] 17
/model/layers.15/input_layernorm/Mul_1 contrib.epu.rms_norm [-7.40155f, 13.516f] 27
/model/layers.15/self_attn/q_proj/MatMul_smooth_mul multiply [-2.20786f, 2.05356f] 27
/model/layers.15/self_attn/k_proj/MatMul_smooth_mul multiply [-1.312f, 1.55167f] 27
/model/layers.15/self_attn/v_proj/MatMul_smooth_mul multiply [-0.944417f, 1.36199f] 28
CustomOp/linalg::channelwiseQuantMatMul<39>63 contrib.epu.quadric_custom_op [-6.83366f, 13.2874f] 27
/model/layers.15/Add add [-2122.7f, 9664.69f] 17
/model/layers.15/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-16.0488f, 6.0668f] 26
/model/layers.15/mlp/gate_proj/MatMul_smooth_mul multiply [-1.4562f, 1.5061f] 26
CustomOp/nn::gateProj<39, 26>61 contrib.epu.quadric_custom_op [-0.278465f, 19.5172f] 26
/model/layers.15/mlp/up_proj/MatMul_smooth_mul multiply [-3.49456f, 0.898868f] 26
CustomOp/linalg::channelwiseQuantMatMul<38>60 contrib.epu.quadric_custom_op [-16.5025f, 59.7275f] 25
/model/layers.15/mlp/Mul multiply [-27.8356f, 48.4799f] 20
/model/layers.15/mlp/down_proj/MatMul_smooth_mul multiply [-3.85041f, 4.73925f] 25
CustomOp/linalg::channelwiseQuantMatMul<36>62 contrib.epu.quadric_custom_op [-16.8693f, 20.4889f] 26
/model/layers.15/Add_1 add [-2123.21f, 9668.25f] 17
/model/layers.16/input_layernorm/Mul_1 contrib.epu.rms_norm [-8.15026f, 15.6546f] 26
/model/layers.16/self_attn/q_proj/MatMul_smooth_mul multiply [-1.3261f, 2.04648f] 27
/model/layers.16/self_attn/k_proj/MatMul_smooth_mul multiply [-1.65707f, 1.55809f] 27
/model/layers.16/self_attn/v_proj/MatMul_smooth_mul multiply [-0.734148f, 0.882209f] 28
CustomOp/linalg::channelwiseQuantMatMul<38>67 contrib.epu.quadric_custom_op [-39.5976f, 22.6377f] 25
/model/layers.16/Add add [-2122.92f, 9666.15f] 17
/model/layers.16/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-36.8359f, 6.53895f] 25
/model/layers.16/mlp/gate_proj/MatMul_smooth_mul multiply [-4.24899f, 2.25932f] 25
CustomOp/nn::gateProj<37, 23>65 contrib.epu.quadric_custom_op [-0.278465f, 78.4472f] 24
/model/layers.16/mlp/up_proj/MatMul_smooth_mul multiply [-4.45054f, 1.66161f] 25
CustomOp/linalg::channelwiseQuantMatMul<37>64 contrib.epu.quadric_custom_op [-39.9357f, 84.0414f] 24
/model/layers.16/mlp/Mul multiply [-191.33f, 4661.68f] 18
/model/layers.16/mlp/down_proj/MatMul_smooth_mul multiply [-40.6463f, 352.551f] 18
CustomOp/linalg::channelwiseQuantMatMul<27>66 contrib.epu.quadric_custom_op [-1202.39f, 13581.2f] 17
/model/layers.16/Add_1 add [-2130f, 13602.6f] 16
/model/layers.17/input_layernorm/Mul_1 contrib.epu.rms_norm [-7.88683f, 13.0965f] 27
/model/layers.17/self_attn/q_proj/MatMul_smooth_mul multiply [-1.33071f, 1.92274f] 27
/model/layers.17/self_attn/k_proj/MatMul_smooth_mul multiply [-1.34317f, 1.59226f] 27
/model/layers.17/self_attn/v_proj/MatMul_smooth_mul multiply [-0.929569f, 1.09737f] 28
CustomOp/linalg::channelwiseQuantMatMul<37>71 contrib.epu.quadric_custom_op [-14.5229f, 61.5807f] 25
/model/layers.17/Add add [-2129.24f, 13637f] 16
/model/layers.17/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-12.2827f, 6.15562f] 27
/model/layers.17/mlp/gate_proj/MatMul_smooth_mul multiply [-1.52264f, 1.37121f] 26
CustomOp/nn::gateProj<39, 28>69 contrib.epu.quadric_custom_op [-0.278465f, 7.73579f] 28
/model/layers.17/mlp/up_proj/MatMul_smooth_mul multiply [-1.52896f, 0.967658f] 26
CustomOp/linalg::channelwiseQuantMatMul<40>68 contrib.epu.quadric_custom_op [-7.26674f, 35.3441f] 25
/model/layers.17/mlp/Mul multiply [-13.8208f, 9.58101f] 22
/model/layers.17/mlp/down_proj/MatMul_smooth_mul multiply [-3.27512f, 3.71965f] 27
CustomOp/linalg::channelwiseQuantMatMul<38>70 contrib.epu.quadric_custom_op [-21.4484f, 22.413f] 26
/model/layers.17/Add_1 add [-2128.21f, 13636.9f] 16
/model/layers.18/input_layernorm/Mul_1 contrib.epu.rms_norm [-10.6242f, 15.4063f] 26
/model/layers.18/self_attn/q_proj/MatMul_smooth_mul multiply [-1.42803f, 2.10296f] 27
/model/layers.18/self_attn/k_proj/MatMul_smooth_mul multiply [-1.48392f, 1.69128f] 27
/model/layers.18/self_attn/v_proj/MatMul_smooth_mul multiply [-0.998706f, 1.36009f] 28
CustomOp/linalg::channelwiseQuantMatMul<40>75 contrib.epu.quadric_custom_op [-11.2483f, 15.3732f] 27
/model/layers.18/Add add [-2127.25f, 13638.1f] 16
/model/layers.18/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-11.7031f, 6.12549f] 27
/model/layers.18/mlp/gate_proj/MatMul_smooth_mul multiply [-1.18675f, 1.43604f] 26
CustomOp/nn::gateProj<39, 27>73 contrib.epu.quadric_custom_op [-0.278465f, 8.13071f] 27
/model/layers.18/mlp/up_proj/MatMul_smooth_mul multiply [-1.42666f, 1.13167f] 26
CustomOp/linalg::channelwiseQuantMatMul<39>72 contrib.epu.quadric_custom_op [-6.42202f, 15.0935f] 27
/model/layers.18/mlp/Mul multiply [-9.16472f, 64.1296f] 24
/model/layers.18/mlp/down_proj/MatMul_smooth_mul multiply [-3.79416f, 8.76246f] 25
CustomOp/linalg::channelwiseQuantMatMul<35>74 contrib.epu.quadric_custom_op [-27.1561f, 62.6349f] 25
/model/layers.18/Add_1 add [-2127.74f, 13638.1f] 16
/model/layers.19/input_layernorm/Mul_1 contrib.epu.rms_norm [-16.3269f, 23.5058f] 26
/model/layers.19/self_attn/q_proj/MatMul_smooth_mul multiply [-1.87381f, 2.36709f] 27
/model/layers.19/self_attn/k_proj/MatMul_smooth_mul multiply [-1.99903f, 2.0809f] 27
/model/layers.19/self_attn/v_proj/MatMul_smooth_mul multiply [-0.96973f, 1.21001f] 27
CustomOp/linalg::channelwiseQuantMatMul<39>79 contrib.epu.quadric_custom_op [-9.69087f, 19.3656f] 26
/model/layers.19/Add add [-2127.19f, 13637.7f] 16
/model/layers.19/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-11.4451f, 6.57414f] 27
/model/layers.19/mlp/gate_proj/MatMul_smooth_mul multiply [-1.50948f, 1.27079f] 27
CustomOp/nn::gateProj<40, 27>77 contrib.epu.quadric_custom_op [-0.278465f, 8.66148f] 27
/model/layers.19/mlp/up_proj/MatMul_smooth_mul multiply [-1.77521f, 1.54593f] 27
CustomOp/linalg::channelwiseQuantMatMul<39>76 contrib.epu.quadric_custom_op [-6.79348f, 5.92249f] 28
/model/layers.19/mlp/Mul multiply [-10.5208f, 11.7491f] 25
/model/layers.19/mlp/down_proj/MatMul_smooth_mul multiply [-3.32359f, 3.64369f] 27
CustomOp/linalg::channelwiseQuantMatMul<36>78 contrib.epu.quadric_custom_op [-28.7991f, 32.1844f] 25
/model/layers.19/Add_1 add [-2127.2f, 13637.6f] 16
/model/layers.20/input_layernorm/Mul_1 contrib.epu.rms_norm [-17.2382f, 20.6114f] 26
/model/layers.20/self_attn/q_proj/MatMul_smooth_mul multiply [-1.79777f, 2.35375f] 27
/model/layers.20/self_attn/k_proj/MatMul_smooth_mul multiply [-2.07954f, 1.97869f] 27
/model/layers.20/self_attn/v_proj/MatMul_smooth_mul multiply [-0.97567f, 1.51478f] 27
CustomOp/linalg::channelwiseQuantMatMul<40>83 contrib.epu.quadric_custom_op [-11.7166f, 31.5278f] 26
/model/layers.20/Add add [-2126.62f, 13634.6f] 16
/model/layers.20/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-12.2707f, 7.12023f] 27
/model/layers.20/mlp/gate_proj/MatMul_smooth_mul multiply [-1.27321f, 1.48049f] 28
CustomOp/nn::gateProj<40, 27>81 contrib.epu.quadric_custom_op [-0.278465f, 8.94129f] 27
/model/layers.20/mlp/up_proj/MatMul_smooth_mul multiply [-1.25212f, 1.368f] 27
CustomOp/linalg::channelwiseQuantMatMul<40>80 contrib.epu.quadric_custom_op [-6.33998f, 8.02105f] 27
/model/layers.20/mlp/Mul multiply [-18.5987f, 19.8616f] 24
/model/layers.20/mlp/down_proj/MatMul_smooth_mul multiply [-4.02727f, 2.56306f] 26
CustomOp/linalg::channelwiseQuantMatMul<38>82 contrib.epu.quadric_custom_op [-16.8789f, 16.8205f] 26
/model/layers.20/Add_1 add [-2125.87f, 13634.7f] 16
/model/layers.21/input_layernorm/Mul_1 contrib.epu.rms_norm [-23.1988f, 24.3195f] 26
/model/layers.21/self_attn/q_proj/MatMul_smooth_mul multiply [-2.32063f, 2.84108f] 27
/model/layers.21/self_attn/k_proj/MatMul_smooth_mul multiply [-2.10805f, 2.21897f] 27
/model/layers.21/self_attn/v_proj/MatMul_smooth_mul multiply [-1.10305f, 1.53724f] 27
CustomOp/linalg::channelwiseQuantMatMul<39>87 contrib.epu.quadric_custom_op [-8.88642f, 21.276f] 26
/model/layers.21/Add add [-2124.82f, 13636.5f] 16
/model/layers.21/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-10.9728f, 7.28116f] 27
/model/layers.21/mlp/gate_proj/MatMul_smooth_mul multiply [-1.61932f, 1.36054f] 26
CustomOp/nn::gateProj<39, 27>85 contrib.epu.quadric_custom_op [-0.278465f, 9.86578f] 27
/model/layers.21/mlp/up_proj/MatMul_smooth_mul multiply [-1.27146f, 1.53064f] 26
CustomOp/linalg::channelwiseQuantMatMul<39>84 contrib.epu.quadric_custom_op [-10.277f, 10.0276f] 27
/model/layers.21/mlp/Mul multiply [-41.594f, 53.735f] 24
/model/layers.21/mlp/down_proj/MatMul_smooth_mul multiply [-5.86662f, 8.1078f] 25
CustomOp/linalg::channelwiseQuantMatMul<37>86 contrib.epu.quadric_custom_op [-32.024f, 16.017f] 25
/model/layers.21/Add_1 add [-2124.91f, 13636.6f] 16
/model/layers.22/input_layernorm/Mul_1 contrib.epu.rms_norm [-30.8369f, 32.7385f] 25
/model/layers.22/self_attn/q_proj/MatMul_smooth_mul multiply [-2.71083f, 3.39645f] 27
/model/layers.22/self_attn/k_proj/MatMul_smooth_mul multiply [-3.44054f, 3.20511f] 26
/model/layers.22/self_attn/v_proj/MatMul_smooth_mul multiply [-1.28456f, 1.37292f] 27
CustomOp/linalg::channelwiseQuantMatMul<39>91 contrib.epu.quadric_custom_op [-12.5842f, 31.0796f] 26
/model/layers.22/Add add [-2124.1f, 13635.5f] 16
/model/layers.22/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-10.0459f, 8.97793f] 27
/model/layers.22/mlp/gate_proj/MatMul_smooth_mul multiply [-1.56994f, 1.54569f] 27
CustomOp/nn::gateProj<39, 26>89 contrib.epu.quadric_custom_op [-0.278465f, 16.0594f] 26
/model/layers.22/mlp/up_proj/MatMul_smooth_mul multiply [-1.54481f, 1.40509f] 27
CustomOp/linalg::channelwiseQuantMatMul<39>88 contrib.epu.quadric_custom_op [-8.84424f, 9.51616f] 27
/model/layers.22/mlp/Mul multiply [-42.2895f, 37.618f] 23
/model/layers.22/mlp/down_proj/MatMul_smooth_mul multiply [-5.37248f, 5.52488f] 26
CustomOp/linalg::channelwiseQuantMatMul<37>90 contrib.epu.quadric_custom_op [-24.8563f, 17.5565f] 26
/model/layers.22/Add_1 add [-2123.61f, 13635.4f] 16
/model/layers.23/input_layernorm/Mul_1 contrib.epu.rms_norm [-32.5652f, 31.0209f] 25
/model/layers.23/self_attn/q_proj/MatMul_smooth_mul multiply [-3.54566f, 3.57557f] 27
/model/layers.23/self_attn/k_proj/MatMul_smooth_mul multiply [-2.95217f, 3.04757f] 27
/model/layers.23/self_attn/v_proj/MatMul_smooth_mul multiply [-1.23035f, 1.36422f] 27
CustomOp/linalg::channelwiseQuantMatMul<39>95 contrib.epu.quadric_custom_op [-8.13639f, 35.0445f] 25
/model/layers.23/Add add [-2123.06f, 13632.6f] 16
/model/layers.23/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-10.1758f, 10.4433f] 27
/model/layers.23/mlp/gate_proj/MatMul_smooth_mul multiply [-1.73753f, 1.58375f] 27
CustomOp/nn::gateProj<39, 26>93 contrib.epu.quadric_custom_op [-0.278465f, 15.0449f] 27
/model/layers.23/mlp/up_proj/MatMul_smooth_mul multiply [-1.42182f, 1.61803f] 27
CustomOp/linalg::channelwiseQuantMatMul<39>92 contrib.epu.quadric_custom_op [-12.3414f, 10.3189f] 27
/model/layers.23/mlp/Mul multiply [-64.6526f, 60.3269f] 23
/model/layers.23/mlp/down_proj/MatMul_smooth_mul multiply [-5.1301f, 6.31523f] 25
CustomOp/linalg::channelwiseQuantMatMul<37>94 contrib.epu.quadric_custom_op [-24.6908f, 31.6436f] 26
/model/layers.23/Add_1 add [-2122.79f, 13632.6f] 16
/model/layers.24/input_layernorm/Mul_1 contrib.epu.rms_norm [-50.8646f, 39.9522f] 25
/model/layers.24/self_attn/q_proj/MatMul_smooth_mul multiply [-4.28403f, 3.79681f] 26
/model/layers.24/self_attn/k_proj/MatMul_smooth_mul multiply [-3.08274f, 3.52826f] 27
/model/layers.24/self_attn/v_proj/MatMul_smooth_mul multiply [-2.11367f, 2.06183f] 27
CustomOp/linalg::channelwiseQuantMatMul<38>99 contrib.epu.quadric_custom_op [-16.7965f, 31.3655f] 26
/model/layers.24/Add add [-2123.99f, 13635.1f] 16
/model/layers.24/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-10.2601f, 12.5924f] 27
/model/layers.24/mlp/gate_proj/MatMul_smooth_mul multiply [-1.79581f, 1.95177f] 26
CustomOp/nn::gateProj<39, 26>97 contrib.epu.quadric_custom_op [-0.278465f, 17.9184f] 26
/model/layers.24/mlp/up_proj/MatMul_smooth_mul multiply [-1.58951f, 1.69809f] 26
CustomOp/linalg::channelwiseQuantMatMul<39>96 contrib.epu.quadric_custom_op [-11.0467f, 11.354f] 27
/model/layers.24/mlp/Mul multiply [-107.554f, 90.9104f] 23
/model/layers.24/mlp/down_proj/MatMul_smooth_mul multiply [-6.40439f, 6.75044f] 25
CustomOp/linalg::channelwiseQuantMatMul<36>98 contrib.epu.quadric_custom_op [-22.1531f, 38.053f] 25
/model/layers.24/Add_1 add [-2123.99f, 13635.3f] 16
/model/layers.25/input_layernorm/Mul_1 contrib.epu.rms_norm [-40.2525f, 33.3074f] 25
/model/layers.25/self_attn/q_proj/MatMul_smooth_mul multiply [-3.55716f, 3.96355f] 26
/model/layers.25/self_attn/k_proj/MatMul_smooth_mul multiply [-2.68001f, 3.21491f] 27
/model/layers.25/self_attn/v_proj/MatMul_smooth_mul multiply [-1.83399f, 1.64135f] 27
CustomOp/linalg::channelwiseQuantMatMul<39>103 contrib.epu.quadric_custom_op [-10.4602f, 17.5029f] 26
/model/layers.25/Add add [-2123.75f, 13635.5f] 16
/model/layers.25/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-11.309f, 12.7584f] 27
/model/layers.25/mlp/gate_proj/MatMul_smooth_mul multiply [-1.95923f, 2.00956f] 26
CustomOp/nn::gateProj<39, 26>101 contrib.epu.quadric_custom_op [-0.278465f, 20.8095f] 26
/model/layers.25/mlp/up_proj/MatMul_smooth_mul multiply [-1.65599f, 1.83448f] 26
CustomOp/linalg::channelwiseQuantMatMul<39>100 contrib.epu.quadric_custom_op [-16.033f, 12.6045f] 26
/model/layers.25/mlp/Mul multiply [-117.555f, 109.813f] 22
/model/layers.25/mlp/down_proj/MatMul_smooth_mul multiply [-7.04222f, 8.54142f] 24
CustomOp/linalg::channelwiseQuantMatMul<36>102 contrib.epu.quadric_custom_op [-19.0471f, 36.3233f] 25
/model/layers.25/Add_1 add [-2123.73f, 13635.5f] 16
/model/layers.26/input_layernorm/Mul_1 contrib.epu.rms_norm [-50.6021f, 38.8768f] 25
/model/layers.26/self_attn/q_proj/MatMul_smooth_mul multiply [-4.46383f, 4.41033f] 26
/model/layers.26/self_attn/k_proj/MatMul_smooth_mul multiply [-3.09661f, 3.23338f] 27
/model/layers.26/self_attn/v_proj/MatMul_smooth_mul multiply [-1.95177f, 1.62671f] 27
CustomOp/linalg::channelwiseQuantMatMul<40>107 contrib.epu.quadric_custom_op [-8.51294f, 19.8805f] 26
/model/layers.26/Add add [-2124.04f, 13636.4f] 16
/model/layers.26/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-12.5561f, 15.4058f] 26
/model/layers.26/mlp/gate_proj/MatMul_smooth_mul multiply [-2.00442f, 1.59836f] 26
CustomOp/nn::gateProj<39, 26>105 contrib.epu.quadric_custom_op [-0.278465f, 22.6042f] 26
/model/layers.26/mlp/up_proj/MatMul_smooth_mul multiply [-1.56655f, 2.01047f] 26
CustomOp/linalg::channelwiseQuantMatMul<38>104 contrib.epu.quadric_custom_op [-12.8656f, 13.9187f] 27
/model/layers.26/mlp/Mul multiply [-105.01f, 132.743f] 22
/model/layers.26/mlp/down_proj/MatMul_smooth_mul multiply [-6.63407f, 7.95295f] 24
CustomOp/linalg::channelwiseQuantMatMul<36>106 contrib.epu.quadric_custom_op [-21.3917f, 49.6387f] 25
/model/layers.26/Add_1 add [-2124.03f, 13636.6f] 16
/model/layers.27/input_layernorm/Mul_1 contrib.epu.rms_norm [-61.2838f, 48.1027f] 25
/model/layers.27/self_attn/q_proj/MatMul_smooth_mul multiply [-4.33818f, 4.62879f] 26
/model/layers.27/self_attn/k_proj/MatMul_smooth_mul multiply [-2.61422f, 3.56265f] 26
/model/layers.27/self_attn/v_proj/MatMul_smooth_mul multiply [-2.02177f, 2.13091f] 27
CustomOp/linalg::channelwiseQuantMatMul<39>111 contrib.epu.quadric_custom_op [-8.8824f, 19.0919f] 26
/model/layers.27/Add add [-2124.55f, 13638.8f] 16
/model/layers.27/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-14.5542f, 17.2044f] 26
/model/layers.27/mlp/gate_proj/MatMul_smooth_mul multiply [-1.75765f, 1.86848f] 26
CustomOp/nn::gateProj<39, 26>109 contrib.epu.quadric_custom_op [-0.278465f, 19.2789f] 26
/model/layers.27/mlp/up_proj/MatMul_smooth_mul multiply [-3.00212f, 1.82197f] 26
CustomOp/linalg::channelwiseQuantMatMul<38>108 contrib.epu.quadric_custom_op [-13.1627f, 12.5415f] 27
/model/layers.27/mlp/Mul multiply [-137.571f, 122.284f] 23
/model/layers.27/mlp/down_proj/MatMul_smooth_mul multiply [-8.10809f, 8.06381f] 24
CustomOp/linalg::channelwiseQuantMatMul<36>110 contrib.epu.quadric_custom_op [-29.7459f, 42.6272f] 25
/model/layers.27/Add_1 add [-2124.49f, 13639.2f] 16
/model/layers.28/input_layernorm/Mul_1 contrib.epu.rms_norm [-65.7556f, 56.4749f] 24
/model/layers.28/self_attn/q_proj/MatMul_smooth_mul multiply [-4.29473f, 4.18318f] 26
/model/layers.28/self_attn/k_proj/MatMul_smooth_mul multiply [-3.32743f, 3.96202f] 27
/model/layers.28/self_attn/v_proj/MatMul_smooth_mul multiply [-2.12739f, 2.4797f] 27
CustomOp/linalg::channelwiseQuantMatMul<39>115 contrib.epu.quadric_custom_op [-16.1447f, 22.5356f] 26
/model/layers.28/Add add [-2125.41f, 13642.1f] 16
/model/layers.28/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-13.3559f, 17.9234f] 26
/model/layers.28/mlp/gate_proj/MatMul_smooth_mul multiply [-2.34241f, 2.10349f] 23
CustomOp/nn::gateProj<38, 26>113 contrib.epu.quadric_custom_op [-0.278465f, 23.2141f] 26
/model/layers.28/mlp/up_proj/MatMul_smooth_mul multiply [-2.12235f, 1.94853f] 23
CustomOp/linalg::channelwiseQuantMatMul<38>112 contrib.epu.quadric_custom_op [-18.7225f, 14.9992f] 26
/model/layers.28/mlp/Mul multiply [-132.766f, 140.229f] 22
/model/layers.28/mlp/down_proj/MatMul_smooth_mul multiply [-14.6529f, 10.7982f] 24
CustomOp/linalg::channelwiseQuantMatMul<35>114 contrib.epu.quadric_custom_op [-38.7672f, 58.2323f] 25
/model/layers.28/Add_1 add [-2125.36f, 13642.5f] 16
/model/layers.29/input_layernorm/Mul_1 contrib.epu.rms_norm [-87.7913f, 63.9043f] 24
/model/layers.29/self_attn/q_proj/MatMul_smooth_mul multiply [-5.64585f, 4.53467f] 26
/model/layers.29/self_attn/k_proj/MatMul_smooth_mul multiply [-4.66197f, 4.02658f] 26
/model/layers.29/self_attn/v_proj/MatMul_smooth_mul multiply [-3.72409f, 3.62137f] 26
CustomOp/linalg::channelwiseQuantMatMul<39>119 contrib.epu.quadric_custom_op [-15.1006f, 23.0286f] 26
/model/layers.29/Add add [-2126.23f, 13643.9f] 16
/model/layers.29/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-14.5663f, 19.9309f] 26
/model/layers.29/mlp/gate_proj/MatMul_smooth_mul multiply [-1.95651f, 1.86312f] 25
CustomOp/nn::gateProj<39, 26>117 contrib.epu.quadric_custom_op [-0.278465f, 19.1744f] 26
/model/layers.29/mlp/up_proj/MatMul_smooth_mul multiply [-2.62913f, 2.32513f] 25
CustomOp/linalg::channelwiseQuantMatMul<38>116 contrib.epu.quadric_custom_op [-16.4508f, 15.0669f] 26
/model/layers.29/mlp/Mul multiply [-150.54f, 123.012f] 22
/model/layers.29/mlp/down_proj/MatMul_smooth_mul multiply [-11.6399f, 11.8174f] 24
CustomOp/linalg::channelwiseQuantMatMul<35>118 contrib.epu.quadric_custom_op [-49.9533f, 76.5029f] 24
/model/layers.29/Add_1 add [-2126.19f, 13644f] 16
/model/layers.30/input_layernorm/Mul_1 contrib.epu.rms_norm [-97.8058f, 70.9147f] 24
/model/layers.30/self_attn/q_proj/MatMul_smooth_mul multiply [-8.32574f, 6.67203f] 26
/model/layers.30/self_attn/k_proj/MatMul_smooth_mul multiply [-5.20679f, 4.87224f] 26
/model/layers.30/self_attn/v_proj/MatMul_smooth_mul multiply [-3.51157f, 3.44255f] 26
CustomOp/linalg::channelwiseQuantMatMul<38>123 contrib.epu.quadric_custom_op [-26.3882f, 42.82f] 25
/model/layers.30/Add add [-2126.46f, 13648.6f] 16
/model/layers.30/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-16.1268f, 19.8625f] 26
/model/layers.30/mlp/gate_proj/MatMul_smooth_mul multiply [-2.32769f, 1.89138f] 27
CustomOp/nn::gateProj<38, 26>121 contrib.epu.quadric_custom_op [-0.278465f, 20.8345f] 26
/model/layers.30/mlp/up_proj/MatMul_smooth_mul multiply [-2.93709f, 3.14719f] 27
CustomOp/linalg::channelwiseQuantMatMul<38>120 contrib.epu.quadric_custom_op [-22.3626f, 17.1751f] 26
/model/layers.30/mlp/Mul multiply [-147.928f, 141.815f] 22
/model/layers.30/mlp/down_proj/MatMul_smooth_mul multiply [-15.4673f, 11.2169f] 24
CustomOp/linalg::channelwiseQuantMatMul<35>122 contrib.epu.quadric_custom_op [-53.484f, 106.991f] 24
/model/layers.30/Add_1 add [-2126.44f, 13648.7f] 16
/model/layers.31/input_layernorm/Mul_1 contrib.epu.rms_norm [-119.533f, 93.6401f] 24
/model/layers.31/self_attn/q_proj/MatMul_smooth_mul multiply [-7.37432f, 6.46369f] 26
/model/layers.31/self_attn/k_proj/MatMul_smooth_mul multiply [-5.50964f, 5.44922f] 25
/model/layers.31/self_attn/v_proj/MatMul_smooth_mul multiply [-3.79376f, 3.57184f] 26
CustomOp/linalg::channelwiseQuantMatMul<38>127 contrib.epu.quadric_custom_op [-17.2974f, 50.1237f] 25
/model/layers.31/Add add [-2126.83f, 13653.5f] 16
/model/layers.31/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-17.2516f, 18.0006f] 26
/model/layers.31/mlp/gate_proj/MatMul_smooth_mul multiply [-2.36857f, 2.28309f] 27
CustomOp/nn::gateProj<38, 26>125 contrib.epu.quadric_custom_op [-0.278465f, 21.2043f] 26
/model/layers.31/mlp/up_proj/MatMul_smooth_mul multiply [-2.60498f, 3.19835f] 27
CustomOp/linalg::channelwiseQuantMatMul<37>124 contrib.epu.quadric_custom_op [-18.9059f, 19.1897f] 26
/model/layers.31/mlp/Mul multiply [-147.492f, 171.108f] 22
/model/layers.31/mlp/down_proj/MatMul_smooth_mul multiply [-18.9471f, 14.1054f] 24
CustomOp/linalg::channelwiseQuantMatMul<34>126 contrib.epu.quadric_custom_op [-42.7217f, 101.298f] 24
/model/layers.31/Add_1 add [-2126.65f, 13653.4f] 16
/model/layers.32/input_layernorm/Mul_1 contrib.epu.rms_norm [-129.345f, 108.642f] 23
/model/layers.32/self_attn/q_proj/MatMul_smooth_mul multiply [-7.50556f, 7.79626f] 26
/model/layers.32/self_attn/k_proj/MatMul_smooth_mul multiply [-6.04744f, 5.6172f] 26
/model/layers.32/self_attn/v_proj/MatMul_smooth_mul multiply [-5.33153f, 4.74853f] 26
CustomOp/linalg::channelwiseQuantMatMul<37>131 contrib.epu.quadric_custom_op [-26.9792f, 52.8894f] 25
/model/layers.32/Add add [-2125.55f, 13676.5f] 16
/model/layers.32/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-17.899f, 23.7326f] 26
/model/layers.32/mlp/gate_proj/MatMul_smooth_mul multiply [-2.44445f, 3.21304f] 26
CustomOp/nn::gateProj<38, 26>129 contrib.epu.quadric_custom_op [-0.278465f, 25.1264f] 26
/model/layers.32/mlp/up_proj/MatMul_smooth_mul multiply [-4.51652f, 3.91687f] 26
CustomOp/linalg::channelwiseQuantMatMul<37>128 contrib.epu.quadric_custom_op [-18.7255f, 22.3183f] 26
/model/layers.32/mlp/Mul multiply [-190.096f, 285.781f] 21
/model/layers.32/mlp/down_proj/MatMul_smooth_mul multiply [-16.2074f, 13.687f] 24
CustomOp/linalg::channelwiseQuantMatMul<34>130 contrib.epu.quadric_custom_op [-67.0631f, 111.421f] 24
/model/layers.32/Add_1 add [-2125.27f, 13672.9f] 16
/model/layers.33/input_layernorm/Mul_1 contrib.epu.rms_norm [-187.593f, 150.813f] 23
/model/layers.33/self_attn/q_proj/MatMul_smooth_mul multiply [-8.17666f, 7.40637f] 25
/model/layers.33/self_attn/k_proj/MatMul_smooth_mul multiply [-7.68112f, 7.72248f] 25
/model/layers.33/self_attn/v_proj/MatMul_smooth_mul multiply [-6.35676f, 6.26976f] 25
CustomOp/linalg::channelwiseQuantMatMul<37>135 contrib.epu.quadric_custom_op [-22.3657f, 115.159f] 24
/model/layers.33/Add add [-2125.11f, 13717f] 16
/model/layers.33/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-29.27f, 27.8766f] 26
/model/layers.33/mlp/gate_proj/MatMul_smooth_mul multiply [-4.70675f, 3.16855f] 26
CustomOp/nn::gateProj<37, 26>133 contrib.epu.quadric_custom_op [-0.278465f, 20.9414f] 26
/model/layers.33/mlp/up_proj/MatMul_smooth_mul multiply [-7.67804f, 4.66365f] 27
CustomOp/linalg::channelwiseQuantMatMul<36>132 contrib.epu.quadric_custom_op [-21.9644f, 28.9324f] 26
/model/layers.33/mlp/Mul multiply [-191.84f, 180.474f] 21
/model/layers.33/mlp/down_proj/MatMul_smooth_mul multiply [-16.8717f, 15.8574f] 24
CustomOp/linalg::channelwiseQuantMatMul<34>134 contrib.epu.quadric_custom_op [-224.43f, 154.382f] 23
/model/layers.33/Add_1 add [-2126.49f, 13736f] 16
/model/layers.34/input_layernorm/Mul_1 contrib.epu.rms_norm [-190.924f, 150.511f] 23
/model/layers.34/self_attn/q_proj/MatMul_smooth_mul multiply [-9.52248f, 9.91184f] 25
/model/layers.34/self_attn/k_proj/MatMul_smooth_mul multiply [-7.38449f, 7.55077f] 26
/model/layers.34/self_attn/v_proj/MatMul_smooth_mul multiply [-5.32747f, 5.83561f] 26
CustomOp/linalg::channelwiseQuantMatMul<36>139 contrib.epu.quadric_custom_op [-134.891f, 252.592f] 23
/model/layers.34/Add add [-2127.57f, 13665.9f] 16
/model/layers.34/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-102.457f, 31.8715f] 24
/model/layers.34/mlp/gate_proj/MatMul_smooth_mul multiply [-11.9706f, 3.27663f] 24
CustomOp/nn::gateProj<35, 25>137 contrib.epu.quadric_custom_op [-0.278465f, 43.9184f] 25
/model/layers.34/mlp/up_proj/MatMul_smooth_mul multiply [-15.5429f, 4.71439f] 24
CustomOp/linalg::channelwiseQuantMatMul<34>136 contrib.epu.quadric_custom_op [-33.824f, 68.3458f] 24
/model/layers.34/mlp/Mul multiply [-1174.96f, 792.102f] 19
/model/layers.34/mlp/down_proj/MatMul_smooth_mul multiply [-110.529f, 78.4391f] 22
CustomOp/linalg::channelwiseQuantMatMul<31>138 contrib.epu.quadric_custom_op [-18980.2f, 553.5f] 16
/model/layers.34/Add_1 add [-7343.94f, 5351.69f] 16
/model/layers.35/input_layernorm/Mul_1 contrib.epu.rms_norm [-298.728f, 121.485f] 22
/model/layers.35/self_attn/q_proj/MatMul_smooth_mul multiply [-40.8404f, 9.94293f] 24
/model/layers.35/self_attn/k_proj/MatMul_smooth_mul multiply [-34.7046f, 8.17391f] 25
/model/layers.35/self_attn/v_proj/MatMul_smooth_mul multiply [-6.21749f, 4.66202f] 25
CustomOp/linalg::channelwiseQuantMatMul<36>143 contrib.epu.quadric_custom_op [-282.141f, 824.661f] 21
/model/layers.35/Add add [-7101.39f, 5072.41f] 16
/model/layers.35/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-601.076f, 47.2325f] 21
/model/layers.35/mlp/gate_proj/MatMul_smooth_mul multiply [-42.4844f, 4.64554f] 22
CustomOp/nn::gateProj<34, 24>141 contrib.epu.quadric_custom_op [-0.278465f, 109.277f] 24
/model/layers.35/mlp/up_proj/MatMul_smooth_mul multiply [-58.8737f, 4.981f] 22
CustomOp/linalg::channelwiseQuantMatMul<33>140 contrib.epu.quadric_custom_op [-159.195f, 162.484f] 23
/model/layers.35/mlp/Mul multiply [-1002.33f, 1755.35f] 16
/model/layers.35/mlp/down_proj/MatMul_smooth_mul multiply [-73.9084f, 97.6708f] 21
CustomOp/linalg::channelwiseQuantMatMul<29>142 contrib.epu.quadric_custom_op [-4812.36f, 1990.75f] 18
/model/layers.35/Add_1 add [-6990.34f, 2581.91f] 16
/model/norm/Mul_1 contrib.epu.rms_norm [-139.062f, 130.451f] 23
/Gather take [-93.9597f, 66.4554f] 23
/lm_head/MatMul_smooth_mul multiply [-5.11149f, 4.01418f] 24
CustomOp/linalg::channelwiseQuantMatMul<36>144 contrib.epu.quadric_custom_op [-15.0257f, 24.2741f] 26
2026-10-23 01:04 - INFO - epu - util - START====================build_cpu_runnable_fx_relay
2026-10-23 01:04 - INFO - epu - util - START=======================quantize_to_chimera_fx
2026-10-23 01:05 - INFO - epu - util - START=================================relay_to_tir
2026-10-23 01:05 - INFO - epu - util - START===========================relay_to_epu_relay
2026-10-23 01:05 - INFO - epu - util - START==============================adapt_and_order
2026-10-23 01:05 - INFO - epu - util - START==============================amend_ctrl_flow
2026-10-23 01:05 - INFO - epu - util - START=============================plan_lrm_virtual
2026-10-23 01:05 - INFO - epu - util - START==============================amend_ctrl_flow
2026-10-23 01:05 - INFO - epu - util - START===============================lrm_alloc_loop
2026-10-23 01:06 - INFO - epu - util - START==============================amend_ctrl_flow
2026-10-23 01:06 - INFO - epu - util - START================================lrm_splitting
2026-10-23 01:07 - INFO - epu - util - START=====================LrmUsageChecker.__call__
2026-10-23 01:07 - INFO - epu - util - START==============================ext_split_relay
2026-10-23 01:07 - INFO - epu - util - START=========================OcmAllocator.analyze
2026-10-23 01:07 - INFO - epu - util - START===============================Compiler.build
2026-10-23 01:07 - INFO - epu - util - START========================RelayToPrimFunc.build
2026-10-23 01:07 - INFO - epu - util - START=========================OcmAllocator.analyze
2026-10-23 01:07 - INFO - epu - util - START====================================build_tir
2026-10-23 01:07 - INFO - epu - chimera_job - Compilation of qwen3_custom_ops_seq1024_QC_P_1d7_2MB_4kB_128GBps_128GBps_16_OFF_x1_x1 successful
╒═════════════════════╤════════════════════════════════════════════════════════════════════════╕
│ Module Name │ qwen3_custom_ops_seq1024_QC_P_1d7_2MB_4kB_128GBps_128GBps_16_OFF_x1_x1 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────┤
│ ONNX File │ qwen3_custom_ops_seq1024.onnx │
├─────────────────────┼────────────────────────────────────────────────────────────────────────┤
│ Product Target │ QC-P │
├─────────────────────┼────────────────────────────────────────────────────────────────────────┤
│ Number of Cores │ 1 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────┤
│ ISS Clock Frequency │ 1.700 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────┤
│ L2M Size │ 2MB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────┤
│ LRM Size │ 4kB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────┤
│ External Read BW │ 128GBps │
├─────────────────────┼────────────────────────────────────────────────────────────────────────┤
│ External Write BW │ 128GBps │
├─────────────────────┼────────────────────────────────────────────────────────────────────────┤
│ MACS per PE │ 16 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────┤
│ Max L2M │ 0.172MB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────┤
│ Max LRM │ 0.609kB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────┤
│ Max Temp Ext Bytes │ 0.583MB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────┤
│ Network GMACs │ 7.870 │
╘═════════════════════╧════════════════════════════════════════════════════════════════════════╛
╒═════╤════════╤══════════════════════════╤═══════════════════╤══════════════════════════╤═══════╕
│ │ Type │ Name │ shape │ type │ mse │
╞═════╪════════╪══════════════════════════╪═══════════════════╪══════════════════════════╪═══════╡
│ 0 │ Input │ input_ids │ [1, 1] │ tensor[int32] │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 1 │ Input │ attention_mask │ [1, 1, 1024] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 2 │ Input │ sin │ [1, 1, 128] │ tensor[FixedPoint32<30>] │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 3 │ Input │ cos │ [1, 1, 128] │ tensor[FixedPoint32<30>] │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 4 │ Input │ past_key_values.0.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 5 │ Input │ past_key_values.0.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 6 │ Input │ past_key_values.1.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 7 │ Input │ past_key_values.1.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 8 │ Input │ past_key_values.2.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 9 │ Input │ past_key_values.2.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 10 │ Input │ past_key_values.3.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 11 │ Input │ past_key_values.3.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 12 │ Input │ past_key_values.4.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 13 │ Input │ past_key_values.4.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 14 │ Input │ past_key_values.5.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 15 │ Input │ past_key_values.5.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 16 │ Input │ past_key_values.6.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 17 │ Input │ past_key_values.6.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 18 │ Input │ past_key_values.7.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 19 │ Input │ past_key_values.7.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 20 │ Input │ past_key_values.8.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 21 │ Input │ past_key_values.8.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 22 │ Input │ past_key_values.9.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 23 │ Input │ past_key_values.9.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 24 │ Input │ past_key_values.10.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 25 │ Input │ past_key_values.10.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 26 │ Input │ past_key_values.11.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 27 │ Input │ past_key_values.11.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 28 │ Input │ past_key_values.12.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 29 │ Input │ past_key_values.12.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 30 │ Input │ past_key_values.13.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 31 │ Input │ past_key_values.13.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 32 │ Input │ past_key_values.14.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 33 │ Input │ past_key_values.14.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 34 │ Input │ past_key_values.15.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 35 │ Input │ past_key_values.15.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 36 │ Input │ past_key_values.16.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 37 │ Input │ past_key_values.16.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 38 │ Input │ past_key_values.17.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 39 │ Input │ past_key_values.17.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 40 │ Input │ past_key_values.18.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 41 │ Input │ past_key_values.18.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 42 │ Input │ past_key_values.19.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 43 │ Input │ past_key_values.19.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 44 │ Input │ past_key_values.20.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 45 │ Input │ past_key_values.20.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 46 │ Input │ past_key_values.21.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 47 │ Input │ past_key_values.21.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 48 │ Input │ past_key_values.22.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 49 │ Input │ past_key_values.22.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 50 │ Input │ past_key_values.23.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 51 │ Input │ past_key_values.23.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 52 │ Input │ past_key_values.24.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 53 │ Input │ past_key_values.24.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 54 │ Input │ past_key_values.25.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 55 │ Input │ past_key_values.25.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 56 │ Input │ past_key_values.26.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 57 │ Input │ past_key_values.26.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 58 │ Input │ past_key_values.27.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 59 │ Input │ past_key_values.27.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 60 │ Input │ past_key_values.28.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 61 │ Input │ past_key_values.28.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 62 │ Input │ past_key_values.29.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 63 │ Input │ past_key_values.29.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 64 │ Input │ past_key_values.30.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 65 │ Input │ past_key_values.30.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 66 │ Input │ past_key_values.31.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 67 │ Input │ past_key_values.31.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 68 │ Input │ past_key_values.32.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 69 │ Input │ past_key_values.32.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 70 │ Input │ past_key_values.33.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 71 │ Input │ past_key_values.33.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 72 │ Input │ past_key_values.34.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 73 │ Input │ past_key_values.34.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 74 │ Input │ past_key_values.35.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 75 │ Input │ past_key_values.35.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 76 │ Output │ logits │ [1, 151936] │ tensor[FixedPoint32<26>] │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 77 │ Output │ present.0.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 78 │ Output │ present.0.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 79 │ Output │ present.1.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 80 │ Output │ present.1.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 81 │ Output │ present.2.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 82 │ Output │ present.2.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 83 │ Output │ present.3.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 84 │ Output │ present.3.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 85 │ Output │ present.4.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 86 │ Output │ present.4.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 87 │ Output │ present.5.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 88 │ Output │ present.5.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 89 │ Output │ present.6.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 90 │ Output │ present.6.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 91 │ Output │ present.7.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 92 │ Output │ present.7.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 93 │ Output │ present.8.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 94 │ Output │ present.8.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 95 │ Output │ present.9.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 96 │ Output │ present.9.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 97 │ Output │ present.10.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 98 │ Output │ present.10.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 99 │ Output │ present.11.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 100 │ Output │ present.11.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 101 │ Output │ present.12.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 102 │ Output │ present.12.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 103 │ Output │ present.13.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 104 │ Output │ present.13.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 105 │ Output │ present.14.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 106 │ Output │ present.14.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 107 │ Output │ present.15.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 108 │ Output │ present.15.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 109 │ Output │ present.16.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 110 │ Output │ present.16.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 111 │ Output │ present.17.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 112 │ Output │ present.17.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 113 │ Output │ present.18.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 114 │ Output │ present.18.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 115 │ Output │ present.19.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 116 │ Output │ present.19.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 117 │ Output │ present.20.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 118 │ Output │ present.20.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 119 │ Output │ present.21.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 120 │ Output │ present.21.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 121 │ Output │ present.22.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 122 │ Output │ present.22.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 123 │ Output │ present.23.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 124 │ Output │ present.23.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 125 │ Output │ present.24.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 126 │ Output │ present.24.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 127 │ Output │ present.25.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 128 │ Output │ present.25.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 129 │ Output │ present.26.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 130 │ Output │ present.26.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 131 │ Output │ present.27.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 132 │ Output │ present.27.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 133 │ Output │ present.28.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 134 │ Output │ present.28.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 135 │ Output │ present.29.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 136 │ Output │ present.29.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 137 │ Output │ present.30.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 138 │ Output │ present.30.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 139 │ Output │ present.31.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 140 │ Output │ present.31.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 141 │ Output │ present.32.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 142 │ Output │ present.32.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 143 │ Output │ present.33.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 144 │ Output │ present.33.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 145 │ Output │ present.34.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 146 │ Output │ present.34.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 147 │ Output │ present.35.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 148 │ Output │ present.35.value │ [1, 8, 1024, 128] │ n/a │ n/a │
╘═════╧════════╧══════════════════════════╧═══════════════════╧══════════════════════════╧═══════╛
6. Prepare the Prompt
The tokenizer runs on the host, in the examples environment (transformers 4.51 or newer, which is where Qwen3 landed). prepare_inputs.py writes the four files the runner reads: input_logits.bin (token ids), sin.bin / cos.bin (RoPE tables for every position in the context) and prompt_len.bin.
The prompt is deliberately short — "The tallest" is two tokens — because every token costs a full pass through 8 billion parameters on the ISS.
!python3 prepare_inputs.py --prompt "The tallest" --new_tokens 1
2026-10-23 01:08:02.026503: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2026-10-23 01:08:02.028725: I external/local_tsl/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2026-10-23 01:08:02.067618: I external/local_tsl/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2026-10-23 01:08:02.196602: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-10-23 01:08:02.851555: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
[ 785 81617]
Prompt length: 2
7. Run Autoregressive Decode on the ISS
qwen3.cpp is the host-side runner around the CGC-generated kernel. It loads the packed weights, then loops over the prompt and the generated tokens: one kernel launch per token, the KV cache appended on-chip, the next token chosen with an on-chip top-1 over the 151,936 logits — prompt_len + new_tokens − 1 passes in total, all inside a single ISS session. Each generated token is streamed back to the host through a kernel parameter register as soon as it is picked.
sdk source compiles the runner and the kernel with Quadric LLVM for the Chimera GPNPU backend and executes them on the ISS with the 4-core QC-P configuration from section 1. The weights CGC emitted first move next to the runner, where it looks for them.
Compile: ~10 minutes · Run: ~30 minutes
stage_const_tensor("ccl_build/qwen3_custom_ops_seq1024*")
Staged const_tensor_data.bin (6.29 GB of INT4/INT8 weights) for the runner
PosixPath('const_tensor_data.bin')
%%bash
set -euo pipefail
sdk source -v qwen3.cpp \
--include-cgc-headers \
--target QC-P \
--num-cores 4 \
--ocm-size 2MB \
--macs-per-pe 16 \
--clock-freq-ghz 1 \
--ext-read-bw 24GBps \
--ext-write-bw 24GBps \
--ddr-axi-width 256 \
--quiet
2026-10-23 01:08 - DEBUG - sdk - cli - Executing command: cmake CMakeLists.txt -B /quadric/sdk-cli/examples/models/qwen/qwen3_8b/qwen3_QC-P_1d0_2MB_4kB_24GBps_24GBps_16_OFF_x1_x4/build -DNUM_GPNPUS=4 -DNUM_CORES=16 -DNUM_BORDERS=2 -DEPU_VERSION=2.0.0 -DQLLVM_ROOT_PATH=/quadric/llvm -DOCM_SIZE_KIBIBYTES=2048 -DNUM_PE_MACS=16 -DASSERT_MLS_WIDTH_LINE_ALIGN=ON -DSTACKOVERFLOW_ERROR=OFF -DHARDWARE_TARGET=OFF -DELS_AXI_WIDTH_BITS=256 -DELS_RD_BW_MBPS=196608 -DELS_WR_BW_MBPS=196608 -DCYCLE_TIME_NS=1.0
2026-10-23 01:08 - DEBUG - sdk - cli - Executing command: make -j8
[SDK-CLI] : Executing on QC-P simulator
2026-10-23 01:11 - DEBUG - sdk - cli - Executing command: ./qwen3_host -c --ddrRdBwTotal 196608.0 --ddrWrBwTotal 196608.0 --ddrAxiWidth 256 --instMemDepth 1310720 --ocmSize 2097152 --cycleTimeNS 1.0 --no-check --ddrRdAvgPct 100 --ddrRdMaxPct 100 --ddrWrAvgPct 100 --ddrWrMaxPct 100 --postKernelFlowTimeoutCycles 4000000 --clusterSize 4 --numClusters 1
2026-10-23 01:19 - WARNING - epu - core - No profile.json found. Not able to show performance results.
2026-10-23 01:19 - INFO - epu - core - If you would like to see performance results, add profiling statements to your source code.
[SDK-CLI] : Execution completed.
8. Decode the Output and Read the Profile
output_seq.bin holds the generated token ids; the tokenizer turns them back into text. The ISS profile then gives the cycle breakdown of core 0 — compute, MAC, data movement, stalls — and, from the total across all four cores, the per-token throughput. The run above made two passes through the network (a two-token prompt plus one generated token), so the cycles per token are the total divided by two.
!python3 decode_outputs.py
Token ids: [4752]
Generated text: building
plot_decode_profile("qwen3_QC-P*/**/profile_core0.json", clock_freq_ghz=1.0)
[SDK-CLI] : TotalCycles: 100,604,124
[SDK-CLI] : Executions/second: 10
compute : ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 18.25M
data_array : ▇▇▇▇▇▇▇ 7.108M
mac : ▇ 1.077M
data_ocm : ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 44.978M
data_external: ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 29.189M

'qwen3_QC-P_1d0_2MB_4kB_24GBps_24GBps_16_OFF_x1_x4/output/profile_core0.json'
prompt_tokens = int(np.fromfile("prompt_len.bin", dtype=np.int32)[0])
report_decode_throughput(
"qwen3_QC-P*/**/profile_core*.json",
passes=prompt_tokens + NEW_TOKENS - 1,
clock_freq_ghz=1.0,
silicon_clock_ghz=1.7,
)
Total cycles : 100,604,136 (2 passes, 4 core(s))
Cycles per token : 50,302,068
@ 1.0 GHz ISS clock : 50.3 ms/token -> 19.9 tokens/s
@ 1.7 GHz silicon clock: 29.6 ms/token -> 33.8 tokens/s
50302068.0
9. Compare Decode Against ONNX Runtime
A generated token that reads well is not a correctness check. The reference is the same shape-fixed graph before custom-op replacement, run in ONNX Runtime on the host: the prompt is replayed token by token with the same left-padded KV cache and the same per-position RoPE tables the runner consumed, and the logits of the last prompt pass are compared with the ones the runner wrote to output_logits.bin. The gate: the top-1 token must match, at least three of the top-5 tokens must be shared, and the two logit vectors must correlate with Pearson r ≥ 0.7 — so a divergence anywhere in the decoder stack surfaces here, not in a sentence someone has to judge by eye. Fixed-point custom kernels do not reproduce ONNX Runtime's fp32 execution of the quantized graph bit for bit, which is why the correlation threshold, not equality, is the gate.
compare_decode_with_ort(decode_seq_onnx)
Prompt tokens : [785, 81617]
ONNX Runtime top-5 : [4752, 315, 11, 883, 825]
ISS top-5 : [4752, 16301, 883, 11, 323]
Top-1 match : True top-5 overlap: 3/5
Logit Pearson r : 0.7655 max |diff|: 12.054
PASS: ISS decode matches the ONNX Runtime reference
{'ort_top5': [4752, 315, 11, 883, 825],
'iss_top5': [4752, 16301, 883, 11, 323],
'pearson': 0.7654599343142197,
'top5_overlap': 3}
Summary
| Model | Qwen3-8B — 36 layers, hidden size 4096, 32 query / 8 KV heads, 151,936-token vocabulary |
| Quantization | W4A8 SmoothQuant (α = 0.36): INT4 weights packed v8i4, INT8 activations |
| Target | QC-P × 4 cores, 2 MB OCM per core, 16 MACs/PE, 1 GHz, 24 GB/s DDR per core |
| Decode | Full autoregressive loop on the ISS — ~18–19 tokens/s average, 20–21 peak; logits match ONNX Runtime (top-1, top-5, Pearson r) |
| Custom Ops | nn::qwen3Attention, nn::gateProj, linalg::channelwiseQuantMatMul |
Key takeaways
- One export, two graphs. Decode and prefill differ only in their fixed shapes; weights, tensor ranges and custom ops are shared, so the prefill program in
qwen3_8b_prefill.ipynbis one shape-fix away from this decode kernel. - Fused attention keeps the KV cache on-chip.
nn::qwen3Attentionappends to the cache and attends over it inside OCM, so the runner never round-trips cache state through the host between tokens. - INT4 weights are a bandwidth play. Packing eight weights per 32-bit word halves the DDR traffic of an INT8 model per token — the quantity that sets decode speed on a memory-bound 8B model.
- Every GPNPU result is checked against a host reference. Decode logits are compared with an ONNX Runtime replay of the same prompt, on top-1, top-5 and Pearson correlation.
sdk sourcecloses the loop. One command compiles the CGC output and a hand-written host runner with Quadric LLVM and executes them on the cycle-accurate multicore ISS.
Citation
@article{qwen3technicalreport,
title = {Qwen3 Technical Report},
author = {Qwen Team},
journal = {arXiv preprint arXiv:2505.09388},
year = {2025}
}
