NOTE: The Jupyter Notebook below is included in the Chimera SDK and can be run interactively by running the following CLI command:
$ quadric sdk notebook
From the Jupyter Notebook window in your browser, select the notebook named /quadric/sdk-cli/examples/models/qwen/qwen3_8b/qwen3_8b_prefill.ipynb.
Qwen3-8B Prefill on Chimera GPNPU
This notebook takes the prefill half of Qwen3-8B inference — the whole 1024-token prompt pushed through all 36 decoder layers in one pass — and validates it on the ISS (Instruction Set Simulator) in a 4-core QC-P configuration, then compiles the complete prefill program with CGC (Chimera Graph Compiler).
Its companion, qwen3_8b.ipynb, covers decode: the autoregressive loop that runs after prefill has filled the KV cache. Both are cut from the same W4A8 SmoothQuant export, and share its tensor ranges and custom operators.
Why prefill is a separate notebook
Prefill and decode differ only in their fixed shapes, so one notebook could cover both. They are split so that each phase runs in its own kernel: prefill's ISS validation is the heaviest step in either phase, and it starts from a fresh session rather than continuing the one that ran decode.
Pipeline
Model: Qwen3-8B (8.2B parameters, 36 decoder layers, hidden size 4096, 32 query / 8 KV heads, 151,936-token vocabulary), W4A8 SmoothQuant (α = 0.36), 1024-token context
Note — this notebook validates one Qwen3-8B prefill decoder layer on the Chimera GPNPU.
- On the Chimera GPNPU: the decoder layer under test, compiled with CGC and run on the ISS — and, in section 3, the complete 36-layer prefill program
- On the host CPU: the decoder layers ahead of it, run through ONNX Runtime to produce its input, and the same layer again as the numerical reference
The host runs are the validation method, not a limit of the target: prefill is compiled in full for the GPNPU here, and one layer at a time is checked against ONNX Runtime because all 36 on the cycle-accurate simulator would take far longer than a notebook run allows.
1. Setup
The prefill graph, its weights and its calibrated ranges come from the same export as decode. The compile in section 3 uses the export's tensor ranges — the same file the decode notebook compiles against — so it is fetched here on its own, without the decode graph or its weights.
import re
from pathlib import Path
from IPython.display import Image, display
from tvm.contrib.epu.chimera_job.chimera_job import ChimeraJob
from tvm.contrib.epu.chimera_job.hw_config import HWConfig
from examples.models.qwen.qwen3_8b.prefill.custom_op_match import (
qwen3_custom_op_replacer as qwen3_prefill_custom_op_replacer,
)
from examples.models.qwen.qwen3_8b.prefill.run_decoder_pipeline import run_pipeline
from examples.models.qwen.qwen_helpers import kv_cache_io_names, raise_stack_limit
from qwen3_8b_helpers import (
download_decode_tranges,
download_prefill_model,
prepare_prefill_prompt,
project_prefill_latency,
)
%matplotlib inline
NUM_DECODERS = 36
EMBED_DIM = 4096
NUM_KV_HEADS = 8 # 32 query heads share 8 key/value heads (grouped-query attention)
SEQ_LEN = 1024 # prompt length for prefill
HW_CONFIG = HWConfig(product="QC-P", ocm_size="2MB", macs_per_pe=16)
PREFILL_DIR = Path("prefill") # prefill graph, tokenized prompt and per-decoder artifacts live here
raise_stack_limit()
tranges_path = str(download_decode_tranges(PREFILL_DIR))
Stack limit raised to 32 MB for CGC compilation
Downloading qwen_0_36.onnx.tranges...
Downloaded qwen_0_36.onnx.tranges (0.00 GB)
2. Validate One Decoder Layer on the ISS
Prefill pushes the whole 1024-token prompt through the network at once, so a single decoder layer already costs on the order of a billion ISS cycles. Rather than run all 36 layers, the pipeline in prefill/ validates the network the way it is built — one decoder at a time, each an independent subgraph with well-defined inputs and outputs:
- Cut decoder N out of the prefill graph (hidden state, attention mask,
cos,sinand KV cache in → hidden state and KV cache out) - Replace its ops with the prefill custom ops —
nn::qwen3PrefillAttention,nn::gateUpProjection,linalg::channelwiseQuantMatMul - Compile with CGC for the same 4-core QC-P target
- Run on the ISS through
qwen3_prefill_runner.cpp - Compare against ONNX Runtime with a tolerance derived from the layer's tensor ranges (
atol + rtol · |ref|), plus the Pearson correlation and a plot of the two output distributions
Feeding each decoder ONNX Runtime's output for the previous layer (isolated mode) exercises the kernels with a clean input, so any mismatch points at that layer alone. The same pipeline also runs in chained mode — the GPNPU's own output fed forward, gated on per-token cosine similarity — to measure how quantization noise accumulates with depth.
The prefill graph is downloaded pre-built: the same W4A8 export, shape-fixed with mode="prefill", with calibrated tensor ranges that set each decoder's fixed-point input and output formats.
prefill_onnx, _prefill_weights, prefill_tranges = map(str, download_prefill_model(PREFILL_DIR))
Downloading qwen3_8b_prefill_1024_context.onnx...
Downloaded qwen3_8b_prefill_1024_context.onnx (0.01 GB)
Downloading 27fe33ec-1773-11f1-b0a3-ea22fe300241...
Downloaded 27fe33ec-1773-11f1-b0a3-ea22fe300241 (6.27 GB)
Downloading qwen_0_36_calibrated.onnx.tranges...
Downloaded qwen_0_36_calibrated.onnx.tranges (0.00 GB)
Prefill tokenizes a real 1024-token prompt (a WikiText passage) rather than two words. The token ids, the RoPE tables and the prompt length land in prefill/, where the per-decoder runner reads them; decoder 0 also gets its own copy of the token ids since it consumes ids rather than a hidden state.
prepare_prefill_prompt(PREFILL_DIR / "prompts" / "0.txt", PREFILL_DIR)
Tokenized 0.txt -> prefill/{input_logits, sin, cos, prompt_len}.bin
Every decoder is cut on the same six input edges and three output edges. Decoder 0 takes the token ids; every later decoder takes the previous layer's residual-stream output:
source = (PREFILL_DIR / "cut_model.py").read_text()
edges = re.search(
r"def get_decoder_io_edges.*?return input_edges, output_edges\n", source, re.DOTALL
)
print(edges.group(0))
def get_decoder_io_edges(decoder_number):
"""Return (input_edges, output_edges) for cutting decoder N from Qwen3 prefill."""
hidden_input = (
"input_ids" if decoder_number == 0 else f"/model/layers.{decoder_number - 1}/Add_1_output_0"
)
input_edges = [
hidden_input,
"attention_mask",
"cos",
"sin",
f"past_key_values.{decoder_number}.key",
f"past_key_values.{decoder_number}.value",
]
output_edges = [
f"/model/layers.{decoder_number}/Add_1_output_0",
f"present.{decoder_number}.key",
f"present.{decoder_number}.value",
]
return input_edges, output_edges
Decoder 2 is the layer under test — a representative middle layer; decoders 1–34 share one compiled kernel. Its isolated-mode input is ONNX Runtime's output for decoders 0 → 1, so the bootstrap first runs decoders 0–2 through ONNX Runtime on the host, cutting each from the prefill graph and chaining the outputs forward as fixed-point inputs.
Runtime: a few minutes per decoder
VALIDATE_DECODER = 2
for decoder in range(VALIDATE_DECODER + 1):
run_pipeline(
decoder=decoder,
ort_only=True,
input_model=prefill_onnx,
tranges=prefill_tranges,
data_dir=str(PREFILL_DIR),
)
=== Decoder 0 pipeline (mode=isolated, cores=4) ===
════════════════════════════════════════════════════════════════
── Step 1: Cut decoder 0 from prefill/qwen3_8b_prefill_1024_context.onnx ──
Cutting decoder 0:
Input edges: ['input_ids', 'attention_mask', 'cos', 'sin', 'past_key_values.0.key', 'past_key_values.0.value']
Output edges: ['/model/layers.0/Add_1_output_0', 'present.0.key', 'present.0.value']
Successfully saved to /quadric/sdk-cli/examples/models/qwen/qwen3_8b/prefill/decoder_0/decoder_0.onnx
── Step 6: Run ORT (--ort-only) ──
✓ ORT complete → /quadric/sdk-cli/examples/models/qwen/qwen3_8b/prefill/decoder_0/onnx_runtime_outputs/output_0.npy
── Step 9: Chain output to decoder 1 ──
✓ Chained ORT 0 → 1 (FixedPoint32<26>, 16.0 MB)
✓ Chained: decoder_0/onnx_runtime_outputs/output_0.npy → decoder_1/input_logits_ort_float.bin
════════════════════════════════════════════════════════════════
Decoder 0 ort-only bootstrap complete
════════════════════════════════════════════════════════════════
=== Decoder 1 pipeline (mode=isolated, cores=4) ===
════════════════════════════════════════════════════════════════
── Step 1: Cut decoder 1 from prefill/qwen3_8b_prefill_1024_context.onnx ──
Cutting decoder 1:
Input edges: ['/model/layers.0/Add_1_output_0', 'attention_mask', 'cos', 'sin', 'past_key_values.1.key', 'past_key_values.1.value']
Output edges: ['/model/layers.1/Add_1_output_0', 'present.1.key', 'present.1.value']
Successfully saved to /quadric/sdk-cli/examples/models/qwen/qwen3_8b/prefill/decoder_1/decoder_1.onnx
── Step 6: Run ORT (--ort-only) ──
✓ ORT complete → /quadric/sdk-cli/examples/models/qwen/qwen3_8b/prefill/decoder_1/onnx_runtime_outputs/output_0.npy
── Step 9: Chain output to decoder 2 ──
✓ Chained ORT 1 → 2 (FixedPoint32<25>, 16.0 MB)
✓ Chained: decoder_1/onnx_runtime_outputs/output_0.npy → decoder_2/input_logits_ort_float.bin
════════════════════════════════════════════════════════════════
Decoder 1 ort-only bootstrap complete
════════════════════════════════════════════════════════════════
=== Decoder 2 pipeline (mode=isolated, cores=4) ===
════════════════════════════════════════════════════════════════
── Step 1: Cut decoder 2 from prefill/qwen3_8b_prefill_1024_context.onnx ──
Cutting decoder 2:
Input edges: ['/model/layers.1/Add_1_output_0', 'attention_mask', 'cos', 'sin', 'past_key_values.2.key', 'past_key_values.2.value']
Output edges: ['/model/layers.2/Add_1_output_0', 'present.2.key', 'present.2.value']
Successfully saved to /quadric/sdk-cli/examples/models/qwen/qwen3_8b/prefill/decoder_2/decoder_2.onnx
── Step 6: Run ORT (--ort-only) ──
✓ ORT complete → /quadric/sdk-cli/examples/models/qwen/qwen3_8b/prefill/decoder_2/onnx_runtime_outputs/output_0.npy
── Step 9: Chain output to decoder 3 ──
✓ Chained ORT 2 → 3 (FixedPoint32<24>, 16.0 MB)
✓ Chained: decoder_2/onnx_runtime_outputs/output_0.npy → decoder_3/input_logits_ort_float.bin
════════════════════════════════════════════════════════════════
Decoder 2 ort-only bootstrap complete
════════════════════════════════════════════════════════════════
Now the GPNPU side: custom ops, CGC compile, sdk source on the ISS, and the comparison. The gate passes when at most 50 of the 4.19 million output elements fall outside atol + rtol · |ref|, with atol derived from the layer's output range and rtol = 1 %; the printout also reports the mean and maximum absolute difference and the Pearson correlation (typically ≥ 0.998).
Compile: ~10 minutes · Run: ~30 minutes
run_pipeline(
decoder=VALIDATE_DECODER,
mode="isolated",
skip_cut=True, # the bootstrap already cut decoder_2.onnx
tranges=prefill_tranges,
rtol=1e-2,
max_failures=50,
data_dir=str(PREFILL_DIR),
)
=== Decoder 2 pipeline (mode=isolated, cores=4) ===
── Step 1: Skipping cut (--skip-cut) ──
── Step 2: Custom op matching ──
Warning: /model/layers.2/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Processing matmul ops for decoder 2...
Decoder 2 gate_up frac bits:
gate_combined_scale=5.992835e-02 frac_bits=35
up_combined_scale=5.992835e-02 frac_bits=35
gate_im_frac_bits=27
up_im_frac_bits=27
gate_output_frac_bits=24
up_output_frac_bits=24
Successfully saved modified model to /quadric/sdk-cli/examples/models/qwen/qwen3_8b/prefill/decoder_2/custom_op_decoder_2.onnx
✓ Custom op complete → /quadric/sdk-cli/examples/models/qwen/qwen3_8b/prefill/decoder_2/custom_op_decoder_2.onnx
── Step 3: CGC compile ──
Starting CGC compilation...
2026-10-23 00:21 - INFO - epu - chimera_job - START==================================onnx_ingest
2026-10-23 00:21 - INFO - epu - chimera_job - Numerical ranges provided
2026-10-23 00:21 - INFO - epu - util - START===============================optimize_relay
2026-10-23 00:21 - INFO - epu - util - START====================quantize_to_cpu_runnable_fx
2026-10-23 00:21 - INFO - epu - fx -
Source name Op Output 0 Range Output 0 Frac Bits
------------------------------------------------- ----------------------------- ---------------------- --------------------
/model/layers.2/input_layernorm/Mul_1 contrib.epu.rms_norm [-0.862898f, 1.48399f] 29
CustomOp/linalg::channelwiseQuantMatMul<42>2 contrib.epu.quadric_custom_op [-1.03252f, 1.31911f] 29
CustomOp/add<MultiCoreMode::All>2 contrib.epu.quadric_custom_op [-18.1085f, 57.4384f] 25
/model/layers.2/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-82.5232f, 81.3135f] 24
CustomOp/nn::gateUpProjection<27,27,24,24,35,35>0 contrib.epu.quadric_custom_op [-26.306f, 24.5606f] 26
CustomOp/linalg::channelwiseQuantMatMul<37>1 contrib.epu.quadric_custom_op [-10.8904f, 26.5761f] 26
CustomOp/add<MultiCoreMode::All>1 contrib.epu.quadric_custom_op [-23.2706f, 81.9635f] 24
2026-10-23 00:21 - INFO - epu - util - START====================build_cpu_runnable_fx_relay
2026-10-23 00:21 - INFO - epu - util - START=======================quantize_to_chimera_fx
2026-10-23 00:21 - INFO - epu - util - START=================================relay_to_tir
2026-10-23 00:21 - INFO - epu - util - START===========================relay_to_epu_relay
2026-10-23 00:21 - INFO - epu - util - START==============================adapt_and_order
2026-10-23 00:21 - INFO - epu - util - START==============================amend_ctrl_flow
2026-10-23 00:21 - INFO - epu - util - START=============================plan_lrm_virtual
2026-10-23 00:21 - INFO - epu - util - START==============================amend_ctrl_flow
2026-10-23 00:21 - INFO - epu - util - START===============================lrm_alloc_loop
2026-10-23 00:21 - INFO - epu - util - START==============================amend_ctrl_flow
2026-10-23 00:21 - INFO - epu - util - START================================lrm_splitting
2026-10-23 00:21 - INFO - epu - util - START=====================LrmUsageChecker.__call__
2026-10-23 00:21 - INFO - epu - util - START==============================ext_split_relay
2026-10-23 00:21 - INFO - epu - util - START=========================OcmAllocator.analyze
2026-10-23 00:21 - INFO - epu - util - START===============================Compiler.build
2026-10-23 00:21 - INFO - epu - util - START========================RelayToPrimFunc.build
2026-10-23 00:21 - INFO - epu - util - START=========================OcmAllocator.analyze
2026-10-23 00:21 - INFO - epu - util - START====================================build_tir
2026-10-23 00:21 - INFO - epu - chimera_job - Compilation of custom_op_decoder_2_QC_P_1d7_2MB_4kB_128GBps_128GBps_16_OFF_x1_x4 successful
Compilation complete!
╒═════════════════════╤═══════════════════════════════════════════════════════════════════════════════════════════╕
│ Module Name │ custom_op_decoder_2_QC_P_1d7_2MB_4kB_128GBps_128GBps_16_OFF_x1_x4 │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ ONNX File │ /quadric/sdk-cli/examples/models/qwen/qwen3_8b/prefill/decoder_2/custom_op_decoder_2.onnx │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ Product Target │ QC-P │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ Number of Cores │ 4 │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ ISS Clock Frequency │ 1.700 │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ L2M Size │ 2MB │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ LRM Size │ 4kB │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ External Read BW │ 128GBps │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ External Write BW │ 128GBps │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ MACS per PE │ 16 │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ Max L2M │ 0.000MB │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ Max LRM │ 0.000kB │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ Max Temp Ext Bytes │ 80.000MB │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ Network GMACs │ 206.158 │
╘═════════════════════╧═══════════════════════════════════════════════════════════════════════════════════════════╛
╒════╤════════╤════════════════════════════════╤═══════════════════╤══════════════════════════╤═══════╕
│ │ Type │ Name │ shape │ type │ mse │
╞════╪════════╪════════════════════════════════╪═══════════════════╪══════════════════════════╪═══════╡
│ 0 │ Input │ /model/layers.1/Add_1_output_0 │ [1, 1024, 4096] │ tensor[FixedPoint32<25>] │ n/a │
├────┼────────┼────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 1 │ Input │ attention_mask │ [1, 1024, 1024] │ n/a │ n/a │
├────┼────────┼────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 2 │ Input │ cos │ [1, 1024, 128] │ tensor[FixedPoint32<30>] │ n/a │
├────┼────────┼────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 3 │ Input │ sin │ [1, 1024, 128] │ tensor[FixedPoint32<30>] │ n/a │
├────┼────────┼────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 4 │ Input │ past_key_values.2.key │ [1, 8, 0, 128] │ n/a │ n/a │
├────┼────────┼────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 5 │ Input │ past_key_values.2.value │ [1, 8, 0, 128] │ n/a │ n/a │
├────┼────────┼────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 6 │ Output │ /model/layers.2/Add_1_output_0 │ [1, 1024, 4096] │ tensor[FixedPoint32<24>] │ n/a │
├────┼────────┼────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 7 │ Output │ present.2.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├────┼────────┼────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 8 │ Output │ present.2.value │ [1, 8, 1024, 128] │ n/a │ n/a │
╘════╧════════╧════════════════════════════════╧═══════════════════╧══════════════════════════╧═══════╛
✓ CGC compile complete
── Step 4: Generate decoder_config.hpp + runner symlink ──
✓ Config + symlink ready → /quadric/sdk-cli/examples/models/qwen/qwen3_8b/prefill/decoder_2/qwen3_prefill_runner.cpp
── Step 5: SDK compile & ISS run ──
$ sdk source --quiet /quadric/sdk-cli/examples/models/qwen/qwen3_8b/prefill/decoder_2/qwen3_prefill_runner.cpp --include-cgc-headers --target QC-P --num-cores 4 --ocm-size 2MB --macs-per-pe 16 --clock-freq-ghz 1 --ext-read-bw 24GBps --ext-write-bw 24GBps --ddr-axi-width 256
✓ SDK compile complete → /quadric/sdk-cli/examples/models/qwen/qwen3_8b/prefill/decoder_2/epu_outputs/output_isolated.bin
── Step 6: Validate against ONNX Runtime ──
auto-atol: p_flip=0.187% auto=0.5106 base=1.0373 -> atol=1.0373
GPNPU vs ORT: max|d|=0.9108 mean|d|=0.0115 | atol=1.04 rtol=0.01 fail=0/4194304 (0.000%) [PASS]
✓ Validation complete
── Step 7: Plot output distributions ──
GPNPU min=-20.0026 max=62.1398 std=0.3093
ORT min=-20.0026 max=62.1398 std=0.3092
Scale factor (GPNPU std / ORT std): 1.00x
Pearson r (raw): 0.9986
Pearson r (scaled): 0.9986
Saved → decoder_2/output_comparison.png
✓ Plot saved → /quadric/sdk-cli/examples/models/qwen/qwen3_8b/prefill/decoder_2/output_comparison.png
── Step 8: Update decoder report ──
Warning: tranges lookup failed (No module named 'build_decoder_config'), using wrapper C++ frac bits
Updated /quadric/sdk-cli/examples/models/qwen/qwen3_8b/prefill/decoder_report.md
decoder 2: in_fb=25, out_fb=24, mean_diff=0.0115, max_diff=0.9108, combined%=0.000%, result=PASS, scale=1.000x, r=0.9986
✓ Report updated → decoder_report.md
── Step 9: Chain output to decoder 3 (mode: isolated) ──
✓ Chained ORT 2 → 3 (FixedPoint32<24>, 16.0 MB)
✓ Chained: /quadric/sdk-cli/examples/models/qwen/qwen3_8b/prefill/decoder_2/onnx_runtime_outputs/output_ort.npy → decoder_3/input_logits_ort.bin (FixedPoint32<24>)
mean_diff : 0.0115
max_diff : 0.9108
n_fail : 0 (0.000%)
pearson_r : 0.9986
result : PASS
── Profile Summary (from iss_run_log.txt) ──
Regions × Cores : 9 × 4
Total cycles : 102,405,943 (Σ regions, max across cores)
MAC cycles : 3,158,964 (3.1% of total)
The pipeline saves a distribution overlap, scatter and error histograms for the two outputs — a quick visual check that the GPNPU's fixed-point output sits on the ONNX Runtime reference across the full value range, not just on average:
display(Image(filename=str(PREFILL_DIR / f"decoder_{VALIDATE_DECODER}" / "output_comparison.png")))
3. Compile All 36 Decoders
With one layer validated, the same custom-op replacement runs over the complete prefill graph — all 36 decoders plus the LM head projection (include_lm_head=True) — and CGC compiles the result into the full prefill program. This is the artifact a deployment ships alongside the decode kernel: prompt in, first-token logits and a filled KV cache out. Like the decode compile, this step uses the export's tensor ranges.
The measured decoder from section 2 also gives a first read on time to first token. Decoders 1–34 share one compiled kernel, so 36 × one layer's cycles bounds the prefill latency from above — a fused 36-layer build amortizes the constant-tensor loads across layer boundaries and comes in at or below this sum. Attention makes prefill roughly quadratic in prompt length, so the numbers are specific to 1024 tokens.
Memory: ~10 GB peak · Runtime: under a minute for the custom ops, ~10 minutes for CGC
prefill_custom_onnx = str(PREFILL_DIR / "qwen3_custom_op_all_dec.onnx")
qwen3_prefill_custom_op_replacer(
prefill_onnx,
prefill_custom_onnx,
tranges_path,
num_heads=NUM_KV_HEADS,
embed_dim=EMBED_DIM,
seq_length=SEQ_LEN,
num_decoders=NUM_DECODERS,
include_lm_head=True,
)
Warning: /model/layers.0/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.1/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.2/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.3/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.4/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.5/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.6/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.7/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.8/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.9/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.10/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.11/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.12/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.13/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.14/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.15/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.16/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.17/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.18/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.19/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.20/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.21/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.22/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.23/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.24/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.25/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.26/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.27/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.28/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.29/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.30/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.31/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.32/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.33/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.34/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Warning: /model/layers.35/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Processing matmul ops for decoder 0...
Decoder 0 gate_up frac bits:
gate_combined_scale=5.489234e-04 frac_bits=41
up_combined_scale=3.307317e-04 frac_bits=42
gate_im_frac_bits=31
up_im_frac_bits=31
gate_output_frac_bits=28
up_output_frac_bits=28
Processing matmul ops for decoder 1...
Decoder 1 gate_up frac bits:
gate_combined_scale=7.408716e-03 frac_bits=38
up_combined_scale=3.896641e-03 frac_bits=39
gate_im_frac_bits=28
up_im_frac_bits=29
gate_output_frac_bits=27
up_output_frac_bits=26
Processing matmul ops for decoder 2...
Decoder 2 gate_up frac bits:
gate_combined_scale=5.992835e-02 frac_bits=35
up_combined_scale=5.992835e-02 frac_bits=35
gate_im_frac_bits=27
up_im_frac_bits=27
gate_output_frac_bits=24
up_output_frac_bits=24
Processing matmul ops for decoder 3...
Decoder 3 gate_up frac bits:
gate_combined_scale=1.153846e-02 frac_bits=37
up_combined_scale=9.374998e-03 frac_bits=37
gate_im_frac_bits=28
up_im_frac_bits=28
gate_output_frac_bits=26
up_output_frac_bits=26
Processing matmul ops for decoder 4...
Decoder 4 gate_up frac bits:
gate_combined_scale=1.012771e-02 frac_bits=37
up_combined_scale=1.079110e-02 frac_bits=37
gate_im_frac_bits=28
up_im_frac_bits=28
gate_output_frac_bits=27
up_output_frac_bits=27
Processing matmul ops for decoder 5...
Decoder 5 gate_up frac bits:
gate_combined_scale=1.041709e-02 frac_bits=37
up_combined_scale=5.855937e-03 frac_bits=38
gate_im_frac_bits=28
up_im_frac_bits=29
gate_output_frac_bits=27
up_output_frac_bits=27
Processing matmul ops for decoder 6...
Decoder 6 gate_up frac bits:
gate_combined_scale=6.929483e-02 frac_bits=34
up_combined_scale=7.354605e-02 frac_bits=34
gate_im_frac_bits=27
up_im_frac_bits=27
gate_output_frac_bits=24
up_output_frac_bits=24
Processing matmul ops for decoder 7...
Decoder 7 gate_up frac bits:
gate_combined_scale=7.519060e-03 frac_bits=38
up_combined_scale=1.755872e-03 frac_bits=40
gate_im_frac_bits=28
up_im_frac_bits=29
gate_output_frac_bits=27
up_output_frac_bits=27
Processing matmul ops for decoder 8...
Decoder 8 gate_up frac bits:
gate_combined_scale=2.171312e-03 frac_bits=39
up_combined_scale=1.607593e-03 frac_bits=40
gate_im_frac_bits=29
up_im_frac_bits=30
gate_output_frac_bits=27
up_output_frac_bits=28
Processing matmul ops for decoder 9...
Decoder 9 gate_up frac bits:
gate_combined_scale=2.659734e-03 frac_bits=39
up_combined_scale=2.695962e-03 frac_bits=39
gate_im_frac_bits=29
up_im_frac_bits=30
gate_output_frac_bits=27
up_output_frac_bits=28
Processing matmul ops for decoder 10...
Decoder 10 gate_up frac bits:
gate_combined_scale=2.187623e-03 frac_bits=39
up_combined_scale=1.581833e-03 frac_bits=40
gate_im_frac_bits=29
up_im_frac_bits=30
gate_output_frac_bits=27
up_output_frac_bits=27
Processing matmul ops for decoder 11...
Decoder 11 gate_up frac bits:
gate_combined_scale=1.958575e-03 frac_bits=39
up_combined_scale=1.510549e-03 frac_bits=40
gate_im_frac_bits=29
up_im_frac_bits=30
gate_output_frac_bits=27
up_output_frac_bits=27
Processing matmul ops for decoder 12...
Decoder 12 gate_up frac bits:
gate_combined_scale=1.847970e-03 frac_bits=40
up_combined_scale=2.534580e-03 frac_bits=39
gate_im_frac_bits=29
up_im_frac_bits=30
gate_output_frac_bits=27
up_output_frac_bits=27
Processing matmul ops for decoder 13...
Decoder 13 gate_up frac bits:
gate_combined_scale=1.677463e-03 frac_bits=40
up_combined_scale=3.220730e-03 frac_bits=39
gate_im_frac_bits=30
up_im_frac_bits=29
gate_output_frac_bits=26
up_output_frac_bits=25
Processing matmul ops for decoder 14...
Decoder 14 gate_up frac bits:
gate_combined_scale=2.226458e-03 frac_bits=39
up_combined_scale=2.970458e-03 frac_bits=39
gate_im_frac_bits=30
up_im_frac_bits=29
gate_output_frac_bits=26
up_output_frac_bits=25
Processing matmul ops for decoder 15...
Decoder 15 gate_up frac bits:
gate_combined_scale=2.298211e-03 frac_bits=39
up_combined_scale=6.424773e-03 frac_bits=38
gate_im_frac_bits=29
up_im_frac_bits=29
gate_output_frac_bits=26
up_output_frac_bits=25
Processing matmul ops for decoder 16...
Decoder 16 gate_up frac bits:
gate_combined_scale=8.712117e-03 frac_bits=37
up_combined_scale=9.366387e-03 frac_bits=37
gate_im_frac_bits=28
up_im_frac_bits=28
gate_output_frac_bits=23
up_output_frac_bits=24
Processing matmul ops for decoder 17...
Decoder 17 gate_up frac bits:
gate_combined_scale=2.006680e-03 frac_bits=39
up_combined_scale=1.808745e-03 frac_bits=40
gate_im_frac_bits=30
up_im_frac_bits=30
gate_output_frac_bits=27
up_output_frac_bits=25
Processing matmul ops for decoder 18...
Decoder 18 gate_up frac bits:
gate_combined_scale=2.155281e-03 frac_bits=39
up_combined_scale=1.964622e-03 frac_bits=39
gate_im_frac_bits=29
up_im_frac_bits=30
gate_output_frac_bits=27
up_output_frac_bits=26
Processing matmul ops for decoder 19...
Decoder 19 gate_up frac bits:
gate_combined_scale=1.855934e-03 frac_bits=40
up_combined_scale=2.282373e-03 frac_bits=39
gate_im_frac_bits=30
up_im_frac_bits=30
gate_output_frac_bits=27
up_output_frac_bits=28
Processing matmul ops for decoder 20...
Decoder 20 gate_up frac bits:
gate_combined_scale=1.904722e-03 frac_bits=40
up_combined_scale=1.601911e-03 frac_bits=40
gate_im_frac_bits=29
up_im_frac_bits=29
gate_output_frac_bits=27
up_output_frac_bits=27
Processing matmul ops for decoder 21...
Decoder 21 gate_up frac bits:
gate_combined_scale=2.035758e-03 frac_bits=39
up_combined_scale=2.340634e-03 frac_bits=39
gate_im_frac_bits=30
up_im_frac_bits=29
gate_output_frac_bits=27
up_output_frac_bits=27
Processing matmul ops for decoder 22...
Decoder 22 gate_up frac bits:
gate_combined_scale=2.405553e-03 frac_bits=39
up_combined_scale=2.320086e-03 frac_bits=39
gate_im_frac_bits=30
up_im_frac_bits=30
gate_output_frac_bits=26
up_output_frac_bits=27
Processing matmul ops for decoder 23...
Decoder 23 gate_up frac bits:
gate_combined_scale=2.210377e-03 frac_bits=39
up_combined_scale=2.788087e-03 frac_bits=39
gate_im_frac_bits=30
up_im_frac_bits=29
gate_output_frac_bits=26
up_output_frac_bits=27
Processing matmul ops for decoder 24...
Decoder 24 gate_up frac bits:
gate_combined_scale=3.017297e-03 frac_bits=39
up_combined_scale=3.114498e-03 frac_bits=39
gate_im_frac_bits=29
up_im_frac_bits=29
gate_output_frac_bits=26
up_output_frac_bits=27
Processing matmul ops for decoder 25...
Decoder 25 gate_up frac bits:
gate_combined_scale=3.542291e-03 frac_bits=39
up_combined_scale=3.030597e-03 frac_bits=39
gate_im_frac_bits=29
up_im_frac_bits=29
gate_output_frac_bits=26
up_output_frac_bits=26
Processing matmul ops for decoder 26...
Decoder 26 gate_up frac bits:
gate_combined_scale=2.672773e-03 frac_bits=39
up_combined_scale=4.103271e-03 frac_bits=38
gate_im_frac_bits=29
up_im_frac_bits=29
gate_output_frac_bits=26
up_output_frac_bits=27
Processing matmul ops for decoder 27...
Decoder 27 gate_up frac bits:
gate_combined_scale=3.190623e-03 frac_bits=39
up_combined_scale=4.202626e-03 frac_bits=38
gate_im_frac_bits=29
up_im_frac_bits=29
gate_output_frac_bits=26
up_output_frac_bits=27
Processing matmul ops for decoder 28...
Decoder 28 gate_up frac bits:
gate_combined_scale=4.199372e-03 frac_bits=38
up_combined_scale=4.026828e-03 frac_bits=38
gate_im_frac_bits=29
up_im_frac_bits=29
gate_output_frac_bits=26
up_output_frac_bits=26
Processing matmul ops for decoder 29...
Decoder 29 gate_up frac bits:
gate_combined_scale=3.686063e-03 frac_bits=39
up_combined_scale=4.187949e-03 frac_bits=38
gate_im_frac_bits=30
up_im_frac_bits=29
gate_output_frac_bits=26
up_output_frac_bits=26
Processing matmul ops for decoder 30...
Decoder 30 gate_up frac bits:
gate_combined_scale=4.159406e-03 frac_bits=38
up_combined_scale=7.111935e-03 frac_bits=38
gate_im_frac_bits=29
up_im_frac_bits=28
gate_output_frac_bits=26
up_output_frac_bits=26
Processing matmul ops for decoder 31...
Decoder 31 gate_up frac bits:
gate_combined_scale=4.884809e-03 frac_bits=38
up_combined_scale=8.509487e-03 frac_bits=37
gate_im_frac_bits=29
up_im_frac_bits=28
gate_output_frac_bits=26
up_output_frac_bits=26
Processing matmul ops for decoder 32...
Decoder 32 gate_up frac bits:
gate_combined_scale=5.781837e-03 frac_bits=38
up_combined_scale=7.929652e-03 frac_bits=37
gate_im_frac_bits=28
up_im_frac_bits=28
gate_output_frac_bits=26
up_output_frac_bits=26
Processing matmul ops for decoder 33...
Decoder 33 gate_up frac bits:
gate_combined_scale=9.199777e-03 frac_bits=37
up_combined_scale=2.238982e-02 frac_bits=36
gate_im_frac_bits=28
up_im_frac_bits=28
gate_output_frac_bits=26
up_output_frac_bits=26
Processing matmul ops for decoder 34...
Decoder 34 gate_up frac bits:
gate_combined_scale=4.481895e-02 frac_bits=35
up_combined_scale=6.740215e-02 frac_bits=34
gate_im_frac_bits=27
up_im_frac_bits=27
gate_output_frac_bits=25
up_output_frac_bits=24
Processing matmul ops for decoder 35...
Decoder 35 gate_up frac bits:
gate_combined_scale=1.067153e-01 frac_bits=34
up_combined_scale=1.776727e-01 frac_bits=33
gate_im_frac_bits=25
up_im_frac_bits=25
gate_output_frac_bits=24
up_output_frac_bits=23
Successfully saved modified model to /quadric/sdk-cli/examples/models/qwen/qwen3_8b/prefill/qwen3_custom_op_all_dec.onnx
prefill_job = ChimeraJob(
prefill_custom_onnx,
hw_config=HW_CONFIG,
trange_file=tranges_path,
target_lang="QIL",
io_to_ignore=kv_cache_io_names(NUM_DECODERS),
)
prefill_job.compile()
print(prefill_job)
2026-10-23 00:37 - INFO - epu - chimera_job - START==================================onnx_ingest
2026-10-23 00:38 - INFO - epu - chimera_job - Numerical ranges provided
2026-10-23 00:38 - INFO - epu - util - START===============================optimize_relay
2026-10-23 00:38 - INFO - epu - util - START====================quantize_to_cpu_runnable_fx
2026-10-23 00:38 - INFO - epu - fx -
Source name Op Output 0 Range Output 0 Frac Bits
--------------------------------------------------- ----------------------------- ---------------------- --------------------
/model/embed_tokens/Gather contrib.epu.embedding [-0.628906f, 0.8125f] 31
/model/layers.0/input_layernorm/Mul_1 contrib.epu.rms_norm [-0.359118f, 0.37561f] 31
CustomOp/linalg::channelwiseQuantMatMul<43>2 contrib.epu.quadric_custom_op [-2.25614f, 3.85482f] 28
CustomOp/add<MultiCoreMode::All>2 contrib.epu.quadric_custom_op [-2.30253f, 4.25466f] 28
/model/layers.0/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-1.23024f, 1.59211f] 29
CustomOp/nn::gateUpProjection<31,31,28,28,41,42>0 contrib.epu.quadric_custom_op [-11.3828f, 9.95375f] 27
CustomOp/linalg::channelwiseQuantMatMul<38>1 contrib.epu.quadric_custom_op [-5.18762f, 16.7897f] 26
CustomOp/add<MultiCoreMode::All>1 contrib.epu.quadric_custom_op [-6.67269f, 19.2419f] 26
/model/layers.1/input_layernorm/Mul_1 contrib.epu.rms_norm [-1.30106f, 1.14032f] 30
CustomOp/linalg::channelwiseQuantMatMul<43>5 contrib.epu.quadric_custom_op [-1.41003f, 1.53298f] 29
CustomOp/add<MultiCoreMode::All>5 contrib.epu.quadric_custom_op [-6.78994f, 19.2634f] 26
/model/layers.1/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-51.933f, 47.6116f] 25
CustomOp/nn::gateUpProjection<28,29,27,26,38,39>3 contrib.epu.quadric_custom_op [-22.4947f, 19.6069f] 26
CustomOp/linalg::channelwiseQuantMatMul<37>4 contrib.epu.quadric_custom_op [-12.4679f, 38.1379f] 25
CustomOp/add<MultiCoreMode::All>4 contrib.epu.quadric_custom_op [-18.1108f, 57.4013f] 25
/model/layers.2/input_layernorm/Mul_1 contrib.epu.rms_norm [-0.862898f, 1.28555f] 29
CustomOp/linalg::channelwiseQuantMatMul<42>8 contrib.epu.quadric_custom_op [-1.03252f, 1.31911f] 29
CustomOp/add<MultiCoreMode::All>8 contrib.epu.quadric_custom_op [-18.1085f, 57.4384f] 25
/model/layers.2/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-82.5232f, 81.3135f] 24
CustomOp/nn::gateUpProjection<27,27,24,24,35,35>6 contrib.epu.quadric_custom_op [-26.306f, 24.5606f] 26
CustomOp/linalg::channelwiseQuantMatMul<37>7 contrib.epu.quadric_custom_op [-10.8904f, 26.5761f] 26
CustomOp/add<MultiCoreMode::All>7 contrib.epu.quadric_custom_op [-23.2706f, 81.9635f] 24
/model/layers.3/input_layernorm/Mul_1 contrib.epu.rms_norm [-1.24747f, 3.00854f] 28
CustomOp/linalg::channelwiseQuantMatMul<42>11 contrib.epu.quadric_custom_op [-1.84038f, 1.97427f] 29
CustomOp/add<MultiCoreMode::All>11 contrib.epu.quadric_custom_op [-23.1594f, 81.7177f] 24
/model/layers.3/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-53.3995f, 62.6125f] 25
CustomOp/nn::gateUpProjection<28,28,26,26,37,37>9 contrib.epu.quadric_custom_op [-23.0629f, 23.7062f] 26
CustomOp/linalg::channelwiseQuantMatMul<39>10 contrib.epu.quadric_custom_op [-7.28584f, 7.4731f] 27
CustomOp/add<MultiCoreMode::All>10 contrib.epu.quadric_custom_op [-23.3718f, 85.3067f] 24
/model/layers.4/input_layernorm/Mul_1 contrib.epu.rms_norm [-1.80589f, 2.00268f] 29
CustomOp/linalg::channelwiseQuantMatMul<42>14 contrib.epu.quadric_custom_op [-3.53964f, 2.62418f] 29
CustomOp/add<MultiCoreMode::All>14 contrib.epu.quadric_custom_op [-23.2038f, 85.5677f] 24
/model/layers.4/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-32.953f, 41.0647f] 25
CustomOp/nn::gateUpProjection<28,28,27,27,37,37>12 contrib.epu.quadric_custom_op [-20.2387f, 20.039f] 26
CustomOp/linalg::channelwiseQuantMatMul<39>13 contrib.epu.quadric_custom_op [-6.69227f, 12.1684f] 27
CustomOp/add<MultiCoreMode::All>13 contrib.epu.quadric_custom_op [-22.4392f, 87.0669f] 24
/model/layers.5/input_layernorm/Mul_1 contrib.epu.rms_norm [-2.14681f, 3.19879f] 28
CustomOp/linalg::channelwiseQuantMatMul<42>17 contrib.epu.quadric_custom_op [-6.97809f, 4.76181f] 28
CustomOp/add<MultiCoreMode::All>17 contrib.epu.quadric_custom_op [-21.9713f, 84.54f] 24
/model/layers.5/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-24.6416f, 19.9332f] 26
CustomOp/nn::gateUpProjection<28,29,27,27,37,38>15 contrib.epu.quadric_custom_op [-18.2442f, 18.1234f] 26
CustomOp/linalg::channelwiseQuantMatMul<38>16 contrib.epu.quadric_custom_op [-37.2974f, 18.5171f] 25
CustomOp/add<MultiCoreMode::All>16 contrib.epu.quadric_custom_op [-18.6574f, 63.1892f] 24
/model/layers.6/input_layernorm/Mul_1 contrib.epu.rms_norm [-1.99346f, 7.23117f] 27
CustomOp/linalg::channelwiseQuantMatMul<40>20 contrib.epu.quadric_custom_op [-31.7815f, 8.99221f] 26
CustomOp/add<MultiCoreMode::All>20 contrib.epu.quadric_custom_op [-12.6824f, 38.3116f] 25
/model/layers.6/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-9.84406f, 199.177f] 23
CustomOp/nn::gateUpProjection<27,27,24,24,34,34>18 contrib.epu.quadric_custom_op [-1282.21f, 4841.01f] 18
CustomOp/linalg::channelwiseQuantMatMul<27>19 contrib.epu.quadric_custom_op [-2118.69f, 9638.07f] 17
CustomOp/add<MultiCoreMode::All>19 contrib.epu.quadric_custom_op [-2121.44f, 9654.1f] 17
/model/layers.7/input_layernorm/Mul_1 contrib.epu.rms_norm [-5.1855f, 10.441f] 27
CustomOp/linalg::channelwiseQuantMatMul<41>23 contrib.epu.quadric_custom_op [-4.27446f, 3.6406f] 28
CustomOp/add<MultiCoreMode::All>23 contrib.epu.quadric_custom_op [-2121.23f, 9654.21f] 17
/model/layers.7/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-6.89828f, 54.4274f] 25
CustomOp/nn::gateUpProjection<28,29,27,27,38,40>21 contrib.epu.quadric_custom_op [-16.9021f, 17.2346f] 26
CustomOp/linalg::channelwiseQuantMatMul<37>22 contrib.epu.quadric_custom_op [-6.76926f, 12.1386f] 27
CustomOp/add<MultiCoreMode::All>22 contrib.epu.quadric_custom_op [-2121.18f, 9654.34f] 17
/model/layers.8/input_layernorm/Mul_1 contrib.epu.rms_norm [-4.59068f, 9.9222f] 27
CustomOp/linalg::channelwiseQuantMatMul<41>26 contrib.epu.quadric_custom_op [-3.77026f, 7.3769f] 27
CustomOp/add<MultiCoreMode::All>26 contrib.epu.quadric_custom_op [-2120.87f, 9653.89f] 17
/model/layers.8/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-7.8127f, 6.98587f] 28
CustomOp/nn::gateUpProjection<29,30,27,28,39,40>24 contrib.epu.quadric_custom_op [-15.8356f, 18.0968f] 26
CustomOp/linalg::channelwiseQuantMatMul<38>25 contrib.epu.quadric_custom_op [-10.3458f, 12.5126f] 27
CustomOp/add<MultiCoreMode::All>25 contrib.epu.quadric_custom_op [-2120.62f, 9654.73f] 17
/model/layers.9/input_layernorm/Mul_1 contrib.epu.rms_norm [-4.52299f, 10.3721f] 27
CustomOp/linalg::channelwiseQuantMatMul<39>29 contrib.epu.quadric_custom_op [-7.31632f, 7.46158f] 27
CustomOp/add<MultiCoreMode::All>29 contrib.epu.quadric_custom_op [-2120.34f, 9654.1f] 17
/model/layers.9/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-8.45564f, 8.75462f] 27
CustomOp/nn::gateUpProjection<29,30,27,28,39,39>27 contrib.epu.quadric_custom_op [-15.1261f, 14.7165f] 27
CustomOp/linalg::channelwiseQuantMatMul<38>28 contrib.epu.quadric_custom_op [-9.86182f, 8.2895f] 27
CustomOp/add<MultiCoreMode::All>28 contrib.epu.quadric_custom_op [-2120.34f, 9656.53f] 17
/model/layers.10/input_layernorm/Mul_1 contrib.epu.rms_norm [-7.60531f, 17.5566f] 26
CustomOp/linalg::channelwiseQuantMatMul<40>32 contrib.epu.quadric_custom_op [-7.399f, 5.84284f] 28
CustomOp/add<MultiCoreMode::All>32 contrib.epu.quadric_custom_op [-2119.36f, 9655.58f] 17
/model/layers.10/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-8.33987f, 9.41487f] 27
CustomOp/nn::gateUpProjection<29,30,27,27,39,40>30 contrib.epu.quadric_custom_op [-15.7846f, 20.8058f] 26
CustomOp/linalg::channelwiseQuantMatMul<37>31 contrib.epu.quadric_custom_op [-5.03255f, 18.6078f] 26
CustomOp/add<MultiCoreMode::All>31 contrib.epu.quadric_custom_op [-2119.79f, 9656.97f] 17
/model/layers.11/input_layernorm/Mul_1 contrib.epu.rms_norm [-5.77417f, 10.4802f] 27
CustomOp/linalg::channelwiseQuantMatMul<40>35 contrib.epu.quadric_custom_op [-7.13255f, 7.60828f] 27
CustomOp/add<MultiCoreMode::All>35 contrib.epu.quadric_custom_op [-2118.77f, 9655.72f] 17
/model/layers.11/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-9.30534f, 7.62255f] 27
CustomOp/nn::gateUpProjection<29,30,27,27,39,40>33 contrib.epu.quadric_custom_op [-12.4377f, 12.5137f] 27
CustomOp/linalg::channelwiseQuantMatMul<38>34 contrib.epu.quadric_custom_op [-7.59268f, 11.5153f] 27
CustomOp/add<MultiCoreMode::All>34 contrib.epu.quadric_custom_op [-2118.11f, 9657.75f] 17
/model/layers.12/input_layernorm/Mul_1 contrib.epu.rms_norm [-6.76928f, 11.1257f] 27
CustomOp/linalg::channelwiseQuantMatMul<40>38 contrib.epu.quadric_custom_op [-7.8656f, 13.7375f] 27
CustomOp/add<MultiCoreMode::All>38 contrib.epu.quadric_custom_op [-2117.66f, 9657.01f] 17
/model/layers.12/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-10.1594f, 7.15519f] 27
CustomOp/nn::gateUpProjection<29,30,27,27,40,39>36 contrib.epu.quadric_custom_op [-60.1034f, 12.2375f] 25
CustomOp/linalg::channelwiseQuantMatMul<37>37 contrib.epu.quadric_custom_op [-12.1023f, 17.851f] 26
CustomOp/add<MultiCoreMode::All>37 contrib.epu.quadric_custom_op [-2118.72f, 9659.33f] 17
/model/layers.13/input_layernorm/Mul_1 contrib.epu.rms_norm [-5.04587f, 9.31552f] 27
CustomOp/linalg::channelwiseQuantMatMul<40>41 contrib.epu.quadric_custom_op [-5.54142f, 8.70166f] 27
CustomOp/add<MultiCoreMode::All>41 contrib.epu.quadric_custom_op [-2117.86f, 9659.14f] 17
/model/layers.13/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-11.9317f, 6.23108f] 27
CustomOp/nn::gateUpProjection<30,29,26,25,40,39>39 contrib.epu.quadric_custom_op [-16.9783f, 33.7621f] 25
CustomOp/linalg::channelwiseQuantMatMul<38>40 contrib.epu.quadric_custom_op [-11.2142f, 11.9456f] 27
CustomOp/add<MultiCoreMode::All>40 contrib.epu.quadric_custom_op [-2120.76f, 9664.39f] 17
/model/layers.14/input_layernorm/Mul_1 contrib.epu.rms_norm [-6.68111f, 13.0119f] 27
CustomOp/linalg::channelwiseQuantMatMul<40>44 contrib.epu.quadric_custom_op [-8.29877f, 19.2297f] 26
CustomOp/add<MultiCoreMode::All>44 contrib.epu.quadric_custom_op [-2120.07f, 9663.34f] 17
/model/layers.14/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-14.412f, 6.25216f] 27
CustomOp/nn::gateUpProjection<30,29,26,25,39,39>42 contrib.epu.quadric_custom_op [-84.1848f, 13.7615f] 24
CustomOp/linalg::channelwiseQuantMatMul<36>43 contrib.epu.quadric_custom_op [-18.0127f, 15.0041f] 26
CustomOp/add<MultiCoreMode::All>43 contrib.epu.quadric_custom_op [-2123.29f, 9666.16f] 17
/model/layers.15/input_layernorm/Mul_1 contrib.epu.rms_norm [-7.40155f, 13.516f] 27
CustomOp/linalg::channelwiseQuantMatMul<39>47 contrib.epu.quadric_custom_op [-6.83366f, 13.2874f] 27
CustomOp/add<MultiCoreMode::All>47 contrib.epu.quadric_custom_op [-2122.7f, 9664.69f] 17
/model/layers.15/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-16.0488f, 6.0668f] 26
CustomOp/nn::gateUpProjection<29,29,26,25,39,38>45 contrib.epu.quadric_custom_op [-27.8356f, 48.4799f] 25
CustomOp/linalg::channelwiseQuantMatMul<36>46 contrib.epu.quadric_custom_op [-16.8693f, 20.4889f] 26
CustomOp/add<MultiCoreMode::All>46 contrib.epu.quadric_custom_op [-2123.21f, 9668.25f] 17
/model/layers.16/input_layernorm/Mul_1 contrib.epu.rms_norm [-8.15026f, 15.6546f] 26
CustomOp/linalg::channelwiseQuantMatMul<38>50 contrib.epu.quadric_custom_op [-39.5976f, 22.6377f] 25
CustomOp/add<MultiCoreMode::All>50 contrib.epu.quadric_custom_op [-2122.92f, 9666.15f] 17
/model/layers.16/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-36.8359f, 6.53895f] 25
CustomOp/nn::gateUpProjection<28,28,23,24,37,37>48 contrib.epu.quadric_custom_op [-191.33f, 4661.68f] 18
CustomOp/linalg::channelwiseQuantMatMul<27>49 contrib.epu.quadric_custom_op [-1202.39f, 13581.2f] 17
CustomOp/add<MultiCoreMode::All>49 contrib.epu.quadric_custom_op [-2130f, 13602.6f] 17
/model/layers.17/input_layernorm/Mul_1 contrib.epu.rms_norm [-7.88683f, 13.0965f] 27
CustomOp/linalg::channelwiseQuantMatMul<37>53 contrib.epu.quadric_custom_op [-14.5229f, 61.5807f] 25
CustomOp/add<MultiCoreMode::All>53 contrib.epu.quadric_custom_op [-2129.24f, 13637f] 17
/model/layers.17/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-12.2827f, 6.15562f] 27
CustomOp/nn::gateUpProjection<30,30,27,25,39,40>51 contrib.epu.quadric_custom_op [-13.8208f, 9.58101f] 27
CustomOp/linalg::channelwiseQuantMatMul<38>52 contrib.epu.quadric_custom_op [-21.4484f, 22.413f] 26
CustomOp/add<MultiCoreMode::All>52 contrib.epu.quadric_custom_op [-2128.21f, 13636.9f] 17
/model/layers.18/input_layernorm/Mul_1 contrib.epu.rms_norm [-10.6242f, 15.4063f] 26
CustomOp/linalg::channelwiseQuantMatMul<40>56 contrib.epu.quadric_custom_op [-11.2483f, 15.3732f] 26
CustomOp/add<MultiCoreMode::All>56 contrib.epu.quadric_custom_op [-2127.25f, 13638.1f] 17
/model/layers.18/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-11.7031f, 6.12549f] 27
CustomOp/nn::gateUpProjection<29,30,27,26,39,39>54 contrib.epu.quadric_custom_op [-9.16472f, 64.1296f] 24
CustomOp/linalg::channelwiseQuantMatMul<35>55 contrib.epu.quadric_custom_op [-27.1561f, 62.6349f] 25
CustomOp/add<MultiCoreMode::All>55 contrib.epu.quadric_custom_op [-2127.74f, 13638.1f] 17
/model/layers.19/input_layernorm/Mul_1 contrib.epu.rms_norm [-16.3269f, 23.5058f] 26
CustomOp/linalg::channelwiseQuantMatMul<39>59 contrib.epu.quadric_custom_op [-9.69087f, 19.3656f] 26
CustomOp/add<MultiCoreMode::All>59 contrib.epu.quadric_custom_op [-2127.19f, 13637.7f] 17
/model/layers.19/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-11.4451f, 6.57414f] 27
CustomOp/nn::gateUpProjection<30,30,27,28,40,39>57 contrib.epu.quadric_custom_op [-10.5208f, 11.7491f] 27
CustomOp/linalg::channelwiseQuantMatMul<36>58 contrib.epu.quadric_custom_op [-28.7991f, 32.1844f] 25
CustomOp/add<MultiCoreMode::All>58 contrib.epu.quadric_custom_op [-2127.2f, 13637.6f] 17
/model/layers.20/input_layernorm/Mul_1 contrib.epu.rms_norm [-17.2382f, 20.6114f] 26
CustomOp/linalg::channelwiseQuantMatMul<40>62 contrib.epu.quadric_custom_op [-11.7166f, 31.5278f] 25
CustomOp/add<MultiCoreMode::All>62 contrib.epu.quadric_custom_op [-2126.62f, 13634.6f] 17
/model/layers.20/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-12.2707f, 7.12023f] 27
CustomOp/nn::gateUpProjection<29,29,27,27,40,40>60 contrib.epu.quadric_custom_op [-18.5987f, 19.8616f] 26
CustomOp/linalg::channelwiseQuantMatMul<38>61 contrib.epu.quadric_custom_op [-16.8789f, 16.8205f] 26
CustomOp/add<MultiCoreMode::All>61 contrib.epu.quadric_custom_op [-2125.87f, 13634.7f] 17
/model/layers.21/input_layernorm/Mul_1 contrib.epu.rms_norm [-23.1988f, 24.3195f] 26
CustomOp/linalg::channelwiseQuantMatMul<39>65 contrib.epu.quadric_custom_op [-8.88642f, 21.276f] 26
CustomOp/add<MultiCoreMode::All>65 contrib.epu.quadric_custom_op [-2124.82f, 13636.5f] 17
/model/layers.21/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-10.9728f, 7.28116f] 27
CustomOp/nn::gateUpProjection<30,29,27,27,39,39>63 contrib.epu.quadric_custom_op [-41.594f, 53.735f] 25
CustomOp/linalg::channelwiseQuantMatMul<37>64 contrib.epu.quadric_custom_op [-32.024f, 16.017f] 25
CustomOp/add<MultiCoreMode::All>64 contrib.epu.quadric_custom_op [-2124.91f, 13636.6f] 17
/model/layers.22/input_layernorm/Mul_1 contrib.epu.rms_norm [-30.8369f, 32.7385f] 25
CustomOp/linalg::channelwiseQuantMatMul<39>68 contrib.epu.quadric_custom_op [-12.5842f, 31.0796f] 25
CustomOp/add<MultiCoreMode::All>68 contrib.epu.quadric_custom_op [-2124.1f, 13635.5f] 17
/model/layers.22/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-10.0459f, 8.97793f] 27
CustomOp/nn::gateUpProjection<30,30,26,27,39,39>66 contrib.epu.quadric_custom_op [-42.2895f, 37.618f] 25
CustomOp/linalg::channelwiseQuantMatMul<37>67 contrib.epu.quadric_custom_op [-24.8563f, 17.5565f] 26
CustomOp/add<MultiCoreMode::All>67 contrib.epu.quadric_custom_op [-2123.61f, 13635.4f] 17
/model/layers.23/input_layernorm/Mul_1 contrib.epu.rms_norm [-32.5652f, 31.0209f] 25
CustomOp/linalg::channelwiseQuantMatMul<39>71 contrib.epu.quadric_custom_op [-8.13639f, 35.0445f] 25
CustomOp/add<MultiCoreMode::All>71 contrib.epu.quadric_custom_op [-2123.06f, 13632.6f] 17
/model/layers.23/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-10.1758f, 10.4433f] 27
CustomOp/nn::gateUpProjection<30,29,26,27,39,39>69 contrib.epu.quadric_custom_op [-64.6526f, 60.3269f] 24
CustomOp/linalg::channelwiseQuantMatMul<37>70 contrib.epu.quadric_custom_op [-24.6908f, 31.6436f] 25
CustomOp/add<MultiCoreMode::All>70 contrib.epu.quadric_custom_op [-2122.79f, 13632.6f] 17
/model/layers.24/input_layernorm/Mul_1 contrib.epu.rms_norm [-50.8646f, 39.9522f] 25
CustomOp/linalg::channelwiseQuantMatMul<38>74 contrib.epu.quadric_custom_op [-16.7965f, 31.3655f] 25
CustomOp/add<MultiCoreMode::All>74 contrib.epu.quadric_custom_op [-2123.99f, 13635.1f] 17
/model/layers.24/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-10.2601f, 12.5924f] 27
CustomOp/nn::gateUpProjection<29,29,26,27,39,39>72 contrib.epu.quadric_custom_op [-107.554f, 90.9104f] 24
CustomOp/linalg::channelwiseQuantMatMul<36>73 contrib.epu.quadric_custom_op [-22.1531f, 38.053f] 25
CustomOp/add<MultiCoreMode::All>73 contrib.epu.quadric_custom_op [-2123.99f, 13635.3f] 17
/model/layers.25/input_layernorm/Mul_1 contrib.epu.rms_norm [-40.2525f, 33.3074f] 25
CustomOp/linalg::channelwiseQuantMatMul<39>77 contrib.epu.quadric_custom_op [-10.4602f, 17.5029f] 26
CustomOp/add<MultiCoreMode::All>77 contrib.epu.quadric_custom_op [-2123.75f, 13635.5f] 17
/model/layers.25/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-11.309f, 12.7584f] 27
CustomOp/nn::gateUpProjection<29,29,26,26,39,39>75 contrib.epu.quadric_custom_op [-117.555f, 109.813f] 24
CustomOp/linalg::channelwiseQuantMatMul<36>76 contrib.epu.quadric_custom_op [-19.0471f, 36.3233f] 25
CustomOp/add<MultiCoreMode::All>76 contrib.epu.quadric_custom_op [-2123.73f, 13635.5f] 17
/model/layers.26/input_layernorm/Mul_1 contrib.epu.rms_norm [-50.6021f, 38.8768f] 25
CustomOp/linalg::channelwiseQuantMatMul<40>80 contrib.epu.quadric_custom_op [-8.51294f, 19.8805f] 26
CustomOp/add<MultiCoreMode::All>80 contrib.epu.quadric_custom_op [-2124.04f, 13636.4f] 17
/model/layers.26/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-12.5561f, 15.4058f] 26
CustomOp/nn::gateUpProjection<29,29,26,27,39,38>78 contrib.epu.quadric_custom_op [-105.01f, 132.743f] 23
CustomOp/linalg::channelwiseQuantMatMul<36>79 contrib.epu.quadric_custom_op [-21.3917f, 49.6387f] 25
CustomOp/add<MultiCoreMode::All>79 contrib.epu.quadric_custom_op [-2124.03f, 13636.6f] 17
/model/layers.27/input_layernorm/Mul_1 contrib.epu.rms_norm [-61.2838f, 48.1027f] 25
CustomOp/linalg::channelwiseQuantMatMul<39>83 contrib.epu.quadric_custom_op [-8.8824f, 19.0919f] 26
CustomOp/add<MultiCoreMode::All>83 contrib.epu.quadric_custom_op [-2124.55f, 13638.8f] 17
/model/layers.27/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-14.5542f, 17.2044f] 26
CustomOp/nn::gateUpProjection<29,29,26,27,39,38>81 contrib.epu.quadric_custom_op [-137.571f, 122.284f] 23
CustomOp/linalg::channelwiseQuantMatMul<36>82 contrib.epu.quadric_custom_op [-29.7459f, 42.6272f] 25
CustomOp/add<MultiCoreMode::All>82 contrib.epu.quadric_custom_op [-2124.49f, 13639.2f] 17
/model/layers.28/input_layernorm/Mul_1 contrib.epu.rms_norm [-65.7556f, 56.4749f] 24
CustomOp/linalg::channelwiseQuantMatMul<39>86 contrib.epu.quadric_custom_op [-16.1447f, 22.5356f] 26
CustomOp/add<MultiCoreMode::All>86 contrib.epu.quadric_custom_op [-2125.41f, 13642.1f] 17
/model/layers.28/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-13.3559f, 17.9234f] 26
CustomOp/nn::gateUpProjection<29,29,26,26,38,38>84 contrib.epu.quadric_custom_op [-132.766f, 140.229f] 23
CustomOp/linalg::channelwiseQuantMatMul<35>85 contrib.epu.quadric_custom_op [-38.7672f, 58.2323f] 25
CustomOp/add<MultiCoreMode::All>85 contrib.epu.quadric_custom_op [-2125.36f, 13642.5f] 17
/model/layers.29/input_layernorm/Mul_1 contrib.epu.rms_norm [-87.7913f, 63.9043f] 24
CustomOp/linalg::channelwiseQuantMatMul<39>89 contrib.epu.quadric_custom_op [-15.1006f, 23.0286f] 26
CustomOp/add<MultiCoreMode::All>89 contrib.epu.quadric_custom_op [-2126.23f, 13643.9f] 17
/model/layers.29/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-14.5663f, 19.9309f] 26
CustomOp/nn::gateUpProjection<30,29,26,26,39,38>87 contrib.epu.quadric_custom_op [-150.54f, 123.012f] 23
CustomOp/linalg::channelwiseQuantMatMul<35>88 contrib.epu.quadric_custom_op [-49.9533f, 76.5029f] 24
CustomOp/add<MultiCoreMode::All>88 contrib.epu.quadric_custom_op [-2126.19f, 13644f] 17
/model/layers.30/input_layernorm/Mul_1 contrib.epu.rms_norm [-97.8058f, 70.9147f] 24
CustomOp/linalg::channelwiseQuantMatMul<38>92 contrib.epu.quadric_custom_op [-26.3882f, 42.82f] 25
CustomOp/add<MultiCoreMode::All>92 contrib.epu.quadric_custom_op [-2126.46f, 13648.6f] 17
/model/layers.30/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-16.1268f, 19.8625f] 26
CustomOp/nn::gateUpProjection<29,28,26,26,38,38>90 contrib.epu.quadric_custom_op [-147.928f, 141.815f] 23
CustomOp/linalg::channelwiseQuantMatMul<35>91 contrib.epu.quadric_custom_op [-53.484f, 106.991f] 24
CustomOp/add<MultiCoreMode::All>91 contrib.epu.quadric_custom_op [-2126.44f, 13648.7f] 17
/model/layers.31/input_layernorm/Mul_1 contrib.epu.rms_norm [-119.533f, 93.6401f] 24
CustomOp/linalg::channelwiseQuantMatMul<38>95 contrib.epu.quadric_custom_op [-17.2974f, 50.1237f] 25
CustomOp/add<MultiCoreMode::All>95 contrib.epu.quadric_custom_op [-2126.83f, 13653.5f] 17
/model/layers.31/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-17.2516f, 18.0006f] 26
CustomOp/nn::gateUpProjection<29,28,26,26,38,37>93 contrib.epu.quadric_custom_op [-147.492f, 171.108f] 23
CustomOp/linalg::channelwiseQuantMatMul<34>94 contrib.epu.quadric_custom_op [-42.7217f, 101.298f] 24
CustomOp/add<MultiCoreMode::All>94 contrib.epu.quadric_custom_op [-2126.65f, 13653.4f] 17
/model/layers.32/input_layernorm/Mul_1 contrib.epu.rms_norm [-129.345f, 108.642f] 23
CustomOp/linalg::channelwiseQuantMatMul<37>98 contrib.epu.quadric_custom_op [-26.9792f, 52.8894f] 25
CustomOp/add<MultiCoreMode::All>98 contrib.epu.quadric_custom_op [-2125.55f, 13676.5f] 17
/model/layers.32/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-17.899f, 23.7326f] 26
CustomOp/nn::gateUpProjection<28,28,26,26,38,37>96 contrib.epu.quadric_custom_op [-190.096f, 285.781f] 22
CustomOp/linalg::channelwiseQuantMatMul<34>97 contrib.epu.quadric_custom_op [-67.0631f, 111.421f] 24
CustomOp/add<MultiCoreMode::All>97 contrib.epu.quadric_custom_op [-2125.27f, 13672.9f] 17
/model/layers.33/input_layernorm/Mul_1 contrib.epu.rms_norm [-187.593f, 150.813f] 23
CustomOp/linalg::channelwiseQuantMatMul<37>101 contrib.epu.quadric_custom_op [-22.3657f, 115.159f] 24
CustomOp/add<MultiCoreMode::All>101 contrib.epu.quadric_custom_op [-2125.11f, 13717f] 17
/model/layers.33/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-29.27f, 27.8766f] 26
CustomOp/nn::gateUpProjection<28,28,26,26,37,36>99 contrib.epu.quadric_custom_op [-191.84f, 180.474f] 23
CustomOp/linalg::channelwiseQuantMatMul<34>100 contrib.epu.quadric_custom_op [-224.43f, 154.382f] 23
CustomOp/add<MultiCoreMode::All>100 contrib.epu.quadric_custom_op [-2126.49f, 13736f] 17
/model/layers.34/input_layernorm/Mul_1 contrib.epu.rms_norm [-190.924f, 150.511f] 23
CustomOp/linalg::channelwiseQuantMatMul<36>104 contrib.epu.quadric_custom_op [-134.891f, 252.592f] 23
CustomOp/add<MultiCoreMode::All>104 contrib.epu.quadric_custom_op [-2127.57f, 13665.9f] 17
/model/layers.34/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-102.457f, 31.8715f] 24
CustomOp/nn::gateUpProjection<27,27,25,24,35,34>102 contrib.epu.quadric_custom_op [-1174.96f, 792.102f] 20
CustomOp/linalg::channelwiseQuantMatMul<31>103 contrib.epu.quadric_custom_op [-18980.2f, 553.5f] 16
CustomOp/add<MultiCoreMode::All>103 contrib.epu.quadric_custom_op [-7343.94f, 5351.69f] 18
/model/layers.35/input_layernorm/Mul_1 contrib.epu.rms_norm [-298.728f, 121.485f] 22
CustomOp/linalg::channelwiseQuantMatMul<36>107 contrib.epu.quadric_custom_op [-282.141f, 824.661f] 21
CustomOp/add<MultiCoreMode::All>107 contrib.epu.quadric_custom_op [-7101.39f, 5072.41f] 18
/model/layers.35/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-601.076f, 47.2325f] 21
CustomOp/nn::gateUpProjection<25,25,24,23,34,33>105 contrib.epu.quadric_custom_op [-1002.33f, 1755.35f] 20
CustomOp/linalg::channelwiseQuantMatMul<29>106 contrib.epu.quadric_custom_op [-4812.36f, 1990.75f] 18
CustomOp/add<MultiCoreMode::All>106 contrib.epu.quadric_custom_op [-6990.34f, 2581.91f] 18
/model/norm/Mul_1 contrib.epu.rms_norm [-139.062f, 130.451f] 23
CustomOp/nn::gather<Direction::Height>108 contrib.epu.quadric_custom_op [-93.9597f, 66.4554f] 24
CustomOp/linalg::channelwiseQuantMatMul<36>108 contrib.epu.quadric_custom_op [-15.0257f, 24.2741f] 26
2026-10-23 00:38 - INFO - epu - util - START====================build_cpu_runnable_fx_relay
2026-10-23 00:38 - INFO - epu - util - START=======================quantize_to_chimera_fx
2026-10-23 00:38 - INFO - epu - util - START=================================relay_to_tir
2026-10-23 00:38 - INFO - epu - util - START===========================relay_to_epu_relay
2026-10-23 00:38 - INFO - epu - util - START==============================adapt_and_order
2026-10-23 00:39 - INFO - epu - util - START==============================amend_ctrl_flow
2026-10-23 00:39 - INFO - epu - util - START=============================plan_lrm_virtual
2026-10-23 00:39 - INFO - epu - util - START==============================amend_ctrl_flow
2026-10-23 00:39 - INFO - epu - util - START===============================lrm_alloc_loop
2026-10-23 00:39 - INFO - epu - util - START==============================amend_ctrl_flow
2026-10-23 00:39 - INFO - epu - util - START================================lrm_splitting
2026-10-23 00:39 - INFO - epu - util - START=====================LrmUsageChecker.__call__
2026-10-23 00:40 - INFO - epu - util - START==============================ext_split_relay
2026-10-23 00:40 - INFO - epu - util - START=========================OcmAllocator.analyze
2026-10-23 00:40 - INFO - epu - util - START===============================Compiler.build
2026-10-23 00:40 - INFO - epu - util - START========================RelayToPrimFunc.build
2026-10-23 00:40 - INFO - epu - util - START=========================OcmAllocator.analyze
2026-10-23 00:40 - INFO - epu - util - START====================================build_tir
2026-10-23 00:40 - INFO - epu - chimera_job - Compilation of qwen3_custom_op_all_dec_QC_P_1d7_2MB_4kB_128GBps_128GBps_16_OFF_x1_x1 successful
╒═════════════════════╤═══════════════════════════════════════════════════════════════════════╕
│ Module Name │ qwen3_custom_op_all_dec_QC_P_1d7_2MB_4kB_128GBps_128GBps_16_OFF_x1_x1 │
├─────────────────────┼───────────────────────────────────────────────────────────────────────┤
│ ONNX File │ prefill/qwen3_custom_op_all_dec.onnx │
├─────────────────────┼───────────────────────────────────────────────────────────────────────┤
│ Product Target │ QC-P │
├─────────────────────┼───────────────────────────────────────────────────────────────────────┤
│ Number of Cores │ 1 │
├─────────────────────┼───────────────────────────────────────────────────────────────────────┤
│ ISS Clock Frequency │ 1.700 │
├─────────────────────┼───────────────────────────────────────────────────────────────────────┤
│ L2M Size │ 2MB │
├─────────────────────┼───────────────────────────────────────────────────────────────────────┤
│ LRM Size │ 4kB │
├─────────────────────┼───────────────────────────────────────────────────────────────────────┤
│ External Read BW │ 128GBps │
├─────────────────────┼───────────────────────────────────────────────────────────────────────┤
│ External Write BW │ 128GBps │
├─────────────────────┼───────────────────────────────────────────────────────────────────────┤
│ MACS per PE │ 16 │
├─────────────────────┼───────────────────────────────────────────────────────────────────────┤
│ Max L2M │ 0.000MB │
├─────────────────────┼───────────────────────────────────────────────────────────────────────┤
│ Max LRM │ 0.000kB │
├─────────────────────┼───────────────────────────────────────────────────────────────────────┤
│ Max Temp Ext Bytes │ 96.000MB │
├─────────────────────┼───────────────────────────────────────────────────────────────────────┤
│ Network GMACs │ │
╘═════════════════════╧═══════════════════════════════════════════════════════════════════════╛
╒═════╤════════╤══════════════════════════╤═══════════════════╤══════════════════════════╤═══════╕
│ │ Type │ Name │ shape │ type │ mse │
╞═════╪════════╪══════════════════════════╪═══════════════════╪══════════════════════════╪═══════╡
│ 0 │ Input │ input_ids │ [1, 1024] │ tensor[int32] │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 1 │ Input │ attention_mask │ [1, 1024, 1024] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 2 │ Input │ sin │ [1, 1024, 128] │ tensor[FixedPoint32<30>] │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 3 │ Input │ cos │ [1, 1024, 128] │ tensor[FixedPoint32<30>] │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 4 │ Input │ past_key_values.0.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 5 │ Input │ past_key_values.0.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 6 │ Input │ past_key_values.1.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 7 │ Input │ past_key_values.1.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 8 │ Input │ past_key_values.2.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 9 │ Input │ past_key_values.2.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 10 │ Input │ past_key_values.3.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 11 │ Input │ past_key_values.3.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 12 │ Input │ past_key_values.4.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 13 │ Input │ past_key_values.4.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 14 │ Input │ past_key_values.5.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 15 │ Input │ past_key_values.5.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 16 │ Input │ past_key_values.6.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 17 │ Input │ past_key_values.6.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 18 │ Input │ past_key_values.7.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 19 │ Input │ past_key_values.7.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 20 │ Input │ past_key_values.8.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 21 │ Input │ past_key_values.8.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 22 │ Input │ past_key_values.9.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 23 │ Input │ past_key_values.9.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 24 │ Input │ past_key_values.10.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 25 │ Input │ past_key_values.10.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 26 │ Input │ past_key_values.11.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 27 │ Input │ past_key_values.11.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 28 │ Input │ past_key_values.12.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 29 │ Input │ past_key_values.12.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 30 │ Input │ past_key_values.13.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 31 │ Input │ past_key_values.13.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 32 │ Input │ past_key_values.14.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 33 │ Input │ past_key_values.14.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 34 │ Input │ past_key_values.15.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 35 │ Input │ past_key_values.15.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 36 │ Input │ past_key_values.16.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 37 │ Input │ past_key_values.16.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 38 │ Input │ past_key_values.17.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 39 │ Input │ past_key_values.17.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 40 │ Input │ past_key_values.18.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 41 │ Input │ past_key_values.18.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 42 │ Input │ past_key_values.19.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 43 │ Input │ past_key_values.19.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 44 │ Input │ past_key_values.20.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 45 │ Input │ past_key_values.20.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 46 │ Input │ past_key_values.21.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 47 │ Input │ past_key_values.21.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 48 │ Input │ past_key_values.22.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 49 │ Input │ past_key_values.22.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 50 │ Input │ past_key_values.23.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 51 │ Input │ past_key_values.23.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 52 │ Input │ past_key_values.24.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 53 │ Input │ past_key_values.24.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 54 │ Input │ past_key_values.25.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 55 │ Input │ past_key_values.25.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 56 │ Input │ past_key_values.26.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 57 │ Input │ past_key_values.26.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 58 │ Input │ past_key_values.27.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 59 │ Input │ past_key_values.27.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 60 │ Input │ past_key_values.28.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 61 │ Input │ past_key_values.28.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 62 │ Input │ past_key_values.29.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 63 │ Input │ past_key_values.29.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 64 │ Input │ past_key_values.30.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 65 │ Input │ past_key_values.30.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 66 │ Input │ past_key_values.31.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 67 │ Input │ past_key_values.31.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 68 │ Input │ past_key_values.32.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 69 │ Input │ past_key_values.32.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 70 │ Input │ past_key_values.33.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 71 │ Input │ past_key_values.33.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 72 │ Input │ past_key_values.34.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 73 │ Input │ past_key_values.34.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 74 │ Input │ past_key_values.35.key │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 75 │ Input │ past_key_values.35.value │ [1, 8, 0, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 76 │ Output │ logits │ [1, 151936] │ tensor[FixedPoint32<26>] │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 77 │ Output │ present.0.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 78 │ Output │ present.0.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 79 │ Output │ present.1.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 80 │ Output │ present.1.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 81 │ Output │ present.2.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 82 │ Output │ present.2.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 83 │ Output │ present.3.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 84 │ Output │ present.3.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 85 │ Output │ present.4.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 86 │ Output │ present.4.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 87 │ Output │ present.5.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 88 │ Output │ present.5.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 89 │ Output │ present.6.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 90 │ Output │ present.6.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 91 │ Output │ present.7.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 92 │ Output │ present.7.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 93 │ Output │ present.8.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 94 │ Output │ present.8.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 95 │ Output │ present.9.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 96 │ Output │ present.9.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 97 │ Output │ present.10.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 98 │ Output │ present.10.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 99 │ Output │ present.11.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 100 │ Output │ present.11.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 101 │ Output │ present.12.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 102 │ Output │ present.12.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 103 │ Output │ present.13.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 104 │ Output │ present.13.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 105 │ Output │ present.14.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 106 │ Output │ present.14.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 107 │ Output │ present.15.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 108 │ Output │ present.15.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 109 │ Output │ present.16.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 110 │ Output │ present.16.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 111 │ Output │ present.17.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 112 │ Output │ present.17.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 113 │ Output │ present.18.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 114 │ Output │ present.18.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 115 │ Output │ present.19.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 116 │ Output │ present.19.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 117 │ Output │ present.20.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 118 │ Output │ present.20.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 119 │ Output │ present.21.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 120 │ Output │ present.21.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 121 │ Output │ present.22.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 122 │ Output │ present.22.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 123 │ Output │ present.23.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 124 │ Output │ present.23.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 125 │ Output │ present.24.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 126 │ Output │ present.24.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 127 │ Output │ present.25.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 128 │ Output │ present.25.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 129 │ Output │ present.26.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 130 │ Output │ present.26.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 131 │ Output │ present.27.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 132 │ Output │ present.27.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 133 │ Output │ present.28.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 134 │ Output │ present.28.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 135 │ Output │ present.29.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 136 │ Output │ present.29.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 137 │ Output │ present.30.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 138 │ Output │ present.30.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 139 │ Output │ present.31.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 140 │ Output │ present.31.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 141 │ Output │ present.32.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 142 │ Output │ present.32.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 143 │ Output │ present.33.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 144 │ Output │ present.33.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 145 │ Output │ present.34.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 146 │ Output │ present.34.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 147 │ Output │ present.35.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 148 │ Output │ present.35.value │ [1, 8, 1024, 128] │ n/a │ n/a │
╘═════╧════════╧══════════════════════════╧═══════════════════╧══════════════════════════╧═══════╛
project_prefill_latency(
PREFILL_DIR / f"decoder_{VALIDATE_DECODER}",
num_decoders=NUM_DECODERS,
seq_len=SEQ_LEN,
clock_ghz=1.7,
)
Decoder cycles (measured) : 102,405,943 MAC util 3.1%
Full prefill (36 decoders) : 3,686,613,948 cycles
Time to first token : 2,169 ms @ 1.7 GHz
Prefill throughput : 472 tokens/s
{'total_cycles': 102405943,
'total_mac_cycles': 3158964,
'mac_util': 3.0847467514653912,
'n_regions': 9,
'iss_log_path': 'prefill/decoder_2/qwen3_prefill_runner_QC-P_1d0_2MB_4kB_24GBps_24GBps_16_OFF_x1_x4/output/iss_run_log.txt'}
Summary
| Model | Qwen3-8B — 36 layers, hidden size 4096, 32 query / 8 KV heads, 151,936-token vocabulary |
| Phase | Prefill — 1024-token prompt in, first-token logits and a filled KV cache out |
| Quantization | W4A8 SmoothQuant (α = 0.36): INT4 weights packed v8i4, INT8 activations |
| Target | QC-P × 4 cores, 2 MB OCM per core, 16 MACs/PE, 1 GHz, 24 GB/s DDR per core |
| Validated | Decoder 2 against ONNX Runtime on the ISS, element by element |
| Compiled | All 36 decoders + LM head through CGC |
| Custom Ops | nn::qwen3PrefillAttention, nn::gateUpProjection, linalg::channelwiseQuantMatMul |
Key takeaways
- Validate a decoder, not the network. Cutting one layer, compiling it and comparing it against ONNX Runtime pins any numerical drift to a single layer — and its measured cycles project the full-prefill time to first token, which a 36-layer ISS run could not deliver in reasonable time.
- One export, two graphs. Prefill and decode differ only in their fixed shapes; weights, tensor ranges and custom ops are shared, so the prefill program is one shape-fix away from the decode kernel in
qwen3_8b.ipynb. sdk sourcecloses the loop. One command compiles the CGC output and a hand-written host runner with Quadric LLVM and executes them on the cycle-accurate multicore ISS.
Citation
@article{qwen3technicalreport,
title = {Qwen3 Technical Report},
author = {Qwen Team},
journal = {arXiv preprint arXiv:2505.09388},
year = {2025}
}
