NOTE: The Jupyter Notebook below is included in the Chimera SDK and can be run interactively by running the following CLI command:
$ quadric sdk notebook
From the Jupyter Notebook window in your browser, select the notebook named /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/qwen3_prefill_validation.ipynb.
Qwen3 Prefill Decoder Validation
Demonstrating the isolated and chained validation workflow on Quadric GPNPU hardware
1. Motivation
The Qwen3 prefill model consists of 36 stacked transformer decoder layers (decoders 0–35). During prefill, the full sequence (up to 1024 tokens) is processed in a single forward pass — generating the initial KV cache before autoregressive decode begins.
Validating the full 36-layer model end-to-end on the GPNPU makes it difficult to isolate where numerical errors originate. Instead, we validate decoder by decoder — the smallest repeatable unit of the network — comparing GPNPU outputs against ONNX Runtime (ORT) float32 reference outputs.
Why decoder-by-decoder?
- Each decoder is an independent subgraph with well-defined float32 inputs and outputs
- Fixed-point quantization errors can accumulate across layers — per-decoder validation lets us track exactly where divergence starts
- A passing decoder in isolation confirms the GPNPU op implementation is correct at that layer's precision
Two validation modes
| Mode | Input to GPNPU | GPNPU Output | Chain to next decoder via |
|---|---|---|---|
| Isolated | ORT output of previous decoder (input_logits_ort.bin) | epu_outputs/output_isolated.bin | ORT output (chain_ort_outputs.py) |
| Chained | GPNPU output of previous decoder (input_logits_epu.bin) | epu_outputs/output_chained.bin | GPNPU output (chain_decoder_outputs.py) |
Isolated mode tests each decoder independently with clean ORT inputs — errors cannot propagate between decoders. This is the first pass to verify each layer's GPNPU implementation.
Chained mode feeds GPNPU outputs into the next decoder, simulating real end-to-end inference. Errors accumulate across decoders, revealing how quantization noise propagates through the full network.
2. Validation Pipeline
All steps are orchestrated by run_decoder_pipeline.py. The diagram below shows the full pipeline for a single decoder.
Input: qwen3_8b_prefill_1024_context.onnx
│
▼
┌─────────────────────────────┐
│ Step 1 — Cut Decoder │ cut_model.cut_decoder()
└──────────────┬──────────────┘ Slice decoder_N subgraph from the full prefill model
│ → decoder_N/decoder_N.onnx
▼
┌─────────────────────────────┐
│ Step 2 — Custom Op Match │ custom_op_match.qwen3_custom_op_replacer()
└──────────────┬──────────────┘ Replace ONNX ops with Quadric custom ops using tranges
│ → decoder_N/custom_op_decoder_N.onnx
▼
┌─────────────────────────────┐
│ Step 3 — CGC Compile │ compile_cpp.run_compile_cpp()
└──────────────┬──────────────┘ Compile custom-op ONNX → CCL C++
│ → decoder_N/ccl_build/
▼
┌─────────────────────────────┐
│ Step 4 — SDK / ISS Run │ sdk source
└──────────────┬──────────────┘ C++ → GPNPU assembly, run on Instruction Set Simulator
│ → epu_outputs/output_isolated.bin (or output_chained.bin)
▼
┌─────────────────────────────┐
│ Step 5 — Validate │ run_compare_onnx.run_and_compare_onnx()
└──────────────┬──────────────┘ isolated: GPNPU vs ORT (atol/rtol gate)
│ chained: chained vs isolated baseline (cosine gate)
▼
┌─────────────────────────────┐
│ Step 6 — Plot Outputs │ plot_comparison.run_plot()
└──────────────┬──────────────┘ Scatter + diff histogram of GPNPU vs ORT
│ → decoder_N/output_comparison.png
▼
┌─────────────────────────────┐
│ Step 7 — Update Report │ update_report.update_report()
└──────────────┬──────────────┘ Append decoder row to decoder_report.md
│ → decoder_report.md
▼
┌─────────────────────────────┐
│ Step 8 — Chain Output │ chain_ort_outputs.py (isolated)
└──────────────┬──────────────┘ chain_decoder_outputs.py (chained)
┌────┴─────┐
▼ ▼
input_logits_ input_logits_
ort.bin epu.bin
(isolated) (chained)
Pipeline step reference
| Step | Module · Function | What it does | Key output |
|---|---|---|---|
| 1 | cut_model.cut_decoder() | Slices a single decoder subgraph from the full 36-layer prefill ONNX | decoder_N/decoder_N.onnx |
| 2 | custom_op_match.qwen3_custom_op_replacer() | Replaces standard ONNX ops (MatMul, LayerNorm, Attention) with Quadric custom ops using calibrated tranges | decoder_N/custom_op_decoder_N.onnx |
| 3 | compile_cpp.run_compile_cpp() (CGC) | Compiles the custom-op ONNX to CCL C++ via the Quadric graph compiler | decoder_N/ccl_build/ |
| 4 | sdk source (ISS) | Compiles C++ to GPNPU assembly and runs the ISS simulator | epu_outputs/output_isolated.bin or output_chained.bin |
| 5 | run_compare_onnx.run_and_compare_onnx() | Isolated: runs ORT on the same input, dequantizes GPNPU output, computes atol/rtol/Pearson r pass/fail. Chained: skips ORT; compares output_chained.bin against output_isolated.bin using a per-token cosine gate (see §8). | Console report + (isolated only) onnx_runtime_outputs/output_ort.npy |
| 6 | plot_comparison.run_plot() | Scatter + histogram of GPNPU vs ORT (isolated) / chained vs isolated baseline (chained) | decoder_N/output_comparison.png |
| 7 | update_report.update_report() | Appends or updates the decoder's row in decoder_report.md (isolated) or decoder_report_chained.md (chained) | decoder_report*.md |
| 8 | chain_ort_outputs.py / chain_decoder_outputs.py | Prepares input_logits_*.bin for the next decoder | decoder_N+1/input_logits_ort.bin or input_logits_epu.bin |
Example invocations
Full pipeline run (decoder 5, isolated):
python3 run_decoder_pipeline.py --decoder 5 --mode isolated \
--tranges qwen_0_36_calibrated.onnx.tranges \
--num-decoders 36 --num-cores 4 --atol 0.7 --rtol 1e-2
Validate-only (existing ISS output, decoder 7):
python3 run_decoder_pipeline.py --decoder 7 --mode isolated --validate-only \
--tranges qwen_0_36_calibrated.onnx.tranges --atol 0.7 --rtol 1e-2
Chained mode (decoder 3):
python3 run_decoder_pipeline.py --decoder 3 --mode chained --skip-cut \
--tranges qwen_0_36_calibrated.onnx.tranges --atol 0.7 --rtol 1e-2
3. Setup and Model Download
The starting point of this notebook is qwen3_8b_prefill_1024_context.onnx — the shape-fixed, quantized Qwen3-8B prefill model (sequence length 1024). The qwen3_single_decoder.ipynb notebook explains in detail the pre-processing steps needed to produce this graph. Here we download it directly from S3, so the shape-fixing step does not need to be repeated.
Download the pre-built prefill model from S3:
qwen3_8b_prefill_1024_context.onnx— the pre-built prefill model (seq_len=1024)27fe33ec-1773-11f1-b0a3-ea22fe300241— the external weights data fileqwen_0_36_calibrated.onnx.tranges— calibrated tensor ranges for fixed-point op parameterization
import os
from examples.models.zoo.zoo_utils import download_file
## Base URL for model files
base_url = "https://sdk-cli-models.s3.us-east-2.amazonaws.com/"
## File names
prefill_onnx_filename = "qwen3_8b_prefill_1024_context.onnx"
prefill_data_filename = "27fe33ec-1773-11f1-b0a3-ea22fe300241"
prefill_tranges_filename = "qwen_0_36_calibrated.onnx.tranges"
current_dir = os.getcwd()
prefill_onnx_path = os.path.join(current_dir, prefill_onnx_filename)
prefill_data_path = os.path.join(current_dir, prefill_data_filename)
prefill_tranges_path = os.path.join(current_dir, prefill_tranges_filename)
## Download files
print(f"Downloading {prefill_onnx_filename}...")
download_file(base_url + prefill_onnx_filename, prefill_onnx_path)
print(f" ✓ Downloaded {prefill_onnx_filename}")
print(f"Downloading {prefill_data_filename}...")
download_file(base_url + prefill_data_filename, prefill_data_path)
print(f" ✓ Downloaded {prefill_data_filename}")
print(f"Downloading {prefill_tranges_filename}...")
download_file(base_url + prefill_tranges_filename, prefill_tranges_path)
print(f" ✓ Downloaded {prefill_tranges_filename}")
print("\nAll prefill model files downloaded successfully!")
Downloading qwen3_8b_prefill_1024_context.onnx...
✓ Downloaded qwen3_8b_prefill_1024_context.onnx
Downloading 27fe33ec-1773-11f1-b0a3-ea22fe300241...
✓ Downloaded 27fe33ec-1773-11f1-b0a3-ea22fe300241
Downloading qwen_0_36_calibrated.onnx.tranges...
✓ Downloaded qwen_0_36_calibrated.onnx.tranges
All prefill model files downloaded successfully!
4. Tokenize the prompt
Before running the bootstrap, we need to produce decoder_0/input_logits.bin — the tokenized prompt (1024 int32 token IDs) that decoder 0's ONNX Runtime session reads as input_ids. Unlike the per-decoder hidden-state inputs for decoders 1–35, this first input cannot be derived from the chain — it has to come from tokenizing an actual prompt string.
The cell below uses prepare_inputs_from_file (from examples.models.qwen.qwen3_8b.prepare_inputs) to:
- Read the prompt at
../qwen3_prefill/prompts/0.txt. - Tokenize it to a 1024-element
int32array, written toinput_logits.binin the working directory. - Compute the RoPE position embeddings →
cos.binandsin.bin(1024 × 128float32each, also at the working-dir level — this is thedata_dirfor the rest of the pipeline). - Record the actual prompt length →
prompt_len.bin.
We then copy input_logits.bin into decoder_0/ because run_compare_onnx.run_and_compare_onnx looks for that file under build_dir = decoder_0/ when running ORT on decoder 0 in non-subgraph mode (i.e. when decoder 0 is the entry point of the prefill graph and consumes raw token IDs, not a hidden-state vector from a previous decoder).
Once this cell completes, the bootstrap loop below has everything it needs to run end-to-end.
import shutil
from pathlib import Path
from examples.models.qwen.qwen3_8b.prepare_inputs import prepare_inputs_from_file
prompt_path = "../qwen3_prefill/prompts/0.txt"
print(f"Tokenizing {prompt_path} → input_logits.bin / cos.bin / sin.bin / prompt_len.bin")
prepare_inputs_from_file(prompt_path, num_tokens=1024, new_tokens=1)
## Copy input_logits.bin into decoder_0/ — run_compare_onnx.run_and_compare_onnx reads it from build_dir for decoder 0.
Path("decoder_0").mkdir(exist_ok=True)
_ = shutil.copy("input_logits.bin", "decoder_0/input_logits.bin")
Tokenizing ../qwen3_prefill/prompts/0.txt → input_logits.bin / cos.bin / sin.bin / prompt_len.bin
Extracted 1024 tokens from ../qwen3_prefill/prompts/0.txt
Token IDs: [78043 1573 279 1809 1154 13505 3234 7228 9787 1419]... (showing first 10)
Prompt length: 1024
5. Bootstrap ORT ground truth
Each per-decoder validation compares GPNPU output against an ONNX Runtime (ORT) ground truth computed on the cut sub-graph for that decoder. For decoder N (N ≥ 1), the ORT session needs the previous decoder's hidden-state output as input.
The bootstrap below runs run_decoder_pipeline.py --ort-only once per decoder, which performs only the steps needed to produce the chain:
- Step 1 — Cut decoder N from the full prefill ONNX (
decoder_N/decoder_N.onnx). - Step 5 — Run ORT on the cut sub-graph (skipping the GPNPU compile/compare path), writing
decoder_N/onnx_runtime_outputs/output_0.npy. - Step 8 — Chain decoder N's ORT output into the next decoder's input directory:
decoder_{N+1}/input_logits_ort_float.bin— raw float32, consumed by ORT in subgraph mode.decoder_{N+1}/input_logits_ort.bin— FixedPoint32<frac_bits>, consumed by the GPNPU wrapper in isolated mode. The frac_bits are auto-deduced from the calibrated.trangesfile.
Steps 2–4 (custom-op match, CGC compile, SDK compile) and Steps 6–7 (plot, report) are skipped — they aren't required to produce the ORT chain.
After this cell completes, every decoder has its output_0.npy, input_logits_ort_float.bin, and input_logits_ort.bin in place, so the per-decoder demo cells below can run isolated-mode validation against a fresh ORT-chained baseline.
from run_decoder_pipeline import run_pipeline
NUM_DECODERS = 36
INPUT_MODEL = "qwen3_8b_prefill_1024_context.onnx"
for n in range(NUM_DECODERS):
print("\n" + "=" * 64)
print(f" ORT bootstrap: decoder {n}")
print("=" * 64)
run_pipeline(decoder=n, ort_only=True, input_model=INPUT_MODEL)
================================================================
ORT bootstrap: decoder 0
================================================================
=== Decoder 0 pipeline (mode=isolated, cores=4) ===
════════════════════════════════════════════════════════════════
── Step 1: Cut decoder 0 from qwen3_8b_prefill_1024_context.onnx ──
Cutting decoder 0:
Input edges: ['input_ids', 'attention_mask', 'cos', 'sin', 'past_key_values.0.key', 'past_key_values.0.value']
Output edges: ['/model/layers.0/Add_1_output_0', 'present.0.key', 'present.0.value']
Successfully saved to /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_0/decoder_0.onnx
── Step 6: Run ORT (--ort-only) ──
✓ ORT complete → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_0/onnx_runtime_outputs/output_0.npy
── Step 9: Chain output to decoder 1 ──
✓ Chained ORT 0 → 1 (FixedPoint32<26>, 16.0 MB)
✓ Chained: decoder_0/onnx_runtime_outputs/output_0.npy → decoder_1/input_logits_ort_float.bin
════════════════════════════════════════════════════════════════
Decoder 0 ort-only bootstrap complete
════════════════════════════════════════════════════════════════
================================================================
ORT bootstrap: decoder 1
================================================================
=== Decoder 1 pipeline (mode=isolated, cores=4) ===
════════════════════════════════════════════════════════════════
── Step 1: Cut decoder 1 from qwen3_8b_prefill_1024_context.onnx ──
Cutting decoder 1:
Input edges: ['/model/layers.0/Add_1_output_0', 'attention_mask', 'cos', 'sin', 'past_key_values.1.key', 'past_key_values.1.value']
Output edges: ['/model/layers.1/Add_1_output_0', 'present.1.key', 'present.1.value']
Successfully saved to /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_1/decoder_1.onnx
── Step 6: Run ORT (--ort-only) ──
✓ ORT complete → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_1/onnx_runtime_outputs/output_0.npy
── Step 9: Chain output to decoder 2 ──
✓ Chained ORT 1 → 2 (FixedPoint32<25>, 16.0 MB)
✓ Chained: decoder_1/onnx_runtime_outputs/output_0.npy → decoder_2/input_logits_ort_float.bin
════════════════════════════════════════════════════════════════
Decoder 1 ort-only bootstrap complete
════════════════════════════════════════════════════════════════
================================================================
ORT bootstrap: decoder 2
================================================================
=== Decoder 2 pipeline (mode=isolated, cores=4) ===
════════════════════════════════════════════════════════════════
── Step 1: Cut decoder 2 from qwen3_8b_prefill_1024_context.onnx ──
Cutting decoder 2:
Input edges: ['/model/layers.1/Add_1_output_0', 'attention_mask', 'cos', 'sin', 'past_key_values.2.key', 'past_key_values.2.value']
Output edges: ['/model/layers.2/Add_1_output_0', 'present.2.key', 'present.2.value']
Successfully saved to /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_2/decoder_2.onnx
── Step 6: Run ORT (--ort-only) ──
✓ ORT complete → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_2/onnx_runtime_outputs/output_0.npy
── Step 9: Chain output to decoder 3 ──
✓ Chained ORT 2 → 3 (FixedPoint32<24>, 16.0 MB)
✓ Chained: decoder_2/onnx_runtime_outputs/output_0.npy → decoder_3/input_logits_ort_float.bin
════════════════════════════════════════════════════════════════
Decoder 2 ort-only bootstrap complete
════════════════════════════════════════════════════════════════
================================================================
ORT bootstrap: decoder 3
================================================================
=== Decoder 3 pipeline (mode=isolated, cores=4) ===
════════════════════════════════════════════════════════════════
── Step 1: Cut decoder 3 from qwen3_8b_prefill_1024_context.onnx ──
Cutting decoder 3:
Input edges: ['/model/layers.2/Add_1_output_0', 'attention_mask', 'cos', 'sin', 'past_key_values.3.key', 'past_key_values.3.value']
Output edges: ['/model/layers.3/Add_1_output_0', 'present.3.key', 'present.3.value']
Successfully saved to /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_3/decoder_3.onnx
── Step 6: Run ORT (--ort-only) ──
✓ ORT complete → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_3/onnx_runtime_outputs/output_0.npy
── Step 9: Chain output to decoder 4 ──
✓ Chained ORT 3 → 4 (FixedPoint32<24>, 16.0 MB)
✓ Chained: decoder_3/onnx_runtime_outputs/output_0.npy → decoder_4/input_logits_ort_float.bin
════════════════════════════════════════════════════════════════
Decoder 3 ort-only bootstrap complete
════════════════════════════════════════════════════════════════
================================================================
ORT bootstrap: decoder 4
================================================================
=== Decoder 4 pipeline (mode=isolated, cores=4) ===
════════════════════════════════════════════════════════════════
── Step 1: Cut decoder 4 from qwen3_8b_prefill_1024_context.onnx ──
Cutting decoder 4:
Input edges: ['/model/layers.3/Add_1_output_0', 'attention_mask', 'cos', 'sin', 'past_key_values.4.key', 'past_key_values.4.value']
Output edges: ['/model/layers.4/Add_1_output_0', 'present.4.key', 'present.4.value']
Successfully saved to /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_4/decoder_4.onnx
── Step 6: Run ORT (--ort-only) ──
✓ ORT complete → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_4/onnx_runtime_outputs/output_0.npy
── Step 9: Chain output to decoder 5 ──
✓ Chained ORT 4 → 5 (FixedPoint32<24>, 16.0 MB)
✓ Chained: decoder_4/onnx_runtime_outputs/output_0.npy → decoder_5/input_logits_ort_float.bin
════════════════════════════════════════════════════════════════
Decoder 4 ort-only bootstrap complete
════════════════════════════════════════════════════════════════
================================================================
ORT bootstrap: decoder 5
================================================================
=== Decoder 5 pipeline (mode=isolated, cores=4) ===
════════════════════════════════════════════════════════════════
── Step 1: Cut decoder 5 from qwen3_8b_prefill_1024_context.onnx ──
Cutting decoder 5:
Input edges: ['/model/layers.4/Add_1_output_0', 'attention_mask', 'cos', 'sin', 'past_key_values.5.key', 'past_key_values.5.value']
Output edges: ['/model/layers.5/Add_1_output_0', 'present.5.key', 'present.5.value']
Successfully saved to /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_5/decoder_5.onnx
── Step 6: Run ORT (--ort-only) ──
✓ ORT complete → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_5/onnx_runtime_outputs/output_0.npy
── Step 9: Chain output to decoder 6 ──
✓ Chained ORT 5 → 6 (FixedPoint32<24>, 16.0 MB)
✓ Chained: decoder_5/onnx_runtime_outputs/output_0.npy → decoder_6/input_logits_ort_float.bin
════════════════════════════════════════════════════════════════
Decoder 5 ort-only bootstrap complete
════════════════════════════════════════════════════════════════
================================================================
ORT bootstrap: decoder 6
================================================================
=== Decoder 6 pipeline (mode=isolated, cores=4) ===
════════════════════════════════════════════════════════════════
── Step 1: Cut decoder 6 from qwen3_8b_prefill_1024_context.onnx ──
Cutting decoder 6:
Input edges: ['/model/layers.5/Add_1_output_0', 'attention_mask', 'cos', 'sin', 'past_key_values.6.key', 'past_key_values.6.value']
Output edges: ['/model/layers.6/Add_1_output_0', 'present.6.key', 'present.6.value']
Successfully saved to /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_6/decoder_6.onnx
── Step 6: Run ORT (--ort-only) ──
✓ ORT complete → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_6/onnx_runtime_outputs/output_0.npy
── Step 9: Chain output to decoder 7 ──
✓ Chained ORT 6 → 7 (FixedPoint32<17>, 16.0 MB)
✓ Chained: decoder_6/onnx_runtime_outputs/output_0.npy → decoder_7/input_logits_ort_float.bin
════════════════════════════════════════════════════════════════
Decoder 6 ort-only bootstrap complete
════════════════════════════════════════════════════════════════
================================================================
ORT bootstrap: decoder 7
================================================================
=== Decoder 7 pipeline (mode=isolated, cores=4) ===
════════════════════════════════════════════════════════════════
── Step 1: Cut decoder 7 from qwen3_8b_prefill_1024_context.onnx ──
Cutting decoder 7:
Input edges: ['/model/layers.6/Add_1_output_0', 'attention_mask', 'cos', 'sin', 'past_key_values.7.key', 'past_key_values.7.value']
Output edges: ['/model/layers.7/Add_1_output_0', 'present.7.key', 'present.7.value']
Successfully saved to /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_7/decoder_7.onnx
── Step 6: Run ORT (--ort-only) ──
✓ ORT complete → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_7/onnx_runtime_outputs/output_0.npy
── Step 9: Chain output to decoder 8 ──
✓ Chained ORT 7 → 8 (FixedPoint32<17>, 16.0 MB)
✓ Chained: decoder_7/onnx_runtime_outputs/output_0.npy → decoder_8/input_logits_ort_float.bin
════════════════════════════════════════════════════════════════
Decoder 7 ort-only bootstrap complete
════════════════════════════════════════════════════════════════
================================================================
ORT bootstrap: decoder 8
================================================================
=== Decoder 8 pipeline (mode=isolated, cores=4) ===
════════════════════════════════════════════════════════════════
── Step 1: Cut decoder 8 from qwen3_8b_prefill_1024_context.onnx ──
Cutting decoder 8:
Input edges: ['/model/layers.7/Add_1_output_0', 'attention_mask', 'cos', 'sin', 'past_key_values.8.key', 'past_key_values.8.value']
Output edges: ['/model/layers.8/Add_1_output_0', 'present.8.key', 'present.8.value']
Successfully saved to /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_8/decoder_8.onnx
── Step 6: Run ORT (--ort-only) ──
✓ ORT complete → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_8/onnx_runtime_outputs/output_0.npy
── Step 9: Chain output to decoder 9 ──
✓ Chained ORT 8 → 9 (FixedPoint32<17>, 16.0 MB)
✓ Chained: decoder_8/onnx_runtime_outputs/output_0.npy → decoder_9/input_logits_ort_float.bin
════════════════════════════════════════════════════════════════
Decoder 8 ort-only bootstrap complete
════════════════════════════════════════════════════════════════
================================================================
ORT bootstrap: decoder 9
================================================================
=== Decoder 9 pipeline (mode=isolated, cores=4) ===
════════════════════════════════════════════════════════════════
── Step 1: Cut decoder 9 from qwen3_8b_prefill_1024_context.onnx ──
Cutting decoder 9:
Input edges: ['/model/layers.8/Add_1_output_0', 'attention_mask', 'cos', 'sin', 'past_key_values.9.key', 'past_key_values.9.value']
Output edges: ['/model/layers.9/Add_1_output_0', 'present.9.key', 'present.9.value']
Successfully saved to /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_9/decoder_9.onnx
── Step 6: Run ORT (--ort-only) ──
✓ ORT complete → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_9/onnx_runtime_outputs/output_0.npy
── Step 9: Chain output to decoder 10 ──
✓ Chained ORT 9 → 10 (FixedPoint32<17>, 16.0 MB)
✓ Chained: decoder_9/onnx_runtime_outputs/output_0.npy → decoder_10/input_logits_ort_float.bin
════════════════════════════════════════════════════════════════
Decoder 9 ort-only bootstrap complete
════════════════════════════════════════════════════════════════
================================================================
ORT bootstrap: decoder 10
================================================================
=== Decoder 10 pipeline (mode=isolated, cores=4) ===
════════════════════════════════════════════════════════════════
── Step 1: Cut decoder 10 from qwen3_8b_prefill_1024_context.onnx ──
Cutting decoder 10:
Input edges: ['/model/layers.9/Add_1_output_0', 'attention_mask', 'cos', 'sin', 'past_key_values.10.key', 'past_key_values.10.value']
Output edges: ['/model/layers.10/Add_1_output_0', 'present.10.key', 'present.10.value']
Successfully saved to /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_10/decoder_10.onnx
── Step 6: Run ORT (--ort-only) ──
✓ ORT complete → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_10/onnx_runtime_outputs/output_0.npy
── Step 9: Chain output to decoder 11 ──
✓ Chained ORT 10 → 11 (FixedPoint32<17>, 16.0 MB)
✓ Chained: decoder_10/onnx_runtime_outputs/output_0.npy → decoder_11/input_logits_ort_float.bin
════════════════════════════════════════════════════════════════
Decoder 10 ort-only bootstrap complete
════════════════════════════════════════════════════════════════
================================================================
ORT bootstrap: decoder 11
================================================================
=== Decoder 11 pipeline (mode=isolated, cores=4) ===
════════════════════════════════════════════════════════════════
── Step 1: Cut decoder 11 from qwen3_8b_prefill_1024_context.onnx ──
Cutting decoder 11:
Input edges: ['/model/layers.10/Add_1_output_0', 'attention_mask', 'cos', 'sin', 'past_key_values.11.key', 'past_key_values.11.value']
Output edges: ['/model/layers.11/Add_1_output_0', 'present.11.key', 'present.11.value']
Successfully saved to /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_11/decoder_11.onnx
── Step 6: Run ORT (--ort-only) ──
✓ ORT complete → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_11/onnx_runtime_outputs/output_0.npy
── Step 9: Chain output to decoder 12 ──
✓ Chained ORT 11 → 12 (FixedPoint32<17>, 16.0 MB)
✓ Chained: decoder_11/onnx_runtime_outputs/output_0.npy → decoder_12/input_logits_ort_float.bin
════════════════════════════════════════════════════════════════
Decoder 11 ort-only bootstrap complete
════════════════════════════════════════════════════════════════
================================================================
ORT bootstrap: decoder 12
================================================================
=== Decoder 12 pipeline (mode=isolated, cores=4) ===
════════════════════════════════════════════════════════════════
── Step 1: Cut decoder 12 from qwen3_8b_prefill_1024_context.onnx ──
Cutting decoder 12:
Input edges: ['/model/layers.11/Add_1_output_0', 'attention_mask', 'cos', 'sin', 'past_key_values.12.key', 'past_key_values.12.value']
Output edges: ['/model/layers.12/Add_1_output_0', 'present.12.key', 'present.12.value']
Successfully saved to /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_12/decoder_12.onnx
── Step 6: Run ORT (--ort-only) ──
✓ ORT complete → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_12/onnx_runtime_outputs/output_0.npy
── Step 9: Chain output to decoder 13 ──
✓ Chained ORT 12 → 13 (FixedPoint32<17>, 16.0 MB)
✓ Chained: decoder_12/onnx_runtime_outputs/output_0.npy → decoder_13/input_logits_ort_float.bin
════════════════════════════════════════════════════════════════
Decoder 12 ort-only bootstrap complete
════════════════════════════════════════════════════════════════
================================================================
ORT bootstrap: decoder 13
================================================================
=== Decoder 13 pipeline (mode=isolated, cores=4) ===
════════════════════════════════════════════════════════════════
── Step 1: Cut decoder 13 from qwen3_8b_prefill_1024_context.onnx ──
Cutting decoder 13:
Input edges: ['/model/layers.12/Add_1_output_0', 'attention_mask', 'cos', 'sin', 'past_key_values.13.key', 'past_key_values.13.value']
Output edges: ['/model/layers.13/Add_1_output_0', 'present.13.key', 'present.13.value']
Successfully saved to /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_13/decoder_13.onnx
── Step 6: Run ORT (--ort-only) ──
✓ ORT complete → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_13/onnx_runtime_outputs/output_0.npy
── Step 9: Chain output to decoder 14 ──
✓ Chained ORT 13 → 14 (FixedPoint32<17>, 16.0 MB)
✓ Chained: decoder_13/onnx_runtime_outputs/output_0.npy → decoder_14/input_logits_ort_float.bin
════════════════════════════════════════════════════════════════
Decoder 13 ort-only bootstrap complete
════════════════════════════════════════════════════════════════
================================================================
ORT bootstrap: decoder 14
================================================================
=== Decoder 14 pipeline (mode=isolated, cores=4) ===
════════════════════════════════════════════════════════════════
── Step 1: Cut decoder 14 from qwen3_8b_prefill_1024_context.onnx ──
Cutting decoder 14:
Input edges: ['/model/layers.13/Add_1_output_0', 'attention_mask', 'cos', 'sin', 'past_key_values.14.key', 'past_key_values.14.value']
Output edges: ['/model/layers.14/Add_1_output_0', 'present.14.key', 'present.14.value']
Successfully saved to /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_14/decoder_14.onnx
── Step 6: Run ORT (--ort-only) ──
✓ ORT complete → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_14/onnx_runtime_outputs/output_0.npy
── Step 9: Chain output to decoder 15 ──
✓ Chained ORT 14 → 15 (FixedPoint32<17>, 16.0 MB)
✓ Chained: decoder_14/onnx_runtime_outputs/output_0.npy → decoder_15/input_logits_ort_float.bin
════════════════════════════════════════════════════════════════
Decoder 14 ort-only bootstrap complete
════════════════════════════════════════════════════════════════
================================================================
ORT bootstrap: decoder 15
================================================================
=== Decoder 15 pipeline (mode=isolated, cores=4) ===
════════════════════════════════════════════════════════════════
── Step 1: Cut decoder 15 from qwen3_8b_prefill_1024_context.onnx ──
Cutting decoder 15:
Input edges: ['/model/layers.14/Add_1_output_0', 'attention_mask', 'cos', 'sin', 'past_key_values.15.key', 'past_key_values.15.value']
Output edges: ['/model/layers.15/Add_1_output_0', 'present.15.key', 'present.15.value']
Successfully saved to /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_15/decoder_15.onnx
── Step 6: Run ORT (--ort-only) ──
✓ ORT complete → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_15/onnx_runtime_outputs/output_0.npy
── Step 9: Chain output to decoder 16 ──
✓ Chained ORT 15 → 16 (FixedPoint32<17>, 16.0 MB)
✓ Chained: decoder_15/onnx_runtime_outputs/output_0.npy → decoder_16/input_logits_ort_float.bin
════════════════════════════════════════════════════════════════
Decoder 15 ort-only bootstrap complete
════════════════════════════════════════════════════════════════
================================================================
ORT bootstrap: decoder 16
================================================================
=== Decoder 16 pipeline (mode=isolated, cores=4) ===
════════════════════════════════════════════════════════════════
── Step 1: Cut decoder 16 from qwen3_8b_prefill_1024_context.onnx ──
Cutting decoder 16:
Input edges: ['/model/layers.15/Add_1_output_0', 'attention_mask', 'cos', 'sin', 'past_key_values.16.key', 'past_key_values.16.value']
Output edges: ['/model/layers.16/Add_1_output_0', 'present.16.key', 'present.16.value']
Successfully saved to /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_16/decoder_16.onnx
── Step 6: Run ORT (--ort-only) ──
✓ ORT complete → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_16/onnx_runtime_outputs/output_0.npy
── Step 9: Chain output to decoder 17 ──
✓ Chained ORT 16 → 17 (FixedPoint32<17>, 16.0 MB)
✓ Chained: decoder_16/onnx_runtime_outputs/output_0.npy → decoder_17/input_logits_ort_float.bin
════════════════════════════════════════════════════════════════
Decoder 16 ort-only bootstrap complete
════════════════════════════════════════════════════════════════
================================================================
ORT bootstrap: decoder 17
================================================================
=== Decoder 17 pipeline (mode=isolated, cores=4) ===
════════════════════════════════════════════════════════════════
── Step 1: Cut decoder 17 from qwen3_8b_prefill_1024_context.onnx ──
Cutting decoder 17:
Input edges: ['/model/layers.16/Add_1_output_0', 'attention_mask', 'cos', 'sin', 'past_key_values.17.key', 'past_key_values.17.value']
Output edges: ['/model/layers.17/Add_1_output_0', 'present.17.key', 'present.17.value']
Successfully saved to /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_17/decoder_17.onnx
── Step 6: Run ORT (--ort-only) ──
✓ ORT complete → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_17/onnx_runtime_outputs/output_0.npy
── Step 9: Chain output to decoder 18 ──
✓ Chained ORT 17 → 18 (FixedPoint32<17>, 16.0 MB)
✓ Chained: decoder_17/onnx_runtime_outputs/output_0.npy → decoder_18/input_logits_ort_float.bin
════════════════════════════════════════════════════════════════
Decoder 17 ort-only bootstrap complete
════════════════════════════════════════════════════════════════
================================================================
ORT bootstrap: decoder 18
================================================================
=== Decoder 18 pipeline (mode=isolated, cores=4) ===
════════════════════════════════════════════════════════════════
── Step 1: Cut decoder 18 from qwen3_8b_prefill_1024_context.onnx ──
Cutting decoder 18:
Input edges: ['/model/layers.17/Add_1_output_0', 'attention_mask', 'cos', 'sin', 'past_key_values.18.key', 'past_key_values.18.value']
Output edges: ['/model/layers.18/Add_1_output_0', 'present.18.key', 'present.18.value']
Successfully saved to /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_18/decoder_18.onnx
── Step 6: Run ORT (--ort-only) ──
✓ ORT complete → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_18/onnx_runtime_outputs/output_0.npy
── Step 9: Chain output to decoder 19 ──
✓ Chained ORT 18 → 19 (FixedPoint32<17>, 16.0 MB)
✓ Chained: decoder_18/onnx_runtime_outputs/output_0.npy → decoder_19/input_logits_ort_float.bin
════════════════════════════════════════════════════════════════
Decoder 18 ort-only bootstrap complete
════════════════════════════════════════════════════════════════
================================================================
ORT bootstrap: decoder 19
================================================================
=== Decoder 19 pipeline (mode=isolated, cores=4) ===
════════════════════════════════════════════════════════════════
── Step 1: Cut decoder 19 from qwen3_8b_prefill_1024_context.onnx ──
Cutting decoder 19:
Input edges: ['/model/layers.18/Add_1_output_0', 'attention_mask', 'cos', 'sin', 'past_key_values.19.key', 'past_key_values.19.value']
Output edges: ['/model/layers.19/Add_1_output_0', 'present.19.key', 'present.19.value']
Successfully saved to /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_19/decoder_19.onnx
── Step 6: Run ORT (--ort-only) ──
✓ ORT complete → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_19/onnx_runtime_outputs/output_0.npy
── Step 9: Chain output to decoder 20 ──
✓ Chained ORT 19 → 20 (FixedPoint32<17>, 16.0 MB)
✓ Chained: decoder_19/onnx_runtime_outputs/output_0.npy → decoder_20/input_logits_ort_float.bin
════════════════════════════════════════════════════════════════
Decoder 19 ort-only bootstrap complete
════════════════════════════════════════════════════════════════
================================================================
ORT bootstrap: decoder 20
================================================================
=== Decoder 20 pipeline (mode=isolated, cores=4) ===
════════════════════════════════════════════════════════════════
── Step 1: Cut decoder 20 from qwen3_8b_prefill_1024_context.onnx ──
Cutting decoder 20:
Input edges: ['/model/layers.19/Add_1_output_0', 'attention_mask', 'cos', 'sin', 'past_key_values.20.key', 'past_key_values.20.value']
Output edges: ['/model/layers.20/Add_1_output_0', 'present.20.key', 'present.20.value']
Successfully saved to /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_20/decoder_20.onnx
── Step 6: Run ORT (--ort-only) ──
✓ ORT complete → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_20/onnx_runtime_outputs/output_0.npy
── Step 9: Chain output to decoder 21 ──
✓ Chained ORT 20 → 21 (FixedPoint32<17>, 16.0 MB)
✓ Chained: decoder_20/onnx_runtime_outputs/output_0.npy → decoder_21/input_logits_ort_float.bin
════════════════════════════════════════════════════════════════
Decoder 20 ort-only bootstrap complete
════════════════════════════════════════════════════════════════
================================================================
ORT bootstrap: decoder 21
================================================================
=== Decoder 21 pipeline (mode=isolated, cores=4) ===
════════════════════════════════════════════════════════════════
── Step 1: Cut decoder 21 from qwen3_8b_prefill_1024_context.onnx ──
Cutting decoder 21:
Input edges: ['/model/layers.20/Add_1_output_0', 'attention_mask', 'cos', 'sin', 'past_key_values.21.key', 'past_key_values.21.value']
Output edges: ['/model/layers.21/Add_1_output_0', 'present.21.key', 'present.21.value']
Successfully saved to /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_21/decoder_21.onnx
── Step 6: Run ORT (--ort-only) ──
✓ ORT complete → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_21/onnx_runtime_outputs/output_0.npy
── Step 9: Chain output to decoder 22 ──
✓ Chained ORT 21 → 22 (FixedPoint32<17>, 16.0 MB)
✓ Chained: decoder_21/onnx_runtime_outputs/output_0.npy → decoder_22/input_logits_ort_float.bin
════════════════════════════════════════════════════════════════
Decoder 21 ort-only bootstrap complete
════════════════════════════════════════════════════════════════
================================================================
ORT bootstrap: decoder 22
================================================================
=== Decoder 22 pipeline (mode=isolated, cores=4) ===
════════════════════════════════════════════════════════════════
── Step 1: Cut decoder 22 from qwen3_8b_prefill_1024_context.onnx ──
Cutting decoder 22:
Input edges: ['/model/layers.21/Add_1_output_0', 'attention_mask', 'cos', 'sin', 'past_key_values.22.key', 'past_key_values.22.value']
Output edges: ['/model/layers.22/Add_1_output_0', 'present.22.key', 'present.22.value']
Successfully saved to /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_22/decoder_22.onnx
── Step 6: Run ORT (--ort-only) ──
✓ ORT complete → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_22/onnx_runtime_outputs/output_0.npy
── Step 9: Chain output to decoder 23 ──
✓ Chained ORT 22 → 23 (FixedPoint32<17>, 16.0 MB)
✓ Chained: decoder_22/onnx_runtime_outputs/output_0.npy → decoder_23/input_logits_ort_float.bin
════════════════════════════════════════════════════════════════
Decoder 22 ort-only bootstrap complete
════════════════════════════════════════════════════════════════
================================================================
ORT bootstrap: decoder 23
================================================================
=== Decoder 23 pipeline (mode=isolated, cores=4) ===
════════════════════════════════════════════════════════════════
── Step 1: Cut decoder 23 from qwen3_8b_prefill_1024_context.onnx ──
Cutting decoder 23:
Input edges: ['/model/layers.22/Add_1_output_0', 'attention_mask', 'cos', 'sin', 'past_key_values.23.key', 'past_key_values.23.value']
Output edges: ['/model/layers.23/Add_1_output_0', 'present.23.key', 'present.23.value']
Successfully saved to /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_23/decoder_23.onnx
── Step 6: Run ORT (--ort-only) ──
✓ ORT complete → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_23/onnx_runtime_outputs/output_0.npy
── Step 9: Chain output to decoder 24 ──
✓ Chained ORT 23 → 24 (FixedPoint32<17>, 16.0 MB)
✓ Chained: decoder_23/onnx_runtime_outputs/output_0.npy → decoder_24/input_logits_ort_float.bin
════════════════════════════════════════════════════════════════
Decoder 23 ort-only bootstrap complete
════════════════════════════════════════════════════════════════
================================================================
ORT bootstrap: decoder 24
================================================================
=== Decoder 24 pipeline (mode=isolated, cores=4) ===
════════════════════════════════════════════════════════════════
── Step 1: Cut decoder 24 from qwen3_8b_prefill_1024_context.onnx ──
Cutting decoder 24:
Input edges: ['/model/layers.23/Add_1_output_0', 'attention_mask', 'cos', 'sin', 'past_key_values.24.key', 'past_key_values.24.value']
Output edges: ['/model/layers.24/Add_1_output_0', 'present.24.key', 'present.24.value']
Successfully saved to /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_24/decoder_24.onnx
── Step 6: Run ORT (--ort-only) ──
✓ ORT complete → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_24/onnx_runtime_outputs/output_0.npy
── Step 9: Chain output to decoder 25 ──
✓ Chained ORT 24 → 25 (FixedPoint32<17>, 16.0 MB)
✓ Chained: decoder_24/onnx_runtime_outputs/output_0.npy → decoder_25/input_logits_ort_float.bin
════════════════════════════════════════════════════════════════
Decoder 24 ort-only bootstrap complete
════════════════════════════════════════════════════════════════
================================================================
ORT bootstrap: decoder 25
================================================================
=== Decoder 25 pipeline (mode=isolated, cores=4) ===
════════════════════════════════════════════════════════════════
── Step 1: Cut decoder 25 from qwen3_8b_prefill_1024_context.onnx ──
Cutting decoder 25:
Input edges: ['/model/layers.24/Add_1_output_0', 'attention_mask', 'cos', 'sin', 'past_key_values.25.key', 'past_key_values.25.value']
Output edges: ['/model/layers.25/Add_1_output_0', 'present.25.key', 'present.25.value']
Successfully saved to /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_25/decoder_25.onnx
── Step 6: Run ORT (--ort-only) ──
✓ ORT complete → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_25/onnx_runtime_outputs/output_0.npy
── Step 9: Chain output to decoder 26 ──
✓ Chained ORT 25 → 26 (FixedPoint32<17>, 16.0 MB)
✓ Chained: decoder_25/onnx_runtime_outputs/output_0.npy → decoder_26/input_logits_ort_float.bin
════════════════════════════════════════════════════════════════
Decoder 25 ort-only bootstrap complete
════════════════════════════════════════════════════════════════
================================================================
ORT bootstrap: decoder 26
================================================================
=== Decoder 26 pipeline (mode=isolated, cores=4) ===
════════════════════════════════════════════════════════════════
── Step 1: Cut decoder 26 from qwen3_8b_prefill_1024_context.onnx ──
Cutting decoder 26:
Input edges: ['/model/layers.25/Add_1_output_0', 'attention_mask', 'cos', 'sin', 'past_key_values.26.key', 'past_key_values.26.value']
Output edges: ['/model/layers.26/Add_1_output_0', 'present.26.key', 'present.26.value']
Successfully saved to /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_26/decoder_26.onnx
── Step 6: Run ORT (--ort-only) ──
✓ ORT complete → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_26/onnx_runtime_outputs/output_0.npy
── Step 9: Chain output to decoder 27 ──
✓ Chained ORT 26 → 27 (FixedPoint32<17>, 16.0 MB)
✓ Chained: decoder_26/onnx_runtime_outputs/output_0.npy → decoder_27/input_logits_ort_float.bin
════════════════════════════════════════════════════════════════
Decoder 26 ort-only bootstrap complete
════════════════════════════════════════════════════════════════
================================================================
ORT bootstrap: decoder 27
================================================================
=== Decoder 27 pipeline (mode=isolated, cores=4) ===
════════════════════════════════════════════════════════════════
── Step 1: Cut decoder 27 from qwen3_8b_prefill_1024_context.onnx ──
Cutting decoder 27:
Input edges: ['/model/layers.26/Add_1_output_0', 'attention_mask', 'cos', 'sin', 'past_key_values.27.key', 'past_key_values.27.value']
Output edges: ['/model/layers.27/Add_1_output_0', 'present.27.key', 'present.27.value']
Successfully saved to /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_27/decoder_27.onnx
── Step 6: Run ORT (--ort-only) ──
✓ ORT complete → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_27/onnx_runtime_outputs/output_0.npy
── Step 9: Chain output to decoder 28 ──
✓ Chained ORT 27 → 28 (FixedPoint32<17>, 16.0 MB)
✓ Chained: decoder_27/onnx_runtime_outputs/output_0.npy → decoder_28/input_logits_ort_float.bin
════════════════════════════════════════════════════════════════
Decoder 27 ort-only bootstrap complete
════════════════════════════════════════════════════════════════
================================================================
ORT bootstrap: decoder 28
================================================================
=== Decoder 28 pipeline (mode=isolated, cores=4) ===
════════════════════════════════════════════════════════════════
── Step 1: Cut decoder 28 from qwen3_8b_prefill_1024_context.onnx ──
Cutting decoder 28:
Input edges: ['/model/layers.27/Add_1_output_0', 'attention_mask', 'cos', 'sin', 'past_key_values.28.key', 'past_key_values.28.value']
Output edges: ['/model/layers.28/Add_1_output_0', 'present.28.key', 'present.28.value']
Successfully saved to /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_28/decoder_28.onnx
── Step 6: Run ORT (--ort-only) ──
✓ ORT complete → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_28/onnx_runtime_outputs/output_0.npy
── Step 9: Chain output to decoder 29 ──
✓ Chained ORT 28 → 29 (FixedPoint32<17>, 16.0 MB)
✓ Chained: decoder_28/onnx_runtime_outputs/output_0.npy → decoder_29/input_logits_ort_float.bin
════════════════════════════════════════════════════════════════
Decoder 28 ort-only bootstrap complete
════════════════════════════════════════════════════════════════
================================================================
ORT bootstrap: decoder 29
================================================================
=== Decoder 29 pipeline (mode=isolated, cores=4) ===
════════════════════════════════════════════════════════════════
── Step 1: Cut decoder 29 from qwen3_8b_prefill_1024_context.onnx ──
Cutting decoder 29:
Input edges: ['/model/layers.28/Add_1_output_0', 'attention_mask', 'cos', 'sin', 'past_key_values.29.key', 'past_key_values.29.value']
Output edges: ['/model/layers.29/Add_1_output_0', 'present.29.key', 'present.29.value']
Successfully saved to /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_29/decoder_29.onnx
── Step 6: Run ORT (--ort-only) ──
✓ ORT complete → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_29/onnx_runtime_outputs/output_0.npy
── Step 9: Chain output to decoder 30 ──
✓ Chained ORT 29 → 30 (FixedPoint32<17>, 16.0 MB)
✓ Chained: decoder_29/onnx_runtime_outputs/output_0.npy → decoder_30/input_logits_ort_float.bin
════════════════════════════════════════════════════════════════
Decoder 29 ort-only bootstrap complete
════════════════════════════════════════════════════════════════
================================================================
ORT bootstrap: decoder 30
================================================================
=== Decoder 30 pipeline (mode=isolated, cores=4) ===
════════════════════════════════════════════════════════════════
── Step 1: Cut decoder 30 from qwen3_8b_prefill_1024_context.onnx ──
Cutting decoder 30:
Input edges: ['/model/layers.29/Add_1_output_0', 'attention_mask', 'cos', 'sin', 'past_key_values.30.key', 'past_key_values.30.value']
Output edges: ['/model/layers.30/Add_1_output_0', 'present.30.key', 'present.30.value']
Successfully saved to /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_30/decoder_30.onnx
── Step 6: Run ORT (--ort-only) ──
✓ ORT complete → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_30/onnx_runtime_outputs/output_0.npy
── Step 9: Chain output to decoder 31 ──
✓ Chained ORT 30 → 31 (FixedPoint32<17>, 16.0 MB)
✓ Chained: decoder_30/onnx_runtime_outputs/output_0.npy → decoder_31/input_logits_ort_float.bin
════════════════════════════════════════════════════════════════
Decoder 30 ort-only bootstrap complete
════════════════════════════════════════════════════════════════
================================================================
ORT bootstrap: decoder 31
================================================================
=== Decoder 31 pipeline (mode=isolated, cores=4) ===
════════════════════════════════════════════════════════════════
── Step 1: Cut decoder 31 from qwen3_8b_prefill_1024_context.onnx ──
Cutting decoder 31:
Input edges: ['/model/layers.30/Add_1_output_0', 'attention_mask', 'cos', 'sin', 'past_key_values.31.key', 'past_key_values.31.value']
Output edges: ['/model/layers.31/Add_1_output_0', 'present.31.key', 'present.31.value']
Successfully saved to /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_31/decoder_31.onnx
── Step 6: Run ORT (--ort-only) ──
✓ ORT complete → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_31/onnx_runtime_outputs/output_0.npy
── Step 9: Chain output to decoder 32 ──
✓ Chained ORT 31 → 32 (FixedPoint32<17>, 16.0 MB)
✓ Chained: decoder_31/onnx_runtime_outputs/output_0.npy → decoder_32/input_logits_ort_float.bin
════════════════════════════════════════════════════════════════
Decoder 31 ort-only bootstrap complete
════════════════════════════════════════════════════════════════
================================================================
ORT bootstrap: decoder 32
================================================================
=== Decoder 32 pipeline (mode=isolated, cores=4) ===
════════════════════════════════════════════════════════════════
── Step 1: Cut decoder 32 from qwen3_8b_prefill_1024_context.onnx ──
Cutting decoder 32:
Input edges: ['/model/layers.31/Add_1_output_0', 'attention_mask', 'cos', 'sin', 'past_key_values.32.key', 'past_key_values.32.value']
Output edges: ['/model/layers.32/Add_1_output_0', 'present.32.key', 'present.32.value']
Successfully saved to /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_32/decoder_32.onnx
── Step 6: Run ORT (--ort-only) ──
✓ ORT complete → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_32/onnx_runtime_outputs/output_0.npy
── Step 9: Chain output to decoder 33 ──
✓ Chained ORT 32 → 33 (FixedPoint32<17>, 16.0 MB)
✓ Chained: decoder_32/onnx_runtime_outputs/output_0.npy → decoder_33/input_logits_ort_float.bin
════════════════════════════════════════════════════════════════
Decoder 32 ort-only bootstrap complete
════════════════════════════════════════════════════════════════
================================================================
ORT bootstrap: decoder 33
================================================================
=== Decoder 33 pipeline (mode=isolated, cores=4) ===
════════════════════════════════════════════════════════════════
── Step 1: Cut decoder 33 from qwen3_8b_prefill_1024_context.onnx ──
Cutting decoder 33:
Input edges: ['/model/layers.32/Add_1_output_0', 'attention_mask', 'cos', 'sin', 'past_key_values.33.key', 'past_key_values.33.value']
Output edges: ['/model/layers.33/Add_1_output_0', 'present.33.key', 'present.33.value']
Successfully saved to /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_33/decoder_33.onnx
── Step 6: Run ORT (--ort-only) ──
✓ ORT complete → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_33/onnx_runtime_outputs/output_0.npy
── Step 9: Chain output to decoder 34 ──
✓ Chained ORT 33 → 34 (FixedPoint32<17>, 16.0 MB)
✓ Chained: decoder_33/onnx_runtime_outputs/output_0.npy → decoder_34/input_logits_ort_float.bin
════════════════════════════════════════════════════════════════
Decoder 33 ort-only bootstrap complete
════════════════════════════════════════════════════════════════
================================================================
ORT bootstrap: decoder 34
================================================================
=== Decoder 34 pipeline (mode=isolated, cores=4) ===
════════════════════════════════════════════════════════════════
── Step 1: Cut decoder 34 from qwen3_8b_prefill_1024_context.onnx ──
Cutting decoder 34:
Input edges: ['/model/layers.33/Add_1_output_0', 'attention_mask', 'cos', 'sin', 'past_key_values.34.key', 'past_key_values.34.value']
Output edges: ['/model/layers.34/Add_1_output_0', 'present.34.key', 'present.34.value']
Successfully saved to /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_34/decoder_34.onnx
── Step 6: Run ORT (--ort-only) ──
✓ ORT complete → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_34/onnx_runtime_outputs/output_0.npy
── Step 9: Chain output to decoder 35 ──
✓ Chained ORT 34 → 35 (FixedPoint32<18>, 16.0 MB)
✓ Chained: decoder_34/onnx_runtime_outputs/output_0.npy → decoder_35/input_logits_ort_float.bin
════════════════════════════════════════════════════════════════
Decoder 34 ort-only bootstrap complete
════════════════════════════════════════════════════════════════
================================================================
ORT bootstrap: decoder 35
================================================================
=== Decoder 35 pipeline (mode=isolated, cores=4) ===
════════════════════════════════════════════════════════════════
── Step 1: Cut decoder 35 from qwen3_8b_prefill_1024_context.onnx ──
Cutting decoder 35:
Input edges: ['/model/layers.34/Add_1_output_0', 'attention_mask', 'cos', 'sin', 'past_key_values.35.key', 'past_key_values.35.value']
Output edges: ['/model/layers.35/Add_1_output_0', 'present.35.key', 'present.35.value']
Successfully saved to /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_35/decoder_35.onnx
── Step 6: Run ORT (--ort-only) ──
✓ ORT complete → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_35/onnx_runtime_outputs/output_0.npy
── Step 9: Chain output to decoder 36 ──
✓ Chained ORT 35 → 36 (FixedPoint32<18>, 16.0 MB)
✓ Chained: decoder_35/onnx_runtime_outputs/output_0.npy → decoder_36/input_logits_ort_float.bin
════════════════════════════════════════════════════════════════
Decoder 35 ort-only bootstrap complete
════════════════════════════════════════════════════════════════
6. Demo: Decoder 2 — Isolated Mode
Decoder 2 sits at the transition from higher-precision outputs (decoder 0: fp26, decoder 1: fp25) into the mid-range fixed-point regime.
| Parameter | Value |
|---|---|
| Input frac bits | 25 (FixedPoint32<25>) |
| Output frac bits | 24 (FixedPoint32<24>) |
| Input | decoder_2/input_logits_ort.bin — ORT output of decoder 1, quantized to fp25 |
| GPNPU output | decoder_2/epu_outputs/output_isolated.bin — raw int32 fixed-point, scale 2⁻²⁴ |
| ORT reference | decoder_2/onnx_runtime_outputs/output_ort.npy — float32 |
| Tolerance | atol=0.7, rtol=0.01 → pass if |a−b| ≤ 0.7 + 0.01·|b| |
run_pipeline(
decoder=2,
mode="isolated",
skip_cut=True,
tranges="qwen_0_36_calibrated.onnx.tranges",
rtol=1e-2,
max_failures=50,
)
=== Decoder 2 pipeline (mode=isolated, cores=4) ===
── Step 1: Skipping cut (--skip-cut) ──
── Step 2: Custom op matching ──
Warning: /model/layers.2/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Processing matmul ops for decoder 2...
Decoder 2 gate_up frac bits:
gate_combined_scale=5.992835e-02 frac_bits=35
up_combined_scale=5.992835e-02 frac_bits=35
gate_im_frac_bits=27
up_im_frac_bits=27
gate_output_frac_bits=24
up_output_frac_bits=24
Successfully saved modified model to /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_2/custom_op_decoder_2.onnx
✓ Custom op complete → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_2/custom_op_decoder_2.onnx
── Step 3: CGC compile ──
Starting CGC compilation...
2026-06-19 04:07 - INFO - epu - chimera_job - START==================================onnx_ingest
2026-06-19 04:07 - INFO - epu - chimera_job - Numerical ranges provided
2026-06-19 04:07 - INFO - epu - codegen - START===============================optimize_relay
2026-06-19 04:07 - INFO - epu - codegen - START====================quantize_to_cpu_runnable_fx
2026-06-19 04:07 - INFO - epu - fx -
Source name Op Output 0 Range Output 0 Frac Bits
------------------------------------------------------------- ----------------------------- ---------------------- --------------------
/model/layers.2/input_layernorm/Mul_1 contrib.epu.rms_norm [-0.862898f, 1.48399f] 29
CustomOp/linalg::channelwiseQuantMatMul<42>2 contrib.epu.quadric_custom_op [-1.03252f, 1.31911f] 29
CustomOp/add<MultiCoreMode::All>2 contrib.epu.quadric_custom_op [-18.1085f, 57.4384f] 25
/model/layers.2/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-82.5232f, 81.3135f] 24
CustomOp/nn::qwen3PrefillGateUpProjection<27,27,24,24,35,35>0 contrib.epu.quadric_custom_op [-26.306f, 24.5606f] 26
CustomOp/linalg::channelwiseQuantMatMul<37>1 contrib.epu.quadric_custom_op [-10.8904f, 26.5761f] 26
CustomOp/add<MultiCoreMode::All>1 contrib.epu.quadric_custom_op [-23.2706f, 81.9635f] 24
2026-06-19 04:07 - INFO - epu - codegen - START====================build_cpu_runnable_fx_relay
2026-06-19 04:07 - INFO - epu - codegen - START=======================quantize_to_chimera_fx
2026-06-19 04:07 - INFO - epu - codegen - START=================================relay_to_tir
2026-06-19 04:07 - INFO - epu - codegen - START===========================relay_to_epu_relay
2026-06-19 04:07 - INFO - epu - codegen - START==============================adapt_and_order
2026-06-19 04:07 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-06-19 04:07 - INFO - epu - codegen - START=============================plan_lrm_virtual
2026-06-19 04:07 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-06-19 04:07 - INFO - epu - codegen - START===============================lrm_alloc_loop
2026-06-19 04:08 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-06-19 04:08 - INFO - epu - codegen - START================================lrm_splitting
2026-06-19 04:08 - INFO - epu - codegen - START==============================ext_split_relay
2026-06-19 04:08 - INFO - epu - codegen - START====================================build_tir
2026-06-19 04:08 - INFO - epu - chimera_job - Compilation of custom_op_decoder_2_QC_P_1d7_2MB_4kB_128GBps_128GBps_16_OFF_x1_x4 successful
Compilation complete!
╒═════════════════════╤════════════════════════════════════════════════════════════════════════════════════════════╕
│ Module Name │ custom_op_decoder_2_QC_P_1d7_2MB_4kB_128GBps_128GBps_16_OFF_x1_x4 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ ONNX File │ /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_2/custom_op_decoder_2.onnx │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ Product Target │ QC-P │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ Number of Cores │ 4 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ ISS Clock Frequency │ 1.700 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ L2M Size │ 2MB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ LRM Size │ 4kB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ External Read BW │ 128GBps │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ External Write BW │ 128GBps │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ MACS per PE │ 16 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ Max L2M │ 0.000MB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ Max LRM │ 0.000kB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ Max Temp Ext Bytes │ 96.000MB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ Network GMACs │ 206.158 │
╘═════════════════════╧════════════════════════════════════════════════════════════════════════════════════════════╛
╒════╤════════╤════════════════════════════════╤═══════════════════╤══════════════════════════╤═══════╕
│ │ Type │ Name │ shape │ type │ mse │
╞════╪════════╪════════════════════════════════╪═══════════════════╪══════════════════════════╪═══════╡
│ 0 │ Input │ /model/layers.1/Add_1_output_0 │ [1, 1024, 4096] │ tensor[FixedPoint32<25>] │ n/a │
├────┼────────┼────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 1 │ Input │ attention_mask │ [1, 1024, 1024] │ n/a │ n/a │
├────┼────────┼────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 2 │ Input │ cos │ [1, 1024, 128] │ tensor[FixedPoint32<30>] │ n/a │
├────┼────────┼────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 3 │ Input │ sin │ [1, 1024, 128] │ tensor[FixedPoint32<30>] │ n/a │
├────┼────────┼────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 4 │ Input │ past_key_values.2.key │ [1, 8, 0, 128] │ n/a │ n/a │
├────┼────────┼────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 5 │ Input │ past_key_values.2.value │ [1, 8, 0, 128] │ n/a │ n/a │
├────┼────────┼────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 6 │ Output │ /model/layers.2/Add_1_output_0 │ [1, 1024, 4096] │ tensor[FixedPoint32<24>] │ n/a │
├────┼────────┼────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 7 │ Output │ present.2.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├────┼────────┼────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 8 │ Output │ present.2.value │ [1, 8, 1024, 128] │ n/a │ n/a │
╘════╧════════╧════════════════════════════════╧═══════════════════╧══════════════════════════╧═══════╛
✓ CGC compile complete
── Step 4: Generate decoder_config.hpp + runner symlink ──
✓ Config + symlink ready → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_2/qwen3_prefill_runner.cpp
── Step 5: SDK compile & ISS run ──
$ sdk source --quiet /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_2/qwen3_prefill_runner.cpp --include-cgc-headers --target QC-P --num-cores 4 --ocm-size 2MB --macs-per-pe 16 --clock-freq-ghz 1 --ext-read-bw 24GBps --ext-write-bw 24GBps --ddr-axi-width 256
✓ SDK compile complete → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_2/epu_outputs/output_isolated.bin
── Step 6: Validate against ONNX Runtime ──
auto-atol: p_flip=0.185% auto=0.2240 base=0.6067 -> atol=0.6067
GPNPU vs ORT: max|d|=0.9108 mean|d|=0.0115 | atol=0.607 rtol=0.01 fail=0/4194304 (0.000%) [PASS]
✓ Validation complete
── Step 7: Plot output distributions ──
GPNPU min=-20.0026 max=62.1398 std=0.3093
ORT min=-20.0026 max=62.1398 std=0.3092
Scale factor (GPNPU std / ORT std): 1.00x
Pearson r (raw): 0.9986
Pearson r (scaled): 0.9986
Saved → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_2/output_comparison.png
✓ Plot saved → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_2/output_comparison.png
── Step 8: Update decoder report ──
Updated /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_report.md
decoder 2: in_fb=25, out_fb=24, mean_diff=0.0115, max_diff=0.9108, combined%=0.000%, result=PASS, scale=1.000x, r=0.9986
✓ Report updated → decoder_report.md
── Step 9: Chain output to decoder 3 (mode: isolated) ──
✓ Chained ORT 2 → 3 (FixedPoint32<24>, 16.0 MB)
✓ Chained: /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_2/onnx_runtime_outputs/output_ort.npy → decoder_3/input_logits_ort.bin (FixedPoint32<24>)
mean_diff : 0.0115
max_diff : 0.9108
n_fail : 0 (0.000%)
pearson_r : 0.9986
result : PASS
── Profile Summary (from iss_run_log.txt) ──
Regions × Cores : 8 × 4
Total cycles : 87,002,877 (Σ regions, max across cores)
MAC cycles : 2,786,456 (3.2% of total)
7. Demo: Decoder 35 — Isolated Mode (lm_head exit)
Decoder 35 is the structural boundary at the end of the prefill chain. Unlike the 34 identical middle decoders that pass a hidden-state tensor forward, decoder 35 carries the final hidden state through the lm_head projection to produce vocab-dim logits.
| Parameter | Value |
|---|---|
| Input frac bits | derived from decoder 34's output via .tranges |
| Output frac bits | derived from .tranges |
| Input | decoder_35/input_logits_ort.bin — ORT output of decoder 34 |
| GPNPU output | decoder_35/epu_outputs/output_isolated.bin |
| ORT reference | decoder_35/onnx_runtime_outputs/output_ort.npy |
run_pipeline(
decoder=35,
mode="isolated",
skip_cut=True,
tranges="qwen_0_36_calibrated.onnx.tranges",
rtol=1e-2,
max_failures=50,
)
=== Decoder 35 pipeline (mode=isolated, cores=4) ===
── Step 1: Skipping cut (--skip-cut) ──
── Step 2: Custom op matching ──
Warning: /model/layers.35/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Processing matmul ops for decoder 35...
Decoder 35 gate_up frac bits:
gate_combined_scale=1.067153e-01 frac_bits=34
up_combined_scale=1.776727e-01 frac_bits=33
gate_im_frac_bits=25
up_im_frac_bits=25
gate_output_frac_bits=24
up_output_frac_bits=23
Successfully saved modified model to /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_35/custom_op_decoder_35.onnx
✓ Custom op complete → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_35/custom_op_decoder_35.onnx
── Step 3: CGC compile ──
Starting CGC compilation...
2026-06-19 04:35 - INFO - epu - chimera_job - START==================================onnx_ingest
2026-06-19 04:35 - INFO - epu - chimera_job - Numerical ranges provided
2026-06-19 04:35 - INFO - epu - codegen - START===============================optimize_relay
2026-06-19 04:35 - INFO - epu - codegen - START====================quantize_to_cpu_runnable_fx
2026-06-19 04:35 - INFO - epu - fx -
Source name Op Output 0 Range Output 0 Frac Bits
------------------------------------------------------------- ----------------------------- --------------------- --------------------
/model/layers.35/input_layernorm/Mul_1 contrib.epu.rms_norm [-298.728f, 121.485f] 22
CustomOp/linalg::channelwiseQuantMatMul<36>2 contrib.epu.quadric_custom_op [-282.141f, 824.661f] 21
CustomOp/add<MultiCoreMode::All>2 contrib.epu.quadric_custom_op [-7101.39f, 5072.41f] 18
/model/layers.35/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-601.076f, 47.2325f] 21
CustomOp/nn::qwen3PrefillGateUpProjection<25,25,24,23,34,33>0 contrib.epu.quadric_custom_op [-1002.33f, 1755.35f] 20
CustomOp/linalg::channelwiseQuantMatMul<29>1 contrib.epu.quadric_custom_op [-4812.36f, 1990.75f] 18
CustomOp/add<MultiCoreMode::All>1 contrib.epu.quadric_custom_op [-6990.34f, 2581.91f] 18
2026-06-19 04:35 - INFO - epu - codegen - START====================build_cpu_runnable_fx_relay
2026-06-19 04:35 - INFO - epu - codegen - START=======================quantize_to_chimera_fx
2026-06-19 04:35 - INFO - epu - codegen - START=================================relay_to_tir
2026-06-19 04:35 - INFO - epu - codegen - START===========================relay_to_epu_relay
2026-06-19 04:35 - INFO - epu - codegen - START==============================adapt_and_order
2026-06-19 04:35 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-06-19 04:35 - INFO - epu - codegen - START=============================plan_lrm_virtual
2026-06-19 04:35 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-06-19 04:35 - INFO - epu - codegen - START===============================lrm_alloc_loop
2026-06-19 04:35 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-06-19 04:35 - INFO - epu - codegen - START================================lrm_splitting
2026-06-19 04:35 - INFO - epu - codegen - START==============================ext_split_relay
2026-06-19 04:35 - INFO - epu - codegen - START====================================build_tir
2026-06-19 04:35 - INFO - epu - chimera_job - Compilation of custom_op_decoder_35_QC_P_1d7_2MB_4kB_128GBps_128GBps_16_OFF_x1_x4 successful
Compilation complete!
╒═════════════════════╤══════════════════════════════════════════════════════════════════════════════════════════════╕
│ Module Name │ custom_op_decoder_35_QC_P_1d7_2MB_4kB_128GBps_128GBps_16_OFF_x1_x4 │
├─────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────┤
│ ONNX File │ /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_35/custom_op_decoder_35.onnx │
├─────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────┤
│ Product Target │ QC-P │
├─────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────┤
│ Number of Cores │ 4 │
├─────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────┤
│ ISS Clock Frequency │ 1.700 │
├─────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────┤
│ L2M Size │ 2MB │
├─────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────┤
│ LRM Size │ 4kB │
├─────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────┤
│ External Read BW │ 128GBps │
├─────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────┤
│ External Write BW │ 128GBps │
├─────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────┤
│ MACS per PE │ 16 │
├─────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────┤
│ Max L2M │ 0.000MB │
├─────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────┤
│ Max LRM │ 0.000kB │
├─────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────┤
│ Max Temp Ext Bytes │ 96.000MB │
├─────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────┤
│ Network GMACs │ 206.158 │
╘═════════════════════╧══════════════════════════════════════════════════════════════════════════════════════════════╛
╒════╤════════╤═════════════════════════════════╤═══════════════════╤══════════════════════════╤═══════╕
│ │ Type │ Name │ shape │ type │ mse │
╞════╪════════╪═════════════════════════════════╪═══════════════════╪══════════════════════════╪═══════╡
│ 0 │ Input │ /model/layers.34/Add_1_output_0 │ [1, 1024, 4096] │ tensor[FixedPoint32<18>] │ n/a │
├────┼────────┼─────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 1 │ Input │ attention_mask │ [1, 1024, 1024] │ n/a │ n/a │
├────┼────────┼─────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 2 │ Input │ cos │ [1, 1024, 128] │ tensor[FixedPoint32<30>] │ n/a │
├────┼────────┼─────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 3 │ Input │ sin │ [1, 1024, 128] │ tensor[FixedPoint32<30>] │ n/a │
├────┼────────┼─────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 4 │ Input │ past_key_values.35.key │ [1, 8, 0, 128] │ n/a │ n/a │
├────┼────────┼─────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 5 │ Input │ past_key_values.35.value │ [1, 8, 0, 128] │ n/a │ n/a │
├────┼────────┼─────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 6 │ Output │ /model/layers.35/Add_1_output_0 │ [1, 1024, 4096] │ tensor[FixedPoint32<18>] │ n/a │
├────┼────────┼─────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 7 │ Output │ present.35.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├────┼────────┼─────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 8 │ Output │ present.35.value │ [1, 8, 1024, 128] │ n/a │ n/a │
╘════╧════════╧═════════════════════════════════╧═══════════════════╧══════════════════════════╧═══════╛
✓ CGC compile complete
── Step 4: Generate decoder_config.hpp + runner symlink ──
✓ Config + symlink ready → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_35/qwen3_prefill_runner.cpp
── Step 5: SDK compile & ISS run ──
$ sdk source --quiet /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_35/qwen3_prefill_runner.cpp --include-cgc-headers --target QC-P --num-cores 4 --ocm-size 2MB --macs-per-pe 16 --clock-freq-ghz 1 --ext-read-bw 24GBps --ext-write-bw 24GBps --ddr-axi-width 256
✓ SDK compile complete → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_35/epu_outputs/output_isolated.bin
── Step 6: Validate against ONNX Runtime ──
auto-atol: p_flip=0.076% auto=16.9461 base=2.3025 -> atol=16.9461
GPNPU vs ORT: max|d|=55.7588 mean|d|=0.5098 | atol=16.9 rtol=0.01 fail=43/4194304 (0.001%) [PASS]
✓ Validation complete
── Step 7: Plot output distributions ──
GPNPU min=-834.5496 max=1859.3101 std=26.7158
ORT min=-834.5496 max=1861.3665 std=26.7205
Scale factor (GPNPU std / ORT std): 1.00x
Pearson r (raw): 0.9995
Pearson r (scaled): 0.9995
Saved → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_35/output_comparison.png
✓ Plot saved → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_35/output_comparison.png
── Step 8: Update decoder report ──
Updated /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_report.md
decoder 35: in_fb=18, out_fb=18, mean_diff=0.5098, max_diff=55.7588, combined%=0.001%, result=PASS, scale=1.000x, r=0.9995
✓ Report updated → decoder_report.md
── Step 9: Chain output to decoder 36 (mode: isolated) ──
✓ Chained ORT 35 → 36 (FixedPoint32<18>, 16.0 MB)
✓ Chained: /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_35/onnx_runtime_outputs/output_ort.npy → decoder_36/input_logits_ort.bin (FixedPoint32<18>)
mean_diff : 0.5098
max_diff : 55.7588
n_fail : 43 (0.001%)
pearson_r : 0.9995
result : PASS
── Profile Summary (from iss_run_log.txt) ──
Regions × Cores : 8 × 4
Total cycles : 86,881,457 (Σ regions, max across cores)
MAC cycles : 2,786,456 (3.2% of total)
Decoder 2 vs Decoder 35 -- key observations
| Metric | Decoder 2 (fp24, mid-layer) | Decoder 35 (fp18, lm_head exit) |
|---|---|---|
| Output range | ~[-20.0, 62.1] | ~[-834.5, 1719.0] |
| Mean |diff| | ~0.012 | ~0.527 |
| Max |diff| | ~0.854 | ~43.25 |
| Combined failures | 0 | 30 (of 4.19 M, within max_failures=50 budget) |
| Pearson r | ~0.9986 | ~0.9995 |
| auto-atol | ~0.61 | ~17.19 |
| Verdict | PASS | PASS |
Decoder 2 is a representative middle decoder: a single worst element near the dynamic-range boundary in fp24 produces max|diff| ~0.854, but only 3 elements exceed 0.5 across all 4.19 M outputs, comfortably inside the combined atol + rtol*|b| gate.
Decoder 35 is the structural endpoint. After the lm_head matmul the output range expands ~40x (max ~1719 vs ~62 for dec 2) and per-output-channel scales grow accordingly, so the auto-atol formula widens the threshold to ~17.19. Even at that scale ~30 tail elements still exceed it and max|diff| reaches 43.25 -- but 30 / 4.19 M is below the max_failures=50 budget, so the run passes. Pearson r remains ~0.9995, so the bulk of the tensor still matches ORT element-wise; the residual is a small population of large-magnitude outliers in the vocab-dim projection, not distributed noise.
8. Chained Mode and Cosine Similarity
Why chained mode
Isolated mode (sections 6–7) validates each decoder against ORT using its ideal input (the float ORT output of decoder N−1, dequantized into FixedPoint32 for the GPNPU). This proves kernel correctness — given a perfect input, does the GPNPU compute the same result as ORT?
But in production, the GPNPU runs the full 36-decoder chain end-to-end: each decoder receives the previous decoder's GPNPU output as input, not ORT's. Small per-decoder quantization noise accumulates across depth. Chained mode measures that accumulation by feeding decoder N−1's actual GPNPU output (epu_outputs/output_chained.bin) into decoder N's input.
Why cosine instead of atol
The atol-based gate that works for isolated mode does not work cleanly for chained mode at deep layers:
- Output magnitudes change ~50× across depth (token norms grow from ~15 at decoder 1 to ~3200 at decoder 35). A fixed atol that fits decoder 1 is meaningless at decoder 35.
- Per-decoder weight norms vary, so the formula-derived atol oscillates non-monotonically with depth even when drift accumulates monotonically — half the failures the atol gate flags are gate noise, not real drift.
- Small absolute element-level diffs ignore token identity; one rogue token causing catastrophic angular drift can sit invisibly inside an atol-pass.
Cosine similarity is scale-invariant and respects token structure. Each of the 1024 token vectors (each 4096-dim) gets its own cosine versus the isolated baseline, giving a direct geometric reading of "did each token's representation rotate?"
The two cosine metrics
The chained-mode gate uses two complementary statistics:
cos_p1— 1st percentile of per-token cosine. Captures bulk drift: ~10 worst tokens of the 1024 are below this value. A lowcos_p1means broad damage across many tokens.cos_min— single worst token of the 1024. Captures catastrophic outliers: one token gone almost-orthogonal to its isolated counterpart.
Reading guide (cosine → angle → meaning):
| cosine | angle | interpretation |
|---|---|---|
| 1.000 | 0° | perfect alignment |
| 0.99 | 8° | minor noise |
| 0.95 | 18° | clear drift |
| 0.80 | 37° | substantial degradation |
| 0.50 | 60° | semantic identity at risk |
| 0.30 | 73° | nearly unrelated direction |
PASS / WARN / FAIL gate
The gate (calibrated across decoders 0–35 of Qwen3-8B) is:
PASS: cos_p1 ≥ 0.60 AND cos_min ≥ 0.40
WARN: cos_p1 ≥ 0.40 AND cos_min ≥ 0.20
FAIL: anything below
The threshold pair is intentionally permissive: 36-layer chain drift through int8 quantization legitimately rotates token vectors by 30–60°. The gate only flags qualitative breaks (single-token catastrophe or broad bulk damage), not the natural drift envelope.
How chaining works in the pipeline
The chained gate compares output_chained.bin (the chained GPNPU run output) against output_isolated.bin (the per-decoder isolated baseline) via cosine. So the only thing chained mode needs from a prior isolated run is the reference baseline file — not an atol cache. atol is computed inline by the isolated validation pass from p_flip + down_proj weight stats; it is no longer persisted to disk because chained mode does not consume it.
For each decoder N:
- Isolated run writes
decoder_N/epu_outputs/output_isolated.bin(the per-decoder reference baseline) and validates against ORT using the auto-atol gate. - Chained run reads
decoder_N/input_logits_epu.bin(decoder N−1's chained output, copied forward by step 9), producesdecoder_N/epu_outputs/output_chained.bin, compares it tooutput_isolated.binvia cosine, and chains its output intodecoder_(N+1)/input_logits_epu.bin.
Decoder 0 is the boundary case: there is no previous decoder, so its chained input is the tokenized embedding (input_logits.bin) produced earlier in this notebook by the tokenization step. Decoder 0 chained vs decoder 0 isolated reads the same data, so the cosine should be ~1.0 — drift only starts accumulating from decoder 1 onward.
The chained report (decoder_report_chained.md) records cos_global, cos_p1, cos_min, and the PASS/WARN/FAIL verdict per decoder.
Demo: chain decoder 0 → decoder 1
The cell below first runs decoders 0 and 1 in isolated mode to produce the per-decoder output_isolated.bin files. These serve as the chained gate's reference baselines.
Then the chained cell uses decoder 0's isolated GPNPU output as decoder 1's chained input (since for decoder 0 the chained and isolated runs receive identical inputs from tokenization, their outputs are equivalent — so we can treat decoder 0's output_isolated.bin as the GPNPU-chain feed into decoder 1). Decoder 1 chained then runs, comparing its output against its own isolated baseline via cosine.
## Isolated baselines for decoder 0 and decoder 1.
## Each produces output_isolated.bin in its epu_outputs/ dir, which the chained
## gate compares against via cosine. Decoder 0's isolated output also serves as
## the input that would feed decoder 1's chained run (next cell).
for decoder_num in (0, 1):
run_pipeline(
decoder=decoder_num,
mode="isolated",
tranges="qwen_0_36_calibrated.onnx.tranges",
rtol=1e-2,
max_failures=50,
skip_cut=True,
)
=== Decoder 0 pipeline (mode=isolated, cores=4) ===
── Step 1: Skipping cut (--skip-cut) ──
── Step 2: Custom op matching ──
Warning: /model/layers.0/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Processing matmul ops for decoder 0...
Decoder 0 gate_up frac bits:
gate_combined_scale=5.489234e-04 frac_bits=41
up_combined_scale=3.307317e-04 frac_bits=42
gate_im_frac_bits=31
up_im_frac_bits=31
gate_output_frac_bits=28
up_output_frac_bits=28
Successfully saved modified model to /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_0/custom_op_decoder_0.onnx
✓ Custom op complete → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_0/custom_op_decoder_0.onnx
── Step 3: CGC compile ──
Starting CGC compilation...
2026-06-19 04:46 - INFO - epu - chimera_job - START==================================onnx_ingest
2026-06-19 04:46 - INFO - epu - chimera_job - Numerical ranges provided
/usr/local/lib/python3.10/dist-packages/tvm/relay/frontend/onnx.py:6270: UserWarning: This protobuf of onnx model is too large (>2GB). Call check_model with model path instead.
warnings.warn(str(e))
2026-06-19 04:47 - INFO - epu - codegen - START===============================optimize_relay
2026-06-19 04:47 - INFO - epu - codegen - START====================quantize_to_cpu_runnable_fx
2026-06-19 04:47 - INFO - epu - fx -
Source name Op Output 0 Range Output 0 Frac Bits
------------------------------------------------------------- ----------------------------- ---------------------- --------------------
/model/embed_tokens/Gather contrib.epu.embedding [-0.628906f, 0.8125f] 31
/model/layers.0/input_layernorm/Mul_1 contrib.epu.rms_norm [-0.359118f, 0.37561f] 31
CustomOp/linalg::channelwiseQuantMatMul<43>2 contrib.epu.quadric_custom_op [-2.25614f, 3.85482f] 28
CustomOp/add<MultiCoreMode::All>2 contrib.epu.quadric_custom_op [-2.30253f, 4.25466f] 28
/model/layers.0/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-1.23024f, 1.84195f] 29
CustomOp/nn::qwen3PrefillGateUpProjection<31,31,28,28,41,42>0 contrib.epu.quadric_custom_op [-11.3828f, 9.95375f] 27
CustomOp/linalg::channelwiseQuantMatMul<38>1 contrib.epu.quadric_custom_op [-5.18762f, 16.7897f] 26
CustomOp/add<MultiCoreMode::All>1 contrib.epu.quadric_custom_op [-6.67269f, 19.2419f] 26
2026-06-19 04:47 - INFO - epu - codegen - START====================build_cpu_runnable_fx_relay
2026-06-19 04:47 - INFO - epu - codegen - START=======================quantize_to_chimera_fx
2026-06-19 04:47 - INFO - epu - codegen - START=================================relay_to_tir
2026-06-19 04:47 - INFO - epu - codegen - START===========================relay_to_epu_relay
2026-06-19 04:47 - INFO - epu - codegen - START==============================adapt_and_order
2026-06-19 04:47 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-06-19 04:47 - INFO - epu - codegen - START=============================plan_lrm_virtual
2026-06-19 04:47 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-06-19 04:47 - INFO - epu - codegen - START===============================lrm_alloc_loop
2026-06-19 04:47 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-06-19 04:47 - INFO - epu - codegen - START================================lrm_splitting
2026-06-19 04:47 - INFO - epu - codegen - START==============================ext_split_relay
2026-06-19 04:47 - INFO - epu - codegen - START====================================build_tir
2026-06-19 04:47 - INFO - epu - chimera_job - Compilation of custom_op_decoder_0_QC_P_1d7_2MB_4kB_128GBps_128GBps_16_OFF_x1_x4 successful
Compilation complete!
╒═════════════════════╤════════════════════════════════════════════════════════════════════════════════════════════╕
│ Module Name │ custom_op_decoder_0_QC_P_1d7_2MB_4kB_128GBps_128GBps_16_OFF_x1_x4 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ ONNX File │ /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_0/custom_op_decoder_0.onnx │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ Product Target │ QC-P │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ Number of Cores │ 4 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ ISS Clock Frequency │ 1.700 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ L2M Size │ 2MB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ LRM Size │ 4kB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ External Read BW │ 128GBps │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ External Write BW │ 128GBps │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ MACS per PE │ 16 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ Max L2M │ 0.000MB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ Max LRM │ 0.000kB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ Max Temp Ext Bytes │ 112.000MB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ Network GMACs │ 206.158 │
╘═════════════════════╧════════════════════════════════════════════════════════════════════════════════════════════╛
╒════╤════════╤════════════════════════════════╤═══════════════════╤══════════════════════════╤═══════╕
│ │ Type │ Name │ shape │ type │ mse │
╞════╪════════╪════════════════════════════════╪═══════════════════╪══════════════════════════╪═══════╡
│ 0 │ Input │ input_ids │ [1, 1024] │ tensor[int32] │ n/a │
├────┼────────┼────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 1 │ Input │ attention_mask │ [1, 1024, 1024] │ n/a │ n/a │
├────┼────────┼────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 2 │ Input │ cos │ [1, 1024, 128] │ tensor[FixedPoint32<30>] │ n/a │
├────┼────────┼────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 3 │ Input │ sin │ [1, 1024, 128] │ tensor[FixedPoint32<30>] │ n/a │
├────┼────────┼────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 4 │ Input │ past_key_values.0.key │ [1, 8, 0, 128] │ n/a │ n/a │
├────┼────────┼────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 5 │ Input │ past_key_values.0.value │ [1, 8, 0, 128] │ n/a │ n/a │
├────┼────────┼────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 6 │ Output │ /model/layers.0/Add_1_output_0 │ [1, 1024, 4096] │ tensor[FixedPoint32<26>] │ n/a │
├────┼────────┼────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 7 │ Output │ present.0.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├────┼────────┼────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 8 │ Output │ present.0.value │ [1, 8, 1024, 128] │ n/a │ n/a │
╘════╧════════╧════════════════════════════════╧═══════════════════╧══════════════════════════╧═══════╛
✓ CGC compile complete
── Step 4: Generate decoder_config.hpp + runner symlink ──
✓ Config + symlink ready → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_0/qwen3_prefill_runner.cpp
── Step 5: SDK compile & ISS run ──
$ sdk source --quiet /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_0/qwen3_prefill_runner.cpp --include-cgc-headers --target QC-P --num-cores 4 --ocm-size 2MB --macs-per-pe 16 --clock-freq-ghz 1 --ext-read-bw 24GBps --ext-write-bw 24GBps --ddr-axi-width 256
✓ SDK compile complete → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_0/epu_outputs/output_isolated.bin
── Step 6: Validate against ONNX Runtime ──
auto-atol: p_flip=0.205% auto=0.1560 base=0.7000 -> atol=0.7000
GPNPU vs ORT: max|d|=0.1581 mean|d|=0.0041 | atol=0.7 rtol=0.01 fail=0/4194304 (0.000%) [PASS]
✓ Validation complete
── Step 7: Plot output distributions ──
GPNPU min=-4.7409 max=12.8273 std=0.1892
ORT min=-4.7386 max=12.8358 std=0.1892
Scale factor (GPNPU std / ORT std): 1.00x
Pearson r (raw): 0.9992
Pearson r (scaled): 0.9992
Saved → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_0/output_comparison.png
✓ Plot saved → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_0/output_comparison.png
── Step 8: Update decoder report ──
Updated /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_report.md
decoder 0: in_fb=None, out_fb=26, mean_diff=0.0041, max_diff=0.1581, combined%=0.000%, result=PASS, scale=1.000x, r=0.9992
✓ Report updated → decoder_report.md
── Step 9: Chain output to decoder 1 (mode: isolated) ──
✓ Chained ORT 0 → 1 (FixedPoint32<26>, 16.0 MB)
✓ Chained: /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_0/onnx_runtime_outputs/output_ort.npy → decoder_1/input_logits_ort.bin (FixedPoint32<26>)
mean_diff : 0.0041
max_diff : 0.1581
n_fail : 0 (0.000%)
pearson_r : 0.9992
result : PASS
── Profile Summary (from iss_run_log.txt) ──
Regions × Cores : 9 × 4
Total cycles : 87,574,803 (Σ regions, max across cores)
MAC cycles : 2,786,456 (3.2% of total)
=== Decoder 1 pipeline (mode=isolated, cores=4) ===
── Step 1: Skipping cut (--skip-cut) ──
── Step 2: Custom op matching ──
Warning: /model/layers.1/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Processing matmul ops for decoder 1...
Decoder 1 gate_up frac bits:
gate_combined_scale=7.408716e-03 frac_bits=38
up_combined_scale=3.896641e-03 frac_bits=39
gate_im_frac_bits=28
up_im_frac_bits=29
gate_output_frac_bits=27
up_output_frac_bits=26
Successfully saved modified model to /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_1/custom_op_decoder_1.onnx
✓ Custom op complete → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_1/custom_op_decoder_1.onnx
── Step 3: CGC compile ──
Starting CGC compilation...
2026-06-19 04:58 - INFO - epu - chimera_job - START==================================onnx_ingest
2026-06-19 04:58 - INFO - epu - chimera_job - Numerical ranges provided
2026-06-19 04:58 - INFO - epu - codegen - START===============================optimize_relay
2026-06-19 04:58 - INFO - epu - codegen - START====================quantize_to_cpu_runnable_fx
2026-06-19 04:58 - INFO - epu - fx -
Source name Op Output 0 Range Output 0 Frac Bits
------------------------------------------------------------- ----------------------------- --------------------- --------------------
/model/layers.1/input_layernorm/Mul_1 contrib.epu.rms_norm [-1.30106f, 1.14032f] 30
CustomOp/linalg::channelwiseQuantMatMul<43>2 contrib.epu.quadric_custom_op [-1.41003f, 1.53298f] 29
CustomOp/add<MultiCoreMode::All>2 contrib.epu.quadric_custom_op [-6.78994f, 19.2634f] 26
/model/layers.1/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-51.933f, 47.6116f] 25
CustomOp/nn::qwen3PrefillGateUpProjection<28,29,27,26,38,39>0 contrib.epu.quadric_custom_op [-22.4947f, 19.6069f] 26
CustomOp/linalg::channelwiseQuantMatMul<37>1 contrib.epu.quadric_custom_op [-12.4679f, 38.1379f] 25
CustomOp/add<MultiCoreMode::All>1 contrib.epu.quadric_custom_op [-18.1108f, 57.4013f] 25
2026-06-19 04:58 - INFO - epu - codegen - START====================build_cpu_runnable_fx_relay
2026-06-19 04:58 - INFO - epu - codegen - START=======================quantize_to_chimera_fx
2026-06-19 04:58 - INFO - epu - codegen - START=================================relay_to_tir
2026-06-19 04:58 - INFO - epu - codegen - START===========================relay_to_epu_relay
2026-06-19 04:58 - INFO - epu - codegen - START==============================adapt_and_order
2026-06-19 04:58 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-06-19 04:58 - INFO - epu - codegen - START=============================plan_lrm_virtual
2026-06-19 04:58 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-06-19 04:58 - INFO - epu - codegen - START===============================lrm_alloc_loop
2026-06-19 04:58 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-06-19 04:58 - INFO - epu - codegen - START================================lrm_splitting
2026-06-19 04:58 - INFO - epu - codegen - START==============================ext_split_relay
2026-06-19 04:58 - INFO - epu - codegen - START====================================build_tir
2026-06-19 04:58 - INFO - epu - chimera_job - Compilation of custom_op_decoder_1_QC_P_1d7_2MB_4kB_128GBps_128GBps_16_OFF_x1_x4 successful
Compilation complete!
╒═════════════════════╤════════════════════════════════════════════════════════════════════════════════════════════╕
│ Module Name │ custom_op_decoder_1_QC_P_1d7_2MB_4kB_128GBps_128GBps_16_OFF_x1_x4 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ ONNX File │ /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_1/custom_op_decoder_1.onnx │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ Product Target │ QC-P │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ Number of Cores │ 4 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ ISS Clock Frequency │ 1.700 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ L2M Size │ 2MB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ LRM Size │ 4kB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ External Read BW │ 128GBps │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ External Write BW │ 128GBps │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ MACS per PE │ 16 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ Max L2M │ 0.000MB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ Max LRM │ 0.000kB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ Max Temp Ext Bytes │ 96.000MB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ Network GMACs │ 206.158 │
╘═════════════════════╧════════════════════════════════════════════════════════════════════════════════════════════╛
╒════╤════════╤════════════════════════════════╤═══════════════════╤══════════════════════════╤═══════╕
│ │ Type │ Name │ shape │ type │ mse │
╞════╪════════╪════════════════════════════════╪═══════════════════╪══════════════════════════╪═══════╡
│ 0 │ Input │ /model/layers.0/Add_1_output_0 │ [1, 1024, 4096] │ tensor[FixedPoint32<26>] │ n/a │
├────┼────────┼────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 1 │ Input │ attention_mask │ [1, 1024, 1024] │ n/a │ n/a │
├────┼────────┼────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 2 │ Input │ cos │ [1, 1024, 128] │ tensor[FixedPoint32<30>] │ n/a │
├────┼────────┼────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 3 │ Input │ sin │ [1, 1024, 128] │ tensor[FixedPoint32<30>] │ n/a │
├────┼────────┼────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 4 │ Input │ past_key_values.1.key │ [1, 8, 0, 128] │ n/a │ n/a │
├────┼────────┼────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 5 │ Input │ past_key_values.1.value │ [1, 8, 0, 128] │ n/a │ n/a │
├────┼────────┼────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 6 │ Output │ /model/layers.1/Add_1_output_0 │ [1, 1024, 4096] │ tensor[FixedPoint32<25>] │ n/a │
├────┼────────┼────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 7 │ Output │ present.1.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├────┼────────┼────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 8 │ Output │ present.1.value │ [1, 8, 1024, 128] │ n/a │ n/a │
╘════╧════════╧════════════════════════════════╧═══════════════════╧══════════════════════════╧═══════╛
✓ CGC compile complete
── Step 4: Generate decoder_config.hpp + runner symlink ──
✓ Config + symlink ready → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_1/qwen3_prefill_runner.cpp
── Step 5: SDK compile & ISS run ──
$ sdk source --quiet /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_1/qwen3_prefill_runner.cpp --include-cgc-headers --target QC-P --num-cores 4 --ocm-size 2MB --macs-per-pe 16 --clock-freq-ghz 1 --ext-read-bw 24GBps --ext-write-bw 24GBps --ddr-axi-width 256
✓ SDK compile complete → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_1/epu_outputs/output_isolated.bin
── Step 6: Validate against ONNX Runtime ──
auto-atol: p_flip=0.040% auto=0.1176 base=0.7598 -> atol=0.7598
GPNPU vs ORT: max|d|=0.3627 mean|d|=0.0050 | atol=0.76 rtol=0.01 fail=0/4194304 (0.000%) [PASS]
✓ Validation complete
── Step 7: Plot output distributions ──
GPNPU min=-15.3526 max=45.0019 std=0.2524
ORT min=-15.3526 max=45.0019 std=0.2524
Scale factor (GPNPU std / ORT std): 1.00x
Pearson r (raw): 0.9995
Pearson r (scaled): 0.9995
Saved → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_1/output_comparison.png
✓ Plot saved → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_1/output_comparison.png
── Step 8: Update decoder report ──
Updated /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_report.md
decoder 1: in_fb=26, out_fb=25, mean_diff=0.0050, max_diff=0.3627, combined%=0.000%, result=PASS, scale=1.000x, r=0.9995
✓ Report updated → decoder_report.md
── Step 9: Chain output to decoder 2 (mode: isolated) ──
✓ Chained ORT 1 → 2 (FixedPoint32<25>, 16.0 MB)
✓ Chained: /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_1/onnx_runtime_outputs/output_ort.npy → decoder_2/input_logits_ort.bin (FixedPoint32<25>)
mean_diff : 0.0050
max_diff : 0.3627
n_fail : 0 (0.000%)
pearson_r : 0.9995
result : PASS
── Profile Summary (from iss_run_log.txt) ──
Regions × Cores : 8 × 4
Total cycles : 86,737,403 (Σ regions, max across cores)
MAC cycles : 2,786,456 (3.2% of total)
## Run decoders 0 and 1 in chained mode.
## Decoder 0's chained input is the tokenized input_logits.bin (identical to its
## isolated input). Its step 9 chains output_chained.bin into
## decoder_1/input_logits_epu.bin, which decoder 1's chained run then consumes.
## Each chained run validates against the isolated baseline from the previous
## cell using the cosine gate (PASS = cos_p1 >= 0.60 AND cos_min >= 0.40).
for decoder_num in (0, 1):
run_pipeline(
decoder=decoder_num,
mode="chained",
tranges="qwen_0_36_calibrated.onnx.tranges",
rtol=1e-2,
skip_cut=True,
)
=== Decoder 0 pipeline (mode=chained, cores=4) ===
── Step 1: Skipping cut (--skip-cut) ──
── Step 2: Custom op matching ──
Warning: /model/layers.0/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Processing matmul ops for decoder 0...
Decoder 0 gate_up frac bits:
gate_combined_scale=5.489234e-04 frac_bits=41
up_combined_scale=3.307317e-04 frac_bits=42
gate_im_frac_bits=31
up_im_frac_bits=31
gate_output_frac_bits=28
up_output_frac_bits=28
Successfully saved modified model to /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_0/custom_op_decoder_0.onnx
✓ Custom op complete → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_0/custom_op_decoder_0.onnx
── Step 3: CGC compile ──
Starting CGC compilation...
2026-06-19 05:08 - INFO - epu - chimera_job - START==================================onnx_ingest
2026-06-19 05:09 - INFO - epu - chimera_job - Numerical ranges provided
/usr/local/lib/python3.10/dist-packages/tvm/relay/frontend/onnx.py:6270: UserWarning: This protobuf of onnx model is too large (>2GB). Call check_model with model path instead.
warnings.warn(str(e))
2026-06-19 05:09 - INFO - epu - codegen - START===============================optimize_relay
2026-06-19 05:09 - INFO - epu - codegen - START====================quantize_to_cpu_runnable_fx
2026-06-19 05:09 - INFO - epu - fx -
Source name Op Output 0 Range Output 0 Frac Bits
------------------------------------------------------------- ----------------------------- ---------------------- --------------------
/model/embed_tokens/Gather contrib.epu.embedding [-0.628906f, 0.8125f] 31
/model/layers.0/input_layernorm/Mul_1 contrib.epu.rms_norm [-0.359118f, 0.37561f] 31
CustomOp/linalg::channelwiseQuantMatMul<43>2 contrib.epu.quadric_custom_op [-2.25614f, 3.85482f] 28
CustomOp/add<MultiCoreMode::All>2 contrib.epu.quadric_custom_op [-2.30253f, 4.25466f] 28
/model/layers.0/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-1.23024f, 1.84195f] 29
CustomOp/nn::qwen3PrefillGateUpProjection<31,31,28,28,41,42>0 contrib.epu.quadric_custom_op [-11.3828f, 9.95375f] 27
CustomOp/linalg::channelwiseQuantMatMul<38>1 contrib.epu.quadric_custom_op [-5.18762f, 16.7897f] 26
CustomOp/add<MultiCoreMode::All>1 contrib.epu.quadric_custom_op [-6.67269f, 19.2419f] 26
2026-06-19 05:09 - INFO - epu - codegen - START====================build_cpu_runnable_fx_relay
2026-06-19 05:09 - INFO - epu - codegen - START=======================quantize_to_chimera_fx
2026-06-19 05:09 - INFO - epu - codegen - START=================================relay_to_tir
2026-06-19 05:09 - INFO - epu - codegen - START===========================relay_to_epu_relay
2026-06-19 05:09 - INFO - epu - codegen - START==============================adapt_and_order
2026-06-19 05:09 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-06-19 05:09 - INFO - epu - codegen - START=============================plan_lrm_virtual
2026-06-19 05:09 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-06-19 05:09 - INFO - epu - codegen - START===============================lrm_alloc_loop
2026-06-19 05:09 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-06-19 05:09 - INFO - epu - codegen - START================================lrm_splitting
2026-06-19 05:09 - INFO - epu - codegen - START==============================ext_split_relay
2026-06-19 05:09 - INFO - epu - codegen - START====================================build_tir
2026-06-19 05:09 - INFO - epu - chimera_job - Compilation of custom_op_decoder_0_QC_P_1d7_2MB_4kB_128GBps_128GBps_16_OFF_x1_x4 successful
Compilation complete!
╒═════════════════════╤════════════════════════════════════════════════════════════════════════════════════════════╕
│ Module Name │ custom_op_decoder_0_QC_P_1d7_2MB_4kB_128GBps_128GBps_16_OFF_x1_x4 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ ONNX File │ /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_0/custom_op_decoder_0.onnx │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ Product Target │ QC-P │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ Number of Cores │ 4 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ ISS Clock Frequency │ 1.700 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ L2M Size │ 2MB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ LRM Size │ 4kB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ External Read BW │ 128GBps │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ External Write BW │ 128GBps │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ MACS per PE │ 16 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ Max L2M │ 0.000MB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ Max LRM │ 0.000kB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ Max Temp Ext Bytes │ 112.000MB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ Network GMACs │ 206.158 │
╘═════════════════════╧════════════════════════════════════════════════════════════════════════════════════════════╛
╒════╤════════╤════════════════════════════════╤═══════════════════╤══════════════════════════╤═══════╕
│ │ Type │ Name │ shape │ type │ mse │
╞════╪════════╪════════════════════════════════╪═══════════════════╪══════════════════════════╪═══════╡
│ 0 │ Input │ input_ids │ [1, 1024] │ tensor[int32] │ n/a │
├────┼────────┼────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 1 │ Input │ attention_mask │ [1, 1024, 1024] │ n/a │ n/a │
├────┼────────┼────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 2 │ Input │ cos │ [1, 1024, 128] │ tensor[FixedPoint32<30>] │ n/a │
├────┼────────┼────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 3 │ Input │ sin │ [1, 1024, 128] │ tensor[FixedPoint32<30>] │ n/a │
├────┼────────┼────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 4 │ Input │ past_key_values.0.key │ [1, 8, 0, 128] │ n/a │ n/a │
├────┼────────┼────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 5 │ Input │ past_key_values.0.value │ [1, 8, 0, 128] │ n/a │ n/a │
├────┼────────┼────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 6 │ Output │ /model/layers.0/Add_1_output_0 │ [1, 1024, 4096] │ tensor[FixedPoint32<26>] │ n/a │
├────┼────────┼────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 7 │ Output │ present.0.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├────┼────────┼────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 8 │ Output │ present.0.value │ [1, 8, 1024, 128] │ n/a │ n/a │
╘════╧════════╧════════════════════════════════╧═══════════════════╧══════════════════════════╧═══════╛
✓ CGC compile complete
── Step 4: Generate decoder_config.hpp + runner symlink ──
✓ Config + symlink ready → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_0/qwen3_prefill_runner.cpp
── Step 5: SDK compile & ISS run ──
$ sdk source --quiet /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_0/qwen3_prefill_runner.cpp --include-cgc-headers --target QC-P --num-cores 4 --ocm-size 2MB --macs-per-pe 16 --clock-freq-ghz 1 --ext-read-bw 24GBps --ext-write-bw 24GBps --ddr-axi-width 256
✓ SDK compile complete → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_0/epu_outputs/output_chained.bin
── Step 6: Validate against ONNX Runtime ──
chained vs isolated: cos_global=1.0000 cos_p1=1.0000 cos_min=1.0000 [PASS]
✓ Validation complete
── Step 7: Plot output distributions ──
chained min=-4.7409 max=12.8273 std=0.1892
isolated min=-4.7409 max=12.8273 std=0.1892
Scale factor (chained std / isolated std): 1.00x
Pearson r (raw): 1.0000
Pearson r (scaled): 1.0000
Saved → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_0/output_comparison.png
✓ Plot saved → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_0/output_comparison.png
── Step 8: Update decoder report ──
Updated /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_report_chained.md
decoder 0: in_fb=None, out_fb=26, mean_diff=0.0000, max_diff=0.0000, cos_global=1.000000, cos_p1=1.0000, cos_min=1.0000, result=PASS, scale=1.000x, r=1.0000
✓ Report updated → decoder_report.md
── Step 9: Chain output to decoder 1 (mode: chained) ──
✓ Chained GPNPU 0 → 1 (16.0 MB)
✓ Chained: /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_0/epu_outputs/output_chained.bin → decoder_1/input_logits_epu.bin
mean_diff : 0.0000
max_diff : 0.0000
cos_p1 : 1.0000 cos_min : 1.0000
result : PASS
── Profile Summary (from iss_run_log.txt) ──
Regions × Cores : 9 × 4
Total cycles : 87,574,803 (Σ regions, max across cores)
MAC cycles : 2,786,456 (3.2% of total)
=== Decoder 1 pipeline (mode=chained, cores=4) ===
── Step 1: Skipping cut (--skip-cut) ──
── Step 2: Custom op matching ──
Warning: /model/layers.1/self_attn/o_proj/MatMul_smooth_output_QuantizeLinear_Output not found in tranges, using default frac_bits=15
Processing matmul ops for decoder 1...
Decoder 1 gate_up frac bits:
gate_combined_scale=7.408716e-03 frac_bits=38
up_combined_scale=3.896641e-03 frac_bits=39
gate_im_frac_bits=28
up_im_frac_bits=29
gate_output_frac_bits=27
up_output_frac_bits=26
Successfully saved modified model to /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_1/custom_op_decoder_1.onnx
✓ Custom op complete → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_1/custom_op_decoder_1.onnx
── Step 3: CGC compile ──
Starting CGC compilation...
2026-06-19 05:20 - INFO - epu - chimera_job - START==================================onnx_ingest
2026-06-19 05:20 - INFO - epu - chimera_job - Numerical ranges provided
2026-06-19 05:20 - INFO - epu - codegen - START===============================optimize_relay
2026-06-19 05:20 - INFO - epu - codegen - START====================quantize_to_cpu_runnable_fx
2026-06-19 05:20 - INFO - epu - fx -
Source name Op Output 0 Range Output 0 Frac Bits
------------------------------------------------------------- ----------------------------- --------------------- --------------------
/model/layers.1/input_layernorm/Mul_1 contrib.epu.rms_norm [-1.30106f, 1.14032f] 30
CustomOp/linalg::channelwiseQuantMatMul<43>2 contrib.epu.quadric_custom_op [-1.41003f, 1.53298f] 29
CustomOp/add<MultiCoreMode::All>2 contrib.epu.quadric_custom_op [-6.78994f, 19.2634f] 26
/model/layers.1/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-51.933f, 47.6116f] 25
CustomOp/nn::qwen3PrefillGateUpProjection<28,29,27,26,38,39>0 contrib.epu.quadric_custom_op [-22.4947f, 19.6069f] 26
CustomOp/linalg::channelwiseQuantMatMul<37>1 contrib.epu.quadric_custom_op [-12.4679f, 38.1379f] 25
CustomOp/add<MultiCoreMode::All>1 contrib.epu.quadric_custom_op [-18.1108f, 57.4013f] 25
2026-06-19 05:20 - INFO - epu - codegen - START====================build_cpu_runnable_fx_relay
2026-06-19 05:20 - INFO - epu - codegen - START=======================quantize_to_chimera_fx
2026-06-19 05:20 - INFO - epu - codegen - START=================================relay_to_tir
2026-06-19 05:20 - INFO - epu - codegen - START===========================relay_to_epu_relay
2026-06-19 05:20 - INFO - epu - codegen - START==============================adapt_and_order
2026-06-19 05:20 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-06-19 05:20 - INFO - epu - codegen - START=============================plan_lrm_virtual
2026-06-19 05:20 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-06-19 05:20 - INFO - epu - codegen - START===============================lrm_alloc_loop
2026-06-19 05:20 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-06-19 05:20 - INFO - epu - codegen - START================================lrm_splitting
2026-06-19 05:20 - INFO - epu - codegen - START==============================ext_split_relay
2026-06-19 05:20 - INFO - epu - codegen - START====================================build_tir
2026-06-19 05:20 - INFO - epu - chimera_job - Compilation of custom_op_decoder_1_QC_P_1d7_2MB_4kB_128GBps_128GBps_16_OFF_x1_x4 successful
Compilation complete!
╒═════════════════════╤════════════════════════════════════════════════════════════════════════════════════════════╕
│ Module Name │ custom_op_decoder_1_QC_P_1d7_2MB_4kB_128GBps_128GBps_16_OFF_x1_x4 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ ONNX File │ /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_1/custom_op_decoder_1.onnx │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ Product Target │ QC-P │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ Number of Cores │ 4 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ ISS Clock Frequency │ 1.700 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ L2M Size │ 2MB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ LRM Size │ 4kB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ External Read BW │ 128GBps │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ External Write BW │ 128GBps │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ MACS per PE │ 16 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ Max L2M │ 0.000MB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ Max LRM │ 0.000kB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ Max Temp Ext Bytes │ 96.000MB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ Network GMACs │ 206.158 │
╘═════════════════════╧════════════════════════════════════════════════════════════════════════════════════════════╛
╒════╤════════╤════════════════════════════════╤═══════════════════╤══════════════════════════╤═══════╕
│ │ Type │ Name │ shape │ type │ mse │
╞════╪════════╪════════════════════════════════╪═══════════════════╪══════════════════════════╪═══════╡
│ 0 │ Input │ /model/layers.0/Add_1_output_0 │ [1, 1024, 4096] │ tensor[FixedPoint32<26>] │ n/a │
├────┼────────┼────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 1 │ Input │ attention_mask │ [1, 1024, 1024] │ n/a │ n/a │
├────┼────────┼────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 2 │ Input │ cos │ [1, 1024, 128] │ tensor[FixedPoint32<30>] │ n/a │
├────┼────────┼────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 3 │ Input │ sin │ [1, 1024, 128] │ tensor[FixedPoint32<30>] │ n/a │
├────┼────────┼────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 4 │ Input │ past_key_values.1.key │ [1, 8, 0, 128] │ n/a │ n/a │
├────┼────────┼────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 5 │ Input │ past_key_values.1.value │ [1, 8, 0, 128] │ n/a │ n/a │
├────┼────────┼────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 6 │ Output │ /model/layers.1/Add_1_output_0 │ [1, 1024, 4096] │ tensor[FixedPoint32<25>] │ n/a │
├────┼────────┼────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 7 │ Output │ present.1.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├────┼────────┼────────────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 8 │ Output │ present.1.value │ [1, 8, 1024, 128] │ n/a │ n/a │
╘════╧════════╧════════════════════════════════╧═══════════════════╧══════════════════════════╧═══════╛
✓ CGC compile complete
── Step 4: Generate decoder_config.hpp + runner symlink ──
✓ Config + symlink ready → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_1/qwen3_prefill_runner.cpp
── Step 5: SDK compile & ISS run ──
$ sdk source --quiet /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_1/qwen3_prefill_runner.cpp --include-cgc-headers --target QC-P --num-cores 4 --ocm-size 2MB --macs-per-pe 16 --clock-freq-ghz 1 --ext-read-bw 24GBps --ext-write-bw 24GBps --ddr-axi-width 256
✓ SDK compile complete → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_1/epu_outputs/output_chained.bin
── Step 6: Validate against ONNX Runtime ──
chained vs isolated: cos_global=0.9970 cos_p1=0.9895 cos_min=0.9764 [PASS]
✓ Validation complete
── Step 7: Plot output distributions ──
chained min=-15.3984 max=45.0176 std=0.2525
isolated min=-15.3526 max=45.0019 std=0.2524
Scale factor (chained std / isolated std): 1.00x
Pearson r (raw): 0.9970
Pearson r (scaled): 0.9970
Saved → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_1/output_comparison.png
✓ Plot saved → /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_1/output_comparison.png
── Step 8: Update decoder report ──
Updated /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_report_chained.md
decoder 1: in_fb=26, out_fb=25, mean_diff=0.0142, max_diff=1.5727, cos_global=0.997043, cos_p1=0.9895, cos_min=0.9764, result=PASS, scale=1.000x, r=0.9970
✓ Report updated → decoder_report.md
── Step 9: Chain output to decoder 2 (mode: chained) ──
✓ Chained GPNPU 1 → 2 (16.0 MB)
✓ Chained: /quadric/sdk-cli/examples/models/qwen/qwen3_prefill_val/decoder_1/epu_outputs/output_chained.bin → decoder_2/input_logits_epu.bin
mean_diff : 0.0142
max_diff : 1.5727
cos_p1 : 0.9895 cos_min : 0.9764
result : PASS
── Profile Summary (from iss_run_log.txt) ──
Regions × Cores : 8 × 4
Total cycles : 86,737,403 (Σ regions, max across cores)
MAC cycles : 2,786,456 (3.2% of total)
Decoder 1 chained — results
| Metric | Value | Interpretation |
|---|---|---|
cos_global | ~0.997 | Whole-tensor cosine — almost identical orientation |
cos_p1 | ~0.989 | 1st-percentile per-token cosine; ~10 worst tokens at ~8° rotation |
cos_min | ~0.977 | Worst single token at ~12° rotation |
| Pearson r | ~0.997 | Correlation between chained and isolated outputs |
| Mean |diff| | ~0.014 | Average element-wise absolute difference |
| Max |diff| | ~1.19 | Largest single-element absolute difference |
| Scale (chained/isolated) | ~1.00× | Output magnitudes match the isolated baseline |
| Gate verdict | PASS | cos_p1 ≥ 0.60 ✓ and cos_min ≥ 0.40 ✓ |
What this means: At depth 1, the chained GPNPU output is geometrically near-identical to the isolated baseline — every token's representation remains within ~12° of its ideal direction. Drift accumulation through one decoder layer of int8 quantization is small. As depth increases, both cos_p1 and cos_min drift downward (e.g. by decoder 33 they sit at ~0.67 / 0.46), but the gate stays satisfied across decoders 0–34. Decoder 35 is the only structural break in the chain.
9. Full-Prefill Performance
The runs above measure each decoder's ISS cycle cost.
Method
prefill_cycles = dec_0_cycles + 34 * dec_mid_cycles + dec_35_cycles
^ ^
embedding entry lm_head exit
Decoder 0 (embedding) and decoder 35 (lm_head) are structurally different from the 34 identical middle decoders, so each is measured directly from its iss_run_log.txt. Decoders 1-34 share the same compiled kernel; we measure decoder 2 as a representative mid-decoder and multiply by 34.
Derived headline metrics:
prefill_latency_s = prefill_cycles / clock_hz
TTFT_ms = prefill_latency_s * 1000 # time-to-first-token
prefill_tok_per_s = seq_len / prefill_latency_s # standard model-throughput metric
Target silicon: QC-P @ 1.7 GHz, sequence length 1024.
Caveats
- This is a per-decoder isolated upper bound. The fused 36-layer compile amortizes constant-tensor loads across decoder boundaries -- real end-to-end prefill latency will be lower than this sum.
- ISS cycles are simulator estimates, not silicon-measured.
- Prefill compute is roughly O(N^2) in
seq_lenbecause of attention. These numbers are specific toseq_len = 1024; doubling the prompt length gives ~4x the latency. - Decoders 1-34 are assumed identical to decoder 2. Small per-decoder variation (region count, MAC saturation) is below ~1% in practice.
## Parse per-decoder cycles from each iss_run_log.txt and compute the prefill
## headline numbers at runtime. This cell is the single source of truth for
## the full-prefill perf story; the markdown above describes the method.
from run_decoder_pipeline import parse_iss_cycles
CLOCK_GHZ = 1.7
SEQ_LEN = 1024
N_MID = 34 # decoders 1-34 share the dec_2 kernel
perf = {n: parse_iss_cycles(f"decoder_{n}") for n in (0, 2, 35)}
missing = [n for n, p in perf.items() if p is None]
if missing:
print(f"Skipping prefill rollup: missing iss_run_log.txt for decoder(s) {missing}.")
else:
dec_0 = perf[0]["total_cycles"]
dec_mid = perf[2]["total_cycles"]
dec_35 = perf[35]["total_cycles"]
prefill_cycles = dec_0 + N_MID * dec_mid + dec_35
latency_ms = prefill_cycles / (CLOCK_GHZ * 1e9) * 1000.0
tok_per_s = SEQ_LEN / (latency_ms / 1000.0)
print(
f"Per-decoder cycles (parsed from decoder_N/qwen3_prefill_runner_QC-P_*/output/iss_run_log.txt):"
)
print()
print(f" {'Decoder':<26} {'Cycles':>14} {'Regions':>7} Note")
print(f" {'-'*26} {'-'*14} {'-'*7} {'-'*40}")
print(f" {'dec_0 (embedding entry)':<26} {dec_0:>14,} {perf[0]['n_regions']:>7} measured")
print(
f" {'dec_mid (decoder 2)':<26} {dec_mid:>14,} {perf[2]['n_regions']:>7} representative of decoders 1-34"
)
print(
f" {'dec_35 (lm_head exit)':<26} {dec_35:>14,} {perf[35]['n_regions']:>7} {dec_35/dec_mid:.2f}x dec_mid (vocab-dim matmul)"
)
print()
print(
f"Full-prefill = dec_0 + {N_MID} * dec_mid + dec_35 @ QC-P, {CLOCK_GHZ} GHz, seq_len={SEQ_LEN}"
)
print()
print(f" Total cycles : {prefill_cycles:>14,}")
print(f" Prefill latency : {latency_ms:>14,.1f} ms")
print(f" TTFT : {latency_ms:>14,.1f} ms")
print(f" Prefill throughput : {tok_per_s:>14,.0f} tokens/s <- headline")
Per-decoder cycles (parsed from decoder_N/qwen3_prefill_runner_QC-P_*/output/iss_run_log.txt):
Decoder Cycles Regions Note
-------------------------- -------------- ------- ----------------------------------------
dec_0 (embedding entry) 87,574,803 9 measured
dec_mid (decoder 2) 87,002,877 8 representative of decoders 1-34
dec_35 (lm_head exit) 86,881,457 8 1.00x dec_mid (vocab-dim matmul)
Full-prefill = dec_0 + 34 * dec_mid + dec_35 @ QC-P, 1.7 GHz, seq_len=1024
Total cycles : 3,132,554,078
Prefill latency : 1,842.7 ms
TTFT : 1,842.7 ms
Prefill throughput : 556 tokens/s <- headline
The fused 36-layer compile measurement, when it lands, should be <= this sum -- the compiler can amortize constant-tensor loads across decoder boundaries, so a single fused build is at worst as slow as 36 isolated runs and typically faster.
