NOTE: The Jupyter Notebook below is included in the Chimera SDK and can be run interactively by running the following CLI command:
$ quadric sdk notebook
From the Jupyter Notebook window in your browser, select the notebook named /quadric/sdk-cli/examples/models/qwen/qwen3_8b/qwen3.ipynb.
Compiling and Running QWEN3 8B on ISS (4-Core Multicore Configuration)
In this notebook, we demonstrate:
- Lowering ONNX to C++ using Chimera Graph Compiler (CGC)
- Compiling C++ to assembly using Quadric LLVM compiler with Chimera GPNPU backend
- Executing on Instruction Set Simulator (ISS) in 4-core multicore configuration
- Autoregressive inference achieving ~18-19 average tokens/sec, ~20-21 peak tokens/sec
Prerequisites
This notebook uses a pre-quantized QWEN3 8B model that has already been optimized with:
- W4A8 per-channel smooth quantization (alpha=0.36)
- Custom RoPE modifications for Quadric hardware
- Attention mask adjustments for prefill and autoregressive stages
Step 1: Download Pre-Quantized Model
Download the W4A8 quantized QWEN3 8B model from S3. This includes:
9b9822ba-a73f-11f0-9995-10ffe060dbe7.data- Runtime data fileqwen_0_36.onnx- The quantized ONNX modelqwen_0_36.onnx.tranges- Tensor range information for the quantized model
from urllib.request import urlretrieve
base_url = "https://sdk-cli-models.s3.amazonaws.com/"
files = [
"9b9822ba-a73f-11f0-9995-10ffe060dbe7.data",
"qwen_0_36.onnx",
"qwen_0_36.onnx.tranges",
]
for filename in files:
print(f"Downloading {filename}...")
urlretrieve(base_url + filename, filename)
print(f" ✓ Downloaded {filename}")
print("\nAll files downloaded successfully!")
Downloading 9b9822ba-a73f-11f0-9995-10ffe060dbe7.data...
✓ Downloaded 9b9822ba-a73f-11f0-9995-10ffe060dbe7.data
Downloading qwen_0_36.onnx...
✓ Downloaded qwen_0_36.onnx
Downloading qwen_0_36.onnx.tranges...
✓ Downloaded qwen_0_36.onnx.tranges
All files downloaded successfully!
Step 2: Fix Shapes for Autoregressive Execution
The downloaded ONNX model has dynamic shapes. We fix these shapes for autoregressive inference with a sequence length of 1024 tokens.
Runtime: ~30-60 seconds
from fix_shapes import fix_shapes
## Input: downloaded quantized model
in_onnx_path = "qwen_0_36.onnx"
tranges_path = "qwen_0_36.onnx.tranges"
## Output: shape-fixed model
autoregressive_onnx_path = "qwen3_seq1024.onnx"
seq_length = 1024
fix_shapes(in_onnx_path, autoregressive_onnx_path, seq_length)
Loading model from qwen_0_36.onnx...
Fixing shapes for autoregressive mode: seq_len=1024
Saving intermediate fixed model to fixed.onnx...
Running quantization pre-processing...
Shape fixing complete! Output saved to qwen3_seq1024.onnx
Step 3: Custom Op Matching
Replace standard ONNX operations with Quadric-optimized custom operations:
- QWEN3 Attention Block - Fused attention optimized for Quadric hardware
- Channelwise Quantized MatMul - Efficient INT8 matrix multiplication
- INT4 Weight Packing - Pack INT4 weights into V8I4 format for optimal memory bandwidth
Runtime: ~30-45 minutes
from custom_op_match import qwen3_custom_op_replacer
## QWEN3 8B architecture parameters
num_heads = 8
embed_dim = 4096
num_decoders = 36
custom_onnx_path = "qwen3_custom_ops_seq1024.onnx"
qwen3_custom_op_replacer(
autoregressive_onnx_path,
custom_onnx_path,
tranges_path,
num_heads=num_heads,
embed_dim=embed_dim,
seq_length=seq_length,
num_decoders=num_decoders,
)
Successfully saved modified model to qwen3_custom_ops_seq1024.onnx
Step 4: Lower ONNX to C++ with Chimera Graph Compiler (CGC)
The Chimera Graph Compiler (CGC) converts the ONNX graph into optimized C++ code for the Quadric platform. This process:
- Analyzes the computational graph and schedules operations
- Generates memory-efficient code that fits within the 2MB OCM constraint
- Optimizes for the QC-P hardware configuration (16 MACs per PE)
Hardware Configuration:
- Product: QC-P (Quadric Chimera Processor)
- Target Language: QIL (Quadric Intermediate Language)
- OCM Size: 2MB on-chip memory
- MACs per PE: 16 multiply-accumulate units per processing element
Note: This compilation requires >100GB RAM and takes ~30 minutes due to the model size (8B parameters).
import resource
## Increase stack size for large model compilation
resource.setrlimit(resource.RLIMIT_STACK, (32768 * 1024, 32768 * 1024))
from tvm.contrib.epu.chimera_job.chimera_job import ChimeraJob
from tvm.contrib.epu.chimera_job.hw_config import HWConfig
## Configure hardware target
hw_config = HWConfig(product="QC-P")
## Specify I/O tensors to ignore during compilation (KV cache management)
io_to_ignore = ["attention_mask"]
for i in range(num_decoders):
io_to_ignore.extend(
[
f"present.{i}.key",
f"present.{i}.value",
f"past_key_values.{i}.key",
f"past_key_values.{i}.value",
]
)
## Create and run CGC compilation job
cgc_job = ChimeraJob(
custom_onnx_path,
hw_config=hw_config,
trange_file=tranges_path,
target_lang="QIL", # Quadric Intermediate Language
ocm_size="2MB", # On-chip memory
macs_per_pe=16, # MACs per processing element
product="QC-P",
io_to_ignore=io_to_ignore,
)
print("Starting CGC compilation (this will take ~30 minutes)...")
cgc_job.compile()
print("\nCompilation complete!")
print(cgc_job)
/tmp/ipykernel_2643/1495230762.py:20: DeprecationWarning: Both hw_config and individual hardware parameters were provided. The hw_config will be used and individual parameters will be ignored. In future releases, specifying individual hardware parameters will be removed.
cgc_job = ChimeraJob(
Starting CGC compilation (this will take ~30 minutes)...
2026-06-19 04:31 - INFO - epu - chimera_job - START==================================onnx_ingest
2026-06-19 04:31 - INFO - epu - chimera_job - Numerical ranges provided
/usr/local/lib/python3.10/dist-packages/tvm/relay/frontend/onnx.py:6270: UserWarning: This protobuf of onnx model is too large (>2GB). Call check_model with model path instead.
warnings.warn(str(e))
2026-06-19 04:31 - INFO - epu - codegen - START===============================optimize_relay
2026-06-19 04:31 - INFO - epu - codegen - START====================quantize_to_cpu_runnable_fx
2026-06-19 04:32 - INFO - epu - fx -
Source name Op Output 0 Range Output 0 Frac Bits
--------------------------------------------------- ----------------------------- ----------------------- --------------------
/model/embed_tokens/Gather contrib.epu.embedding [-0.628906f, 0.8125f] 31
/model/layers.0/input_layernorm/Mul_1 contrib.epu.rms_norm [-0.359118f, 0.37561f] 31
/model/layers.0/self_attn/q_proj/MatMul_smooth_mul multiply [-0.287793f, 0.307279f] 31
/model/layers.0/self_attn/k_proj/MatMul_smooth_mul multiply [-0.247679f, 0.223768f] 31
/model/layers.0/self_attn/v_proj/MatMul_smooth_mul multiply [-0.100749f, 0.107016f] 31
CustomOp/linalg::channelwiseQuantMatMul<43>3 contrib.epu.quadric_custom_op [-2.25614f, 3.85482f] 29
/model/layers.0/Add add [-2.30253f, 4.25466f] 28
/model/layers.0/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-1.23024f, 1.59211f] 29
/model/layers.0/mlp/gate_proj/MatMul_smooth_mul multiply [-0.682836f, 0.548619f] 28
CustomOp/nn::gateProj<41, 29>1 contrib.epu.quadric_custom_op [-0.278465f, 3.85478f] 29
/model/layers.0/mlp/up_proj/MatMul_smooth_mul multiply [-0.421256f, 0.474667f] 29
CustomOp/linalg::channelwiseQuantMatMul<42>0 contrib.epu.quadric_custom_op [-3.34433f, 3.38188f] 29
/model/layers.0/mlp/Mul multiply [-11.3828f, 9.95375f] 27
/model/layers.0/mlp/down_proj/MatMul_smooth_mul multiply [-2.19349f, 2.29969f] 26
CustomOp/linalg::channelwiseQuantMatMul<38>2 contrib.epu.quadric_custom_op [-5.18762f, 16.7897f] 26
/model/layers.0/Add_1 add [-6.67269f, 19.2419f] 26
/model/layers.1/input_layernorm/Mul_1 contrib.epu.rms_norm [-1.30106f, 1.14032f] 30
/model/layers.1/self_attn/q_proj/MatMul_smooth_mul multiply [-0.549389f, 0.618498f] 28
/model/layers.1/self_attn/k_proj/MatMul_smooth_mul multiply [-0.547089f, 0.290293f] 28
/model/layers.1/self_attn/v_proj/MatMul_smooth_mul multiply [-0.18635f, 0.285787f] 28
CustomOp/linalg::channelwiseQuantMatMul<43>7 contrib.epu.quadric_custom_op [-1.41003f, 1.53298f] 30
/model/layers.1/Add add [-6.78994f, 19.2634f] 26
/model/layers.1/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-51.933f, 47.6116f] 25
/model/layers.1/mlp/gate_proj/MatMul_smooth_mul multiply [-3.17403f, 3.52375f] 22
CustomOp/nn::gateProj<38, 27>5 contrib.epu.quadric_custom_op [-0.278465f, 7.77009f] 28
/model/layers.1/mlp/up_proj/MatMul_smooth_mul multiply [-2.03129f, 2.22485f] 22
CustomOp/linalg::channelwiseQuantMatMul<39>4 contrib.epu.quadric_custom_op [-13.6112f, 16.4607f] 26
/model/layers.1/mlp/Mul multiply [-22.4947f, 19.6069f] 23
/model/layers.1/mlp/down_proj/MatMul_smooth_mul multiply [-3.95853f, 6.61841f] 24
CustomOp/linalg::channelwiseQuantMatMul<37>6 contrib.epu.quadric_custom_op [-12.4679f, 38.1379f] 25
/model/layers.1/Add_1 add [-18.1108f, 57.4013f] 25
/model/layers.2/input_layernorm/Mul_1 contrib.epu.rms_norm [-0.862898f, 1.28555f] 29
/model/layers.2/self_attn/q_proj/MatMul_smooth_mul multiply [-0.369643f, 0.667969f] 28
/model/layers.2/self_attn/k_proj/MatMul_smooth_mul multiply [-0.560969f, 0.593967f] 28
/model/layers.2/self_attn/v_proj/MatMul_smooth_mul multiply [-0.209467f, 0.313265f] 28
CustomOp/linalg::channelwiseQuantMatMul<42>11 contrib.epu.quadric_custom_op [-1.03252f, 1.31911f] 30
/model/layers.2/Add add [-18.1085f, 57.4384f] 25
/model/layers.2/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-82.5232f, 81.3135f] 24
/model/layers.2/mlp/gate_proj/MatMul_smooth_mul multiply [-12.8995f, 14.1477f] 25
CustomOp/nn::gateProj<35, 24>9 contrib.epu.quadric_custom_op [-0.278465f, 7.80494f] 28
/model/layers.2/mlp/up_proj/MatMul_smooth_mul multiply [-12.5991f, 14.1477f] 25
CustomOp/linalg::channelwiseQuantMatMul<35>8 contrib.epu.quadric_custom_op [-122.194f, 103.963f] 24
/model/layers.2/mlp/Mul multiply [-26.306f, 24.5606f] 21
/model/layers.2/mlp/down_proj/MatMul_smooth_mul multiply [-5.16373f, 2.29959f] 22
CustomOp/linalg::channelwiseQuantMatMul<37>10 contrib.epu.quadric_custom_op [-10.8904f, 26.5761f] 26
/model/layers.2/Add_1 add [-23.2706f, 81.9635f] 24
/model/layers.3/input_layernorm/Mul_1 contrib.epu.rms_norm [-1.24747f, 3.00854f] 28
/model/layers.3/self_attn/q_proj/MatMul_smooth_mul multiply [-0.43238f, 0.730476f] 27
/model/layers.3/self_attn/k_proj/MatMul_smooth_mul multiply [-0.371823f, 1.19244f] 27
/model/layers.3/self_attn/v_proj/MatMul_smooth_mul multiply [-0.246321f, 0.362407f] 27
CustomOp/linalg::channelwiseQuantMatMul<42>15 contrib.epu.quadric_custom_op [-1.84038f, 1.97427f] 30
/model/layers.3/Add add [-23.1594f, 81.7177f] 24
/model/layers.3/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-53.3995f, 62.6125f] 25
/model/layers.3/mlp/gate_proj/MatMul_smooth_mul multiply [-3.54248f, 4.5413f] 22
CustomOp/nn::gateProj<37, 26>13 contrib.epu.quadric_custom_op [-0.278465f, 8.11204f] 27
/model/layers.3/mlp/up_proj/MatMul_smooth_mul multiply [-2.58039f, 4.31592f] 22
CustomOp/linalg::channelwiseQuantMatMul<37>12 contrib.epu.quadric_custom_op [-17.4614f, 10.6391f] 26
/model/layers.3/mlp/Mul multiply [-23.0629f, 23.7062f] 23
/model/layers.3/mlp/down_proj/MatMul_smooth_mul multiply [-3.35231f, 3.48879f] 25
CustomOp/linalg::channelwiseQuantMatMul<39>14 contrib.epu.quadric_custom_op [-7.28584f, 7.4731f] 28
/model/layers.3/Add_1 add [-23.3718f, 85.3067f] 24
/model/layers.4/input_layernorm/Mul_1 contrib.epu.rms_norm [-1.80589f, 2.00268f] 29
/model/layers.4/self_attn/q_proj/MatMul_smooth_mul multiply [-0.898383f, 0.659137f] 28
/model/layers.4/self_attn/k_proj/MatMul_smooth_mul multiply [-0.707868f, 1.08651f] 28
/model/layers.4/self_attn/v_proj/MatMul_smooth_mul multiply [-0.439939f, 0.234896f] 28
CustomOp/linalg::channelwiseQuantMatMul<42>19 contrib.epu.quadric_custom_op [-3.53964f, 2.62418f] 29
/model/layers.4/Add add [-23.2038f, 85.5677f] 24
/model/layers.4/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-32.953f, 41.0647f] 25
/model/layers.4/mlp/gate_proj/MatMul_smooth_mul multiply [-3.60791f, 4.06511f] 25
CustomOp/nn::gateProj<37, 27>17 contrib.epu.quadric_custom_op [-0.278465f, 9.32974f] 27
/model/layers.4/mlp/up_proj/MatMul_smooth_mul multiply [-2.50043f, 4.23358f] 25
CustomOp/linalg::channelwiseQuantMatMul<37>16 contrib.epu.quadric_custom_op [-8.54877f, 10.1004f] 27
/model/layers.4/mlp/Mul multiply [-20.2387f, 20.039f] 24
/model/layers.4/mlp/down_proj/MatMul_smooth_mul multiply [-2.69153f, 2.8874f] 26
CustomOp/linalg::channelwiseQuantMatMul<39>18 contrib.epu.quadric_custom_op [-6.69227f, 12.1684f] 27
/model/layers.4/Add_1 add [-22.4392f, 87.0669f] 24
/model/layers.5/input_layernorm/Mul_1 contrib.epu.rms_norm [-2.14681f, 3.19879f] 28
/model/layers.5/self_attn/q_proj/MatMul_smooth_mul multiply [-1.45607f, 0.922822f] 28
/model/layers.5/self_attn/k_proj/MatMul_smooth_mul multiply [-1.13808f, 1.1609f] 28
/model/layers.5/self_attn/v_proj/MatMul_smooth_mul multiply [-0.502765f, 0.449179f] 28
CustomOp/linalg::channelwiseQuantMatMul<42>23 contrib.epu.quadric_custom_op [-6.97809f, 4.76181f] 28
/model/layers.5/Add add [-21.9713f, 84.54f] 24
/model/layers.5/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-24.6416f, 19.9332f] 26
/model/layers.5/mlp/gate_proj/MatMul_smooth_mul multiply [-3.30293f, 4.25595f] 25
CustomOp/nn::gateProj<37, 27>21 contrib.epu.quadric_custom_op [-0.278465f, 7.52237f] 28
/model/layers.5/mlp/up_proj/MatMul_smooth_mul multiply [-1.51029f, 2.94375f] 25
CustomOp/linalg::channelwiseQuantMatMul<38>20 contrib.epu.quadric_custom_op [-6.38602f, 7.07606f] 28
/model/layers.5/mlp/Mul multiply [-18.2442f, 18.1234f] 25
/model/layers.5/mlp/down_proj/MatMul_smooth_mul multiply [-3.21112f, 2.15665f] 26
CustomOp/linalg::channelwiseQuantMatMul<38>22 contrib.epu.quadric_custom_op [-37.2974f, 18.5171f] 25
/model/layers.5/Add_1 add [-18.6574f, 63.1892f] 24
/model/layers.6/input_layernorm/Mul_1 contrib.epu.rms_norm [-1.99346f, 7.23117f] 27
/model/layers.6/self_attn/q_proj/MatMul_smooth_mul multiply [-0.737696f, 4.47784f] 27
/model/layers.6/self_attn/k_proj/MatMul_smooth_mul multiply [-0.58721f, 1.42655f] 27
/model/layers.6/self_attn/v_proj/MatMul_smooth_mul multiply [-0.422668f, 1.02092f] 27
CustomOp/linalg::channelwiseQuantMatMul<40>27 contrib.epu.quadric_custom_op [-31.7815f, 8.99221f] 26
/model/layers.6/Add add [-12.6824f, 38.3116f] 24
/model/layers.6/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-9.84406f, 199.177f] 23
/model/layers.6/mlp/gate_proj/MatMul_smooth_mul multiply [-0.969524f, 14.0399f] 22
CustomOp/nn::gateProj<34, 24>25 contrib.epu.quadric_custom_op [-0.278465f, 65.6915f] 24
/model/layers.6/mlp/up_proj/MatMul_smooth_mul multiply [-0.875245f, 14.5852f] 22
CustomOp/linalg::channelwiseQuantMatMul<34>24 contrib.epu.quadric_custom_op [-41.8197f, 74.1344f] 24
/model/layers.6/mlp/Mul multiply [-1282.21f, 4841.01f] 18
/model/layers.6/mlp/down_proj/MatMul_smooth_mul multiply [-112.681f, 254.839f] 18
CustomOp/linalg::channelwiseQuantMatMul<27>26 contrib.epu.quadric_custom_op [-2118.69f, 9638.07f] 17
/model/layers.6/Add_1 add [-2121.44f, 9654.1f] 17
/model/layers.7/input_layernorm/Mul_1 contrib.epu.rms_norm [-5.1855f, 10.441f] 27
/model/layers.7/self_attn/q_proj/MatMul_smooth_mul multiply [-1.18851f, 2.07018f] 27
/model/layers.7/self_attn/k_proj/MatMul_smooth_mul multiply [-1.36347f, 1.4916f] 26
/model/layers.7/self_attn/v_proj/MatMul_smooth_mul multiply [-0.655511f, 1.0774f] 27
CustomOp/linalg::channelwiseQuantMatMul<41>31 contrib.epu.quadric_custom_op [-4.27446f, 3.6406f] 28
/model/layers.7/Add add [-2121.23f, 9654.21f] 17
/model/layers.7/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-6.89828f, 54.4274f] 25
/model/layers.7/mlp/gate_proj/MatMul_smooth_mul multiply [-1.00739f, 4.84292f] 24
CustomOp/nn::gateProj<38, 27>29 contrib.epu.quadric_custom_op [-0.278465f, 11.2674f] 27
/model/layers.7/mlp/up_proj/MatMul_smooth_mul multiply [-1.13376f, 1.88627f] 24
CustomOp/linalg::channelwiseQuantMatMul<40>28 contrib.epu.quadric_custom_op [-13.397f, 10.9477f] 27
/model/layers.7/mlp/Mul multiply [-16.9021f, 17.2346f] 23
/model/layers.7/mlp/down_proj/MatMul_smooth_mul multiply [-2.6115f, 2.85751f] 26
CustomOp/linalg::channelwiseQuantMatMul<37>30 contrib.epu.quadric_custom_op [-6.76926f, 12.1386f] 27
/model/layers.7/Add_1 add [-2121.18f, 9654.34f] 17
/model/layers.8/input_layernorm/Mul_1 contrib.epu.rms_norm [-4.59068f, 9.9222f] 27
/model/layers.8/self_attn/q_proj/MatMul_smooth_mul multiply [-1.28958f, 1.87046f] 27
/model/layers.8/self_attn/k_proj/MatMul_smooth_mul multiply [-0.814643f, 1.79062f] 27
/model/layers.8/self_attn/v_proj/MatMul_smooth_mul multiply [-0.422026f, 0.896034f] 27
CustomOp/linalg::channelwiseQuantMatMul<41>35 contrib.epu.quadric_custom_op [-3.77026f, 7.3769f] 28
/model/layers.8/Add add [-2120.87f, 9653.89f] 17
/model/layers.8/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-7.8127f, 6.98587f] 28
/model/layers.8/mlp/gate_proj/MatMul_smooth_mul multiply [-1.26424f, 1.58606f] 27
CustomOp/nn::gateProj<39, 27>33 contrib.epu.quadric_custom_op [-0.278465f, 6.3098f] 28
/model/layers.8/mlp/up_proj/MatMul_smooth_mul multiply [-1.4333f, 1.11036f] 27
CustomOp/linalg::channelwiseQuantMatMul<40>32 contrib.epu.quadric_custom_op [-6.76552f, 5.97511f] 28
/model/layers.8/mlp/Mul multiply [-15.8356f, 18.0968f] 25
/model/layers.8/mlp/down_proj/MatMul_smooth_mul multiply [-2.78291f, 3.87402f] 27
CustomOp/linalg::channelwiseQuantMatMul<38>34 contrib.epu.quadric_custom_op [-10.3458f, 12.5126f] 27
/model/layers.8/Add_1 add [-2120.62f, 9654.73f] 17
/model/layers.9/input_layernorm/Mul_1 contrib.epu.rms_norm [-4.52299f, 10.3721f] 27
/model/layers.9/self_attn/q_proj/MatMul_smooth_mul multiply [-1.02019f, 1.58785f] 26
/model/layers.9/self_attn/k_proj/MatMul_smooth_mul multiply [-1.23027f, 1.70964f] 26
/model/layers.9/self_attn/v_proj/MatMul_smooth_mul multiply [-0.594754f, 0.902843f] 26
CustomOp/linalg::channelwiseQuantMatMul<39>39 contrib.epu.quadric_custom_op [-7.31632f, 7.46158f] 28
/model/layers.9/Add add [-2120.34f, 9654.1f] 17
/model/layers.9/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-8.45564f, 8.75462f] 27
/model/layers.9/mlp/gate_proj/MatMul_smooth_mul multiply [-1.24338f, 1.8245f] 27
CustomOp/nn::gateProj<39, 27>37 contrib.epu.quadric_custom_op [-0.278465f, 9.10514f] 27
/model/layers.9/mlp/up_proj/MatMul_smooth_mul multiply [-1.7809f, 1.67158f] 27
CustomOp/linalg::channelwiseQuantMatMul<39>36 contrib.epu.quadric_custom_op [-7.51536f, 6.51499f] 28
/model/layers.9/mlp/Mul multiply [-15.1261f, 14.7165f] 24
/model/layers.9/mlp/down_proj/MatMul_smooth_mul multiply [-2.19853f, 3.02141f] 27
CustomOp/linalg::channelwiseQuantMatMul<38>38 contrib.epu.quadric_custom_op [-9.86182f, 8.2895f] 27
/model/layers.9/Add_1 add [-2120.34f, 9656.53f] 17
/model/layers.10/input_layernorm/Mul_1 contrib.epu.rms_norm [-7.60531f, 17.5566f] 26
/model/layers.10/self_attn/q_proj/MatMul_smooth_mul multiply [-1.31638f, 2.45222f] 26
/model/layers.10/self_attn/k_proj/MatMul_smooth_mul multiply [-1.14123f, 2.85072f] 26
/model/layers.10/self_attn/v_proj/MatMul_smooth_mul multiply [-0.635305f, 0.964866f] 26
CustomOp/linalg::channelwiseQuantMatMul<40>43 contrib.epu.quadric_custom_op [-7.399f, 5.84284f] 28
/model/layers.10/Add add [-2119.36f, 9655.58f] 17
/model/layers.10/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-8.33987f, 9.41487f] 27
/model/layers.10/mlp/gate_proj/MatMul_smooth_mul multiply [-1.41946f, 1.68869f] 26
CustomOp/nn::gateProj<39, 27>41 contrib.epu.quadric_custom_op [-0.278465f, 8.82315f] 27
/model/layers.10/mlp/up_proj/MatMul_smooth_mul multiply [-1.41946f, 1.19373f] 27
CustomOp/linalg::channelwiseQuantMatMul<40>40 contrib.epu.quadric_custom_op [-5.76626f, 8.25113f] 27
/model/layers.10/mlp/Mul multiply [-15.7846f, 20.8058f] 24
/model/layers.10/mlp/down_proj/MatMul_smooth_mul multiply [-5.70319f, 2.03262f] 26
CustomOp/linalg::channelwiseQuantMatMul<37>42 contrib.epu.quadric_custom_op [-5.03255f, 18.6078f] 26
/model/layers.10/Add_1 add [-2119.79f, 9656.97f] 17
/model/layers.11/input_layernorm/Mul_1 contrib.epu.rms_norm [-5.77417f, 10.4802f] 27
/model/layers.11/self_attn/q_proj/MatMul_smooth_mul multiply [-1.24383f, 1.49914f] 28
/model/layers.11/self_attn/k_proj/MatMul_smooth_mul multiply [-0.911936f, 1.83978f] 28
/model/layers.11/self_attn/v_proj/MatMul_smooth_mul multiply [-0.494405f, 0.873254f] 28
CustomOp/linalg::channelwiseQuantMatMul<40>47 contrib.epu.quadric_custom_op [-7.13255f, 7.60828f] 28
/model/layers.11/Add add [-2118.77f, 9655.72f] 17
/model/layers.11/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-9.30534f, 7.62255f] 27
/model/layers.11/mlp/gate_proj/MatMul_smooth_mul multiply [-1.12876f, 1.54745f] 26
CustomOp/nn::gateProj<39, 27>45 contrib.epu.quadric_custom_op [-0.278465f, 12.7414f] 27
/model/layers.11/mlp/up_proj/MatMul_smooth_mul multiply [-1.37915f, 0.865526f] 26
CustomOp/linalg::channelwiseQuantMatMul<40>44 contrib.epu.quadric_custom_op [-8.25134f, 6.12666f] 27
/model/layers.11/mlp/Mul multiply [-12.4377f, 12.5137f] 24
/model/layers.11/mlp/down_proj/MatMul_smooth_mul multiply [-3.31324f, 2.34436f] 27
CustomOp/linalg::channelwiseQuantMatMul<38>46 contrib.epu.quadric_custom_op [-7.59268f, 11.5153f] 27
/model/layers.11/Add_1 add [-2118.11f, 9657.75f] 17
/model/layers.12/input_layernorm/Mul_1 contrib.epu.rms_norm [-6.76928f, 11.1257f] 27
/model/layers.12/self_attn/q_proj/MatMul_smooth_mul multiply [-1.31845f, 1.85694f] 28
/model/layers.12/self_attn/k_proj/MatMul_smooth_mul multiply [-1.1593f, 1.7855f] 27
/model/layers.12/self_attn/v_proj/MatMul_smooth_mul multiply [-0.844806f, 0.934658f] 28
CustomOp/linalg::channelwiseQuantMatMul<40>51 contrib.epu.quadric_custom_op [-7.8656f, 13.7375f] 27
/model/layers.12/Add add [-2117.66f, 9657.01f] 17
/model/layers.12/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-10.1594f, 7.15519f] 27
/model/layers.12/mlp/gate_proj/MatMul_smooth_mul multiply [-1.37687f, 1.39484f] 26
CustomOp/nn::gateProj<40, 27>49 contrib.epu.quadric_custom_op [-0.278465f, 7.39754f] 28
/model/layers.12/mlp/up_proj/MatMul_smooth_mul multiply [-1.92166f, 1.10387f] 26
CustomOp/linalg::channelwiseQuantMatMul<39>48 contrib.epu.quadric_custom_op [-9.58119f, 5.33228f] 27
/model/layers.12/mlp/Mul multiply [-60.1034f, 12.2375f] 24
/model/layers.12/mlp/down_proj/MatMul_smooth_mul multiply [-3.8142f, 3.11107f] 25
CustomOp/linalg::channelwiseQuantMatMul<37>50 contrib.epu.quadric_custom_op [-12.1023f, 17.851f] 26
/model/layers.12/Add_1 add [-2118.72f, 9659.33f] 17
/model/layers.13/input_layernorm/Mul_1 contrib.epu.rms_norm [-5.04587f, 9.31552f] 27
/model/layers.13/self_attn/q_proj/MatMul_smooth_mul multiply [-1.00549f, 1.73688f] 28
/model/layers.13/self_attn/k_proj/MatMul_smooth_mul multiply [-1.76774f, 1.88082f] 28
/model/layers.13/self_attn/v_proj/MatMul_smooth_mul multiply [-0.604113f, 0.910164f] 28
CustomOp/linalg::channelwiseQuantMatMul<40>55 contrib.epu.quadric_custom_op [-5.54142f, 8.70166f] 27
/model/layers.13/Add add [-2117.86f, 9659.14f] 17
/model/layers.13/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-11.9317f, 6.23108f] 27
/model/layers.13/mlp/gate_proj/MatMul_smooth_mul multiply [-1.47657f, 1.26196f] 26
CustomOp/nn::gateProj<40, 26>53 contrib.epu.quadric_custom_op [-0.278465f, 17.2259f] 26
/model/layers.13/mlp/up_proj/MatMul_smooth_mul multiply [-2.24165f, 1.21306f] 26
CustomOp/linalg::channelwiseQuantMatMul<39>52 contrib.epu.quadric_custom_op [-30.4806f, 32.1002f] 25
/model/layers.13/mlp/Mul multiply [-16.9783f, 33.7621f] 21
/model/layers.13/mlp/down_proj/MatMul_smooth_mul multiply [-2.68679f, 2.99732f] 25
CustomOp/linalg::channelwiseQuantMatMul<38>54 contrib.epu.quadric_custom_op [-11.2142f, 11.9456f] 27
/model/layers.13/Add_1 add [-2120.76f, 9664.39f] 17
/model/layers.14/input_layernorm/Mul_1 contrib.epu.rms_norm [-6.68111f, 13.0119f] 27
/model/layers.14/self_attn/q_proj/MatMul_smooth_mul multiply [-1.40054f, 1.87548f] 27
/model/layers.14/self_attn/k_proj/MatMul_smooth_mul multiply [-1.56909f, 1.68693f] 28
/model/layers.14/self_attn/v_proj/MatMul_smooth_mul multiply [-0.623238f, 0.820612f] 28
CustomOp/linalg::channelwiseQuantMatMul<40>59 contrib.epu.quadric_custom_op [-8.29877f, 19.2297f] 26
/model/layers.14/Add add [-2120.07f, 9663.34f] 17
/model/layers.14/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-14.412f, 6.25216f] 27
/model/layers.14/mlp/gate_proj/MatMul_smooth_mul multiply [-1.47538f, 1.42171f] 26
CustomOp/nn::gateProj<39, 26>57 contrib.epu.quadric_custom_op [-0.278465f, 17.3895f] 26
/model/layers.14/mlp/up_proj/MatMul_smooth_mul multiply [-2.12944f, 0.916271f] 26
CustomOp/linalg::channelwiseQuantMatMul<39>56 contrib.epu.quadric_custom_op [-24.6362f, 32.0892f] 25
/model/layers.14/mlp/Mul multiply [-84.1848f, 13.7615f] 21
/model/layers.14/mlp/down_proj/MatMul_smooth_mul multiply [-4.83579f, 2.83515f] 24
CustomOp/linalg::channelwiseQuantMatMul<36>58 contrib.epu.quadric_custom_op [-18.0127f, 15.0041f] 26
/model/layers.14/Add_1 add [-2123.29f, 9666.16f] 17
/model/layers.15/input_layernorm/Mul_1 contrib.epu.rms_norm [-7.40155f, 13.516f] 27
/model/layers.15/self_attn/q_proj/MatMul_smooth_mul multiply [-2.20786f, 2.05356f] 27
/model/layers.15/self_attn/k_proj/MatMul_smooth_mul multiply [-1.312f, 1.55167f] 27
/model/layers.15/self_attn/v_proj/MatMul_smooth_mul multiply [-0.944417f, 1.36199f] 28
CustomOp/linalg::channelwiseQuantMatMul<39>63 contrib.epu.quadric_custom_op [-6.83366f, 13.2874f] 27
/model/layers.15/Add add [-2122.7f, 9664.69f] 17
/model/layers.15/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-16.0488f, 6.0668f] 26
/model/layers.15/mlp/gate_proj/MatMul_smooth_mul multiply [-1.4562f, 1.5061f] 26
CustomOp/nn::gateProj<39, 26>61 contrib.epu.quadric_custom_op [-0.278465f, 19.5172f] 26
/model/layers.15/mlp/up_proj/MatMul_smooth_mul multiply [-3.49456f, 0.898868f] 26
CustomOp/linalg::channelwiseQuantMatMul<38>60 contrib.epu.quadric_custom_op [-16.5025f, 59.7275f] 25
/model/layers.15/mlp/Mul multiply [-27.8356f, 48.4799f] 20
/model/layers.15/mlp/down_proj/MatMul_smooth_mul multiply [-3.85041f, 4.73925f] 25
CustomOp/linalg::channelwiseQuantMatMul<36>62 contrib.epu.quadric_custom_op [-16.8693f, 20.4889f] 26
/model/layers.15/Add_1 add [-2123.21f, 9668.25f] 17
/model/layers.16/input_layernorm/Mul_1 contrib.epu.rms_norm [-8.15026f, 15.6546f] 26
/model/layers.16/self_attn/q_proj/MatMul_smooth_mul multiply [-1.3261f, 2.04648f] 27
/model/layers.16/self_attn/k_proj/MatMul_smooth_mul multiply [-1.65707f, 1.55809f] 27
/model/layers.16/self_attn/v_proj/MatMul_smooth_mul multiply [-0.734148f, 0.882209f] 28
CustomOp/linalg::channelwiseQuantMatMul<38>67 contrib.epu.quadric_custom_op [-39.5976f, 22.6377f] 25
/model/layers.16/Add add [-2122.92f, 9666.15f] 17
/model/layers.16/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-36.8359f, 6.53895f] 25
/model/layers.16/mlp/gate_proj/MatMul_smooth_mul multiply [-4.24899f, 2.25932f] 25
CustomOp/nn::gateProj<37, 23>65 contrib.epu.quadric_custom_op [-0.278465f, 78.4472f] 24
/model/layers.16/mlp/up_proj/MatMul_smooth_mul multiply [-4.45054f, 1.66161f] 25
CustomOp/linalg::channelwiseQuantMatMul<37>64 contrib.epu.quadric_custom_op [-39.9357f, 84.0414f] 24
/model/layers.16/mlp/Mul multiply [-191.33f, 4661.68f] 18
/model/layers.16/mlp/down_proj/MatMul_smooth_mul multiply [-40.6463f, 352.551f] 18
CustomOp/linalg::channelwiseQuantMatMul<27>66 contrib.epu.quadric_custom_op [-1202.39f, 13581.2f] 17
/model/layers.16/Add_1 add [-2130f, 13602.6f] 16
/model/layers.17/input_layernorm/Mul_1 contrib.epu.rms_norm [-7.88683f, 13.0965f] 27
/model/layers.17/self_attn/q_proj/MatMul_smooth_mul multiply [-1.33071f, 1.92274f] 27
/model/layers.17/self_attn/k_proj/MatMul_smooth_mul multiply [-1.34317f, 1.59226f] 27
/model/layers.17/self_attn/v_proj/MatMul_smooth_mul multiply [-0.929569f, 1.09737f] 28
CustomOp/linalg::channelwiseQuantMatMul<37>71 contrib.epu.quadric_custom_op [-14.5229f, 61.5807f] 25
/model/layers.17/Add add [-2129.24f, 13637f] 16
/model/layers.17/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-12.2827f, 6.15562f] 27
/model/layers.17/mlp/gate_proj/MatMul_smooth_mul multiply [-1.52264f, 1.37121f] 26
CustomOp/nn::gateProj<39, 28>69 contrib.epu.quadric_custom_op [-0.278465f, 7.73579f] 28
/model/layers.17/mlp/up_proj/MatMul_smooth_mul multiply [-1.52896f, 0.967658f] 26
CustomOp/linalg::channelwiseQuantMatMul<40>68 contrib.epu.quadric_custom_op [-7.26674f, 35.3441f] 25
/model/layers.17/mlp/Mul multiply [-13.8208f, 9.58101f] 22
/model/layers.17/mlp/down_proj/MatMul_smooth_mul multiply [-3.27512f, 3.71965f] 27
CustomOp/linalg::channelwiseQuantMatMul<38>70 contrib.epu.quadric_custom_op [-21.4484f, 22.413f] 26
/model/layers.17/Add_1 add [-2128.21f, 13636.9f] 16
/model/layers.18/input_layernorm/Mul_1 contrib.epu.rms_norm [-10.6242f, 15.4063f] 26
/model/layers.18/self_attn/q_proj/MatMul_smooth_mul multiply [-1.42803f, 2.10296f] 27
/model/layers.18/self_attn/k_proj/MatMul_smooth_mul multiply [-1.48392f, 1.69128f] 27
/model/layers.18/self_attn/v_proj/MatMul_smooth_mul multiply [-0.998706f, 1.36009f] 28
CustomOp/linalg::channelwiseQuantMatMul<40>75 contrib.epu.quadric_custom_op [-11.2483f, 15.3732f] 27
/model/layers.18/Add add [-2127.25f, 13638.1f] 16
/model/layers.18/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-11.7031f, 6.12549f] 27
/model/layers.18/mlp/gate_proj/MatMul_smooth_mul multiply [-1.18675f, 1.43604f] 26
CustomOp/nn::gateProj<39, 27>73 contrib.epu.quadric_custom_op [-0.278465f, 8.13071f] 27
/model/layers.18/mlp/up_proj/MatMul_smooth_mul multiply [-1.42666f, 1.13167f] 26
CustomOp/linalg::channelwiseQuantMatMul<39>72 contrib.epu.quadric_custom_op [-6.42202f, 15.0935f] 27
/model/layers.18/mlp/Mul multiply [-9.16472f, 64.1296f] 24
/model/layers.18/mlp/down_proj/MatMul_smooth_mul multiply [-3.79416f, 8.76246f] 25
CustomOp/linalg::channelwiseQuantMatMul<35>74 contrib.epu.quadric_custom_op [-27.1561f, 62.6349f] 25
/model/layers.18/Add_1 add [-2127.74f, 13638.1f] 16
/model/layers.19/input_layernorm/Mul_1 contrib.epu.rms_norm [-16.3269f, 23.5058f] 26
/model/layers.19/self_attn/q_proj/MatMul_smooth_mul multiply [-1.87381f, 2.36709f] 27
/model/layers.19/self_attn/k_proj/MatMul_smooth_mul multiply [-1.99903f, 2.0809f] 27
/model/layers.19/self_attn/v_proj/MatMul_smooth_mul multiply [-0.96973f, 1.21001f] 27
CustomOp/linalg::channelwiseQuantMatMul<39>79 contrib.epu.quadric_custom_op [-9.69087f, 19.3656f] 26
/model/layers.19/Add add [-2127.19f, 13637.7f] 16
/model/layers.19/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-11.4451f, 6.57414f] 27
/model/layers.19/mlp/gate_proj/MatMul_smooth_mul multiply [-1.50948f, 1.27079f] 27
CustomOp/nn::gateProj<40, 27>77 contrib.epu.quadric_custom_op [-0.278465f, 8.66148f] 27
/model/layers.19/mlp/up_proj/MatMul_smooth_mul multiply [-1.77521f, 1.54593f] 27
CustomOp/linalg::channelwiseQuantMatMul<39>76 contrib.epu.quadric_custom_op [-6.79348f, 5.92249f] 28
/model/layers.19/mlp/Mul multiply [-10.5208f, 11.7491f] 25
/model/layers.19/mlp/down_proj/MatMul_smooth_mul multiply [-3.32359f, 3.64369f] 27
CustomOp/linalg::channelwiseQuantMatMul<36>78 contrib.epu.quadric_custom_op [-28.7991f, 32.1844f] 25
/model/layers.19/Add_1 add [-2127.2f, 13637.6f] 16
/model/layers.20/input_layernorm/Mul_1 contrib.epu.rms_norm [-17.2382f, 20.6114f] 26
/model/layers.20/self_attn/q_proj/MatMul_smooth_mul multiply [-1.79777f, 2.35375f] 27
/model/layers.20/self_attn/k_proj/MatMul_smooth_mul multiply [-2.07954f, 1.97869f] 27
/model/layers.20/self_attn/v_proj/MatMul_smooth_mul multiply [-0.97567f, 1.51478f] 27
CustomOp/linalg::channelwiseQuantMatMul<40>83 contrib.epu.quadric_custom_op [-11.7166f, 31.5278f] 26
/model/layers.20/Add add [-2126.62f, 13634.6f] 16
/model/layers.20/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-12.2707f, 7.12023f] 27
/model/layers.20/mlp/gate_proj/MatMul_smooth_mul multiply [-1.27321f, 1.48049f] 28
CustomOp/nn::gateProj<40, 27>81 contrib.epu.quadric_custom_op [-0.278465f, 8.94129f] 27
/model/layers.20/mlp/up_proj/MatMul_smooth_mul multiply [-1.25212f, 1.368f] 27
CustomOp/linalg::channelwiseQuantMatMul<40>80 contrib.epu.quadric_custom_op [-6.33998f, 8.02105f] 27
/model/layers.20/mlp/Mul multiply [-18.5987f, 19.8616f] 24
/model/layers.20/mlp/down_proj/MatMul_smooth_mul multiply [-4.02727f, 2.56306f] 26
CustomOp/linalg::channelwiseQuantMatMul<38>82 contrib.epu.quadric_custom_op [-16.8789f, 16.8205f] 26
/model/layers.20/Add_1 add [-2125.87f, 13634.7f] 16
/model/layers.21/input_layernorm/Mul_1 contrib.epu.rms_norm [-23.1988f, 24.3195f] 26
/model/layers.21/self_attn/q_proj/MatMul_smooth_mul multiply [-2.32063f, 2.84108f] 27
/model/layers.21/self_attn/k_proj/MatMul_smooth_mul multiply [-2.10805f, 2.21897f] 27
/model/layers.21/self_attn/v_proj/MatMul_smooth_mul multiply [-1.10305f, 1.53724f] 27
CustomOp/linalg::channelwiseQuantMatMul<39>87 contrib.epu.quadric_custom_op [-8.88642f, 21.276f] 26
/model/layers.21/Add add [-2124.82f, 13636.5f] 16
/model/layers.21/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-10.9728f, 7.28116f] 27
/model/layers.21/mlp/gate_proj/MatMul_smooth_mul multiply [-1.61932f, 1.36054f] 26
CustomOp/nn::gateProj<39, 27>85 contrib.epu.quadric_custom_op [-0.278465f, 9.86578f] 27
/model/layers.21/mlp/up_proj/MatMul_smooth_mul multiply [-1.27146f, 1.53064f] 26
CustomOp/linalg::channelwiseQuantMatMul<39>84 contrib.epu.quadric_custom_op [-10.277f, 10.0276f] 27
/model/layers.21/mlp/Mul multiply [-41.594f, 53.735f] 24
/model/layers.21/mlp/down_proj/MatMul_smooth_mul multiply [-5.86662f, 8.1078f] 25
CustomOp/linalg::channelwiseQuantMatMul<37>86 contrib.epu.quadric_custom_op [-32.024f, 16.017f] 25
/model/layers.21/Add_1 add [-2124.91f, 13636.6f] 16
/model/layers.22/input_layernorm/Mul_1 contrib.epu.rms_norm [-30.8369f, 32.7385f] 25
/model/layers.22/self_attn/q_proj/MatMul_smooth_mul multiply [-2.71083f, 3.39645f] 27
/model/layers.22/self_attn/k_proj/MatMul_smooth_mul multiply [-3.44054f, 3.20511f] 26
/model/layers.22/self_attn/v_proj/MatMul_smooth_mul multiply [-1.28456f, 1.37292f] 27
CustomOp/linalg::channelwiseQuantMatMul<39>91 contrib.epu.quadric_custom_op [-12.5842f, 31.0796f] 26
/model/layers.22/Add add [-2124.1f, 13635.5f] 16
/model/layers.22/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-10.0459f, 8.97793f] 27
/model/layers.22/mlp/gate_proj/MatMul_smooth_mul multiply [-1.56994f, 1.54569f] 27
CustomOp/nn::gateProj<39, 26>89 contrib.epu.quadric_custom_op [-0.278465f, 16.0594f] 26
/model/layers.22/mlp/up_proj/MatMul_smooth_mul multiply [-1.54481f, 1.40509f] 27
CustomOp/linalg::channelwiseQuantMatMul<39>88 contrib.epu.quadric_custom_op [-8.84424f, 9.51616f] 27
/model/layers.22/mlp/Mul multiply [-42.2895f, 37.618f] 23
/model/layers.22/mlp/down_proj/MatMul_smooth_mul multiply [-5.37248f, 5.52488f] 26
CustomOp/linalg::channelwiseQuantMatMul<37>90 contrib.epu.quadric_custom_op [-24.8563f, 17.5565f] 26
/model/layers.22/Add_1 add [-2123.61f, 13635.4f] 16
/model/layers.23/input_layernorm/Mul_1 contrib.epu.rms_norm [-32.5652f, 31.0209f] 25
/model/layers.23/self_attn/q_proj/MatMul_smooth_mul multiply [-3.54566f, 3.57557f] 27
/model/layers.23/self_attn/k_proj/MatMul_smooth_mul multiply [-2.95217f, 3.04757f] 27
/model/layers.23/self_attn/v_proj/MatMul_smooth_mul multiply [-1.23035f, 1.36422f] 27
CustomOp/linalg::channelwiseQuantMatMul<39>95 contrib.epu.quadric_custom_op [-8.13639f, 35.0445f] 25
/model/layers.23/Add add [-2123.06f, 13632.6f] 16
/model/layers.23/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-10.1758f, 10.4433f] 27
/model/layers.23/mlp/gate_proj/MatMul_smooth_mul multiply [-1.73753f, 1.58375f] 27
CustomOp/nn::gateProj<39, 26>93 contrib.epu.quadric_custom_op [-0.278465f, 15.0449f] 27
/model/layers.23/mlp/up_proj/MatMul_smooth_mul multiply [-1.42182f, 1.61803f] 27
CustomOp/linalg::channelwiseQuantMatMul<39>92 contrib.epu.quadric_custom_op [-12.3414f, 10.3189f] 27
/model/layers.23/mlp/Mul multiply [-64.6526f, 60.3269f] 23
/model/layers.23/mlp/down_proj/MatMul_smooth_mul multiply [-5.1301f, 6.31523f] 25
CustomOp/linalg::channelwiseQuantMatMul<37>94 contrib.epu.quadric_custom_op [-24.6908f, 31.6436f] 26
/model/layers.23/Add_1 add [-2122.79f, 13632.6f] 16
/model/layers.24/input_layernorm/Mul_1 contrib.epu.rms_norm [-50.8646f, 39.9522f] 25
/model/layers.24/self_attn/q_proj/MatMul_smooth_mul multiply [-4.28403f, 3.79681f] 26
/model/layers.24/self_attn/k_proj/MatMul_smooth_mul multiply [-3.08274f, 3.52826f] 27
/model/layers.24/self_attn/v_proj/MatMul_smooth_mul multiply [-2.11367f, 2.06183f] 27
CustomOp/linalg::channelwiseQuantMatMul<38>99 contrib.epu.quadric_custom_op [-16.7965f, 31.3655f] 26
/model/layers.24/Add add [-2123.99f, 13635.1f] 16
/model/layers.24/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-10.2601f, 12.5924f] 27
/model/layers.24/mlp/gate_proj/MatMul_smooth_mul multiply [-1.79581f, 1.95177f] 26
CustomOp/nn::gateProj<39, 26>97 contrib.epu.quadric_custom_op [-0.278465f, 17.9184f] 26
/model/layers.24/mlp/up_proj/MatMul_smooth_mul multiply [-1.58951f, 1.69809f] 26
CustomOp/linalg::channelwiseQuantMatMul<39>96 contrib.epu.quadric_custom_op [-11.0467f, 11.354f] 27
/model/layers.24/mlp/Mul multiply [-107.554f, 90.9104f] 23
/model/layers.24/mlp/down_proj/MatMul_smooth_mul multiply [-6.40439f, 6.75044f] 25
CustomOp/linalg::channelwiseQuantMatMul<36>98 contrib.epu.quadric_custom_op [-22.1531f, 38.053f] 25
/model/layers.24/Add_1 add [-2123.99f, 13635.3f] 16
/model/layers.25/input_layernorm/Mul_1 contrib.epu.rms_norm [-40.2525f, 33.3074f] 25
/model/layers.25/self_attn/q_proj/MatMul_smooth_mul multiply [-3.55716f, 3.96355f] 26
/model/layers.25/self_attn/k_proj/MatMul_smooth_mul multiply [-2.68001f, 3.21491f] 27
/model/layers.25/self_attn/v_proj/MatMul_smooth_mul multiply [-1.83399f, 1.64135f] 27
CustomOp/linalg::channelwiseQuantMatMul<39>103 contrib.epu.quadric_custom_op [-10.4602f, 17.5029f] 26
/model/layers.25/Add add [-2123.75f, 13635.5f] 16
/model/layers.25/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-11.309f, 12.7584f] 27
/model/layers.25/mlp/gate_proj/MatMul_smooth_mul multiply [-1.95923f, 2.00956f] 26
CustomOp/nn::gateProj<39, 26>101 contrib.epu.quadric_custom_op [-0.278465f, 20.8095f] 26
/model/layers.25/mlp/up_proj/MatMul_smooth_mul multiply [-1.65599f, 1.83448f] 26
CustomOp/linalg::channelwiseQuantMatMul<39>100 contrib.epu.quadric_custom_op [-16.033f, 12.6045f] 26
/model/layers.25/mlp/Mul multiply [-117.555f, 109.813f] 22
/model/layers.25/mlp/down_proj/MatMul_smooth_mul multiply [-7.04222f, 8.54142f] 24
CustomOp/linalg::channelwiseQuantMatMul<36>102 contrib.epu.quadric_custom_op [-19.0471f, 36.3233f] 25
/model/layers.25/Add_1 add [-2123.73f, 13635.5f] 16
/model/layers.26/input_layernorm/Mul_1 contrib.epu.rms_norm [-50.6021f, 38.8768f] 25
/model/layers.26/self_attn/q_proj/MatMul_smooth_mul multiply [-4.46383f, 4.41033f] 26
/model/layers.26/self_attn/k_proj/MatMul_smooth_mul multiply [-3.09661f, 3.23338f] 27
/model/layers.26/self_attn/v_proj/MatMul_smooth_mul multiply [-1.95177f, 1.62671f] 27
CustomOp/linalg::channelwiseQuantMatMul<40>107 contrib.epu.quadric_custom_op [-8.51294f, 19.8805f] 26
/model/layers.26/Add add [-2124.04f, 13636.4f] 16
/model/layers.26/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-12.5561f, 15.4058f] 26
/model/layers.26/mlp/gate_proj/MatMul_smooth_mul multiply [-2.00442f, 1.59836f] 26
CustomOp/nn::gateProj<39, 26>105 contrib.epu.quadric_custom_op [-0.278465f, 22.6042f] 26
/model/layers.26/mlp/up_proj/MatMul_smooth_mul multiply [-1.56655f, 2.01047f] 26
CustomOp/linalg::channelwiseQuantMatMul<38>104 contrib.epu.quadric_custom_op [-12.8656f, 13.9187f] 27
/model/layers.26/mlp/Mul multiply [-105.01f, 132.743f] 22
/model/layers.26/mlp/down_proj/MatMul_smooth_mul multiply [-6.63407f, 7.95295f] 24
CustomOp/linalg::channelwiseQuantMatMul<36>106 contrib.epu.quadric_custom_op [-21.3917f, 49.6387f] 25
/model/layers.26/Add_1 add [-2124.03f, 13636.6f] 16
/model/layers.27/input_layernorm/Mul_1 contrib.epu.rms_norm [-61.2838f, 48.1027f] 25
/model/layers.27/self_attn/q_proj/MatMul_smooth_mul multiply [-4.33818f, 4.62879f] 26
/model/layers.27/self_attn/k_proj/MatMul_smooth_mul multiply [-2.61422f, 3.56265f] 26
/model/layers.27/self_attn/v_proj/MatMul_smooth_mul multiply [-2.02177f, 2.13091f] 27
CustomOp/linalg::channelwiseQuantMatMul<39>111 contrib.epu.quadric_custom_op [-8.8824f, 19.0919f] 26
/model/layers.27/Add add [-2124.55f, 13638.8f] 16
/model/layers.27/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-14.5542f, 17.2044f] 26
/model/layers.27/mlp/gate_proj/MatMul_smooth_mul multiply [-1.75765f, 1.86848f] 26
CustomOp/nn::gateProj<39, 26>109 contrib.epu.quadric_custom_op [-0.278465f, 19.2789f] 26
/model/layers.27/mlp/up_proj/MatMul_smooth_mul multiply [-3.00212f, 1.82197f] 26
CustomOp/linalg::channelwiseQuantMatMul<38>108 contrib.epu.quadric_custom_op [-13.1627f, 12.5415f] 27
/model/layers.27/mlp/Mul multiply [-137.571f, 122.284f] 23
/model/layers.27/mlp/down_proj/MatMul_smooth_mul multiply [-8.10809f, 8.06381f] 24
CustomOp/linalg::channelwiseQuantMatMul<36>110 contrib.epu.quadric_custom_op [-29.7459f, 42.6272f] 25
/model/layers.27/Add_1 add [-2124.49f, 13639.2f] 16
/model/layers.28/input_layernorm/Mul_1 contrib.epu.rms_norm [-65.7556f, 56.4749f] 24
/model/layers.28/self_attn/q_proj/MatMul_smooth_mul multiply [-4.29473f, 4.18318f] 26
/model/layers.28/self_attn/k_proj/MatMul_smooth_mul multiply [-3.32743f, 3.96202f] 27
/model/layers.28/self_attn/v_proj/MatMul_smooth_mul multiply [-2.12739f, 2.4797f] 27
CustomOp/linalg::channelwiseQuantMatMul<39>115 contrib.epu.quadric_custom_op [-16.1447f, 22.5356f] 26
/model/layers.28/Add add [-2125.41f, 13642.1f] 16
/model/layers.28/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-13.3559f, 17.9234f] 26
/model/layers.28/mlp/gate_proj/MatMul_smooth_mul multiply [-2.34241f, 2.10349f] 23
CustomOp/nn::gateProj<38, 26>113 contrib.epu.quadric_custom_op [-0.278465f, 23.2141f] 26
/model/layers.28/mlp/up_proj/MatMul_smooth_mul multiply [-2.12235f, 1.94853f] 23
CustomOp/linalg::channelwiseQuantMatMul<38>112 contrib.epu.quadric_custom_op [-18.7225f, 14.9992f] 26
/model/layers.28/mlp/Mul multiply [-132.766f, 140.229f] 22
/model/layers.28/mlp/down_proj/MatMul_smooth_mul multiply [-14.6529f, 10.7982f] 24
CustomOp/linalg::channelwiseQuantMatMul<35>114 contrib.epu.quadric_custom_op [-38.7672f, 58.2323f] 25
/model/layers.28/Add_1 add [-2125.36f, 13642.5f] 16
/model/layers.29/input_layernorm/Mul_1 contrib.epu.rms_norm [-87.7913f, 63.9043f] 24
/model/layers.29/self_attn/q_proj/MatMul_smooth_mul multiply [-5.64585f, 4.53467f] 26
/model/layers.29/self_attn/k_proj/MatMul_smooth_mul multiply [-4.66197f, 4.02658f] 26
/model/layers.29/self_attn/v_proj/MatMul_smooth_mul multiply [-3.72409f, 3.62137f] 26
CustomOp/linalg::channelwiseQuantMatMul<39>119 contrib.epu.quadric_custom_op [-15.1006f, 23.0286f] 26
/model/layers.29/Add add [-2126.23f, 13643.9f] 16
/model/layers.29/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-14.5663f, 19.9309f] 26
/model/layers.29/mlp/gate_proj/MatMul_smooth_mul multiply [-1.95651f, 1.86312f] 25
CustomOp/nn::gateProj<39, 26>117 contrib.epu.quadric_custom_op [-0.278465f, 19.1744f] 26
/model/layers.29/mlp/up_proj/MatMul_smooth_mul multiply [-2.62913f, 2.32513f] 25
CustomOp/linalg::channelwiseQuantMatMul<38>116 contrib.epu.quadric_custom_op [-16.4508f, 15.0669f] 26
/model/layers.29/mlp/Mul multiply [-150.54f, 123.012f] 22
/model/layers.29/mlp/down_proj/MatMul_smooth_mul multiply [-11.6399f, 11.8174f] 24
CustomOp/linalg::channelwiseQuantMatMul<35>118 contrib.epu.quadric_custom_op [-49.9533f, 76.5029f] 24
/model/layers.29/Add_1 add [-2126.19f, 13644f] 16
/model/layers.30/input_layernorm/Mul_1 contrib.epu.rms_norm [-97.8058f, 70.9147f] 24
/model/layers.30/self_attn/q_proj/MatMul_smooth_mul multiply [-8.32574f, 6.67203f] 26
/model/layers.30/self_attn/k_proj/MatMul_smooth_mul multiply [-5.20679f, 4.87224f] 26
/model/layers.30/self_attn/v_proj/MatMul_smooth_mul multiply [-3.51157f, 3.44255f] 26
CustomOp/linalg::channelwiseQuantMatMul<38>123 contrib.epu.quadric_custom_op [-26.3882f, 42.82f] 25
/model/layers.30/Add add [-2126.46f, 13648.6f] 16
/model/layers.30/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-16.1268f, 19.8625f] 26
/model/layers.30/mlp/gate_proj/MatMul_smooth_mul multiply [-2.32769f, 1.89138f] 27
CustomOp/nn::gateProj<38, 26>121 contrib.epu.quadric_custom_op [-0.278465f, 20.8345f] 26
/model/layers.30/mlp/up_proj/MatMul_smooth_mul multiply [-2.93709f, 3.14719f] 27
CustomOp/linalg::channelwiseQuantMatMul<38>120 contrib.epu.quadric_custom_op [-22.3626f, 17.1751f] 26
/model/layers.30/mlp/Mul multiply [-147.928f, 141.815f] 22
/model/layers.30/mlp/down_proj/MatMul_smooth_mul multiply [-15.4673f, 11.2169f] 24
CustomOp/linalg::channelwiseQuantMatMul<35>122 contrib.epu.quadric_custom_op [-53.484f, 106.991f] 24
/model/layers.30/Add_1 add [-2126.44f, 13648.7f] 16
/model/layers.31/input_layernorm/Mul_1 contrib.epu.rms_norm [-119.533f, 93.6401f] 24
/model/layers.31/self_attn/q_proj/MatMul_smooth_mul multiply [-7.37432f, 6.46369f] 26
/model/layers.31/self_attn/k_proj/MatMul_smooth_mul multiply [-5.50964f, 5.44922f] 25
/model/layers.31/self_attn/v_proj/MatMul_smooth_mul multiply [-3.79376f, 3.57184f] 26
CustomOp/linalg::channelwiseQuantMatMul<38>127 contrib.epu.quadric_custom_op [-17.2974f, 50.1237f] 25
/model/layers.31/Add add [-2126.83f, 13653.5f] 16
/model/layers.31/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-17.2516f, 18.0006f] 26
/model/layers.31/mlp/gate_proj/MatMul_smooth_mul multiply [-2.36857f, 2.28309f] 27
CustomOp/nn::gateProj<38, 26>125 contrib.epu.quadric_custom_op [-0.278465f, 21.2043f] 26
/model/layers.31/mlp/up_proj/MatMul_smooth_mul multiply [-2.60498f, 3.19835f] 27
CustomOp/linalg::channelwiseQuantMatMul<37>124 contrib.epu.quadric_custom_op [-18.9059f, 19.1897f] 26
/model/layers.31/mlp/Mul multiply [-147.492f, 171.108f] 22
/model/layers.31/mlp/down_proj/MatMul_smooth_mul multiply [-18.9471f, 14.1054f] 24
CustomOp/linalg::channelwiseQuantMatMul<34>126 contrib.epu.quadric_custom_op [-42.7217f, 101.298f] 24
/model/layers.31/Add_1 add [-2126.65f, 13653.4f] 16
/model/layers.32/input_layernorm/Mul_1 contrib.epu.rms_norm [-129.345f, 108.642f] 23
/model/layers.32/self_attn/q_proj/MatMul_smooth_mul multiply [-7.50556f, 7.79626f] 26
/model/layers.32/self_attn/k_proj/MatMul_smooth_mul multiply [-6.04744f, 5.6172f] 26
/model/layers.32/self_attn/v_proj/MatMul_smooth_mul multiply [-5.33153f, 4.74853f] 26
CustomOp/linalg::channelwiseQuantMatMul<37>131 contrib.epu.quadric_custom_op [-26.9792f, 52.8894f] 25
/model/layers.32/Add add [-2125.55f, 13676.5f] 16
/model/layers.32/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-17.899f, 23.7326f] 26
/model/layers.32/mlp/gate_proj/MatMul_smooth_mul multiply [-2.44445f, 3.21304f] 26
CustomOp/nn::gateProj<38, 26>129 contrib.epu.quadric_custom_op [-0.278465f, 25.1264f] 26
/model/layers.32/mlp/up_proj/MatMul_smooth_mul multiply [-4.51652f, 3.91687f] 26
CustomOp/linalg::channelwiseQuantMatMul<37>128 contrib.epu.quadric_custom_op [-18.7255f, 22.3183f] 26
/model/layers.32/mlp/Mul multiply [-190.096f, 285.781f] 21
/model/layers.32/mlp/down_proj/MatMul_smooth_mul multiply [-16.2074f, 13.687f] 24
CustomOp/linalg::channelwiseQuantMatMul<34>130 contrib.epu.quadric_custom_op [-67.0631f, 111.421f] 24
/model/layers.32/Add_1 add [-2125.27f, 13672.9f] 16
/model/layers.33/input_layernorm/Mul_1 contrib.epu.rms_norm [-187.593f, 150.813f] 23
/model/layers.33/self_attn/q_proj/MatMul_smooth_mul multiply [-8.17666f, 7.40637f] 25
/model/layers.33/self_attn/k_proj/MatMul_smooth_mul multiply [-7.68112f, 7.72248f] 25
/model/layers.33/self_attn/v_proj/MatMul_smooth_mul multiply [-6.35676f, 6.26976f] 25
CustomOp/linalg::channelwiseQuantMatMul<37>135 contrib.epu.quadric_custom_op [-22.3657f, 115.159f] 24
/model/layers.33/Add add [-2125.11f, 13717f] 16
/model/layers.33/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-29.27f, 27.8766f] 26
/model/layers.33/mlp/gate_proj/MatMul_smooth_mul multiply [-4.70675f, 3.16855f] 26
CustomOp/nn::gateProj<37, 26>133 contrib.epu.quadric_custom_op [-0.278465f, 20.9414f] 26
/model/layers.33/mlp/up_proj/MatMul_smooth_mul multiply [-7.67804f, 4.66365f] 27
CustomOp/linalg::channelwiseQuantMatMul<36>132 contrib.epu.quadric_custom_op [-21.9644f, 28.9324f] 26
/model/layers.33/mlp/Mul multiply [-191.84f, 180.474f] 21
/model/layers.33/mlp/down_proj/MatMul_smooth_mul multiply [-16.8717f, 15.8574f] 24
CustomOp/linalg::channelwiseQuantMatMul<34>134 contrib.epu.quadric_custom_op [-224.43f, 154.382f] 23
/model/layers.33/Add_1 add [-2126.49f, 13736f] 16
/model/layers.34/input_layernorm/Mul_1 contrib.epu.rms_norm [-190.924f, 150.511f] 23
/model/layers.34/self_attn/q_proj/MatMul_smooth_mul multiply [-9.52248f, 9.91184f] 25
/model/layers.34/self_attn/k_proj/MatMul_smooth_mul multiply [-7.38449f, 7.55077f] 26
/model/layers.34/self_attn/v_proj/MatMul_smooth_mul multiply [-5.32747f, 5.83561f] 26
CustomOp/linalg::channelwiseQuantMatMul<36>139 contrib.epu.quadric_custom_op [-134.891f, 252.592f] 23
/model/layers.34/Add add [-2127.57f, 13665.9f] 16
/model/layers.34/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-102.457f, 31.8715f] 24
/model/layers.34/mlp/gate_proj/MatMul_smooth_mul multiply [-11.9706f, 3.27663f] 24
CustomOp/nn::gateProj<35, 25>137 contrib.epu.quadric_custom_op [-0.278465f, 43.9184f] 25
/model/layers.34/mlp/up_proj/MatMul_smooth_mul multiply [-15.5429f, 4.71439f] 24
CustomOp/linalg::channelwiseQuantMatMul<34>136 contrib.epu.quadric_custom_op [-33.824f, 68.3458f] 24
/model/layers.34/mlp/Mul multiply [-1174.96f, 792.102f] 19
/model/layers.34/mlp/down_proj/MatMul_smooth_mul multiply [-110.529f, 78.4391f] 22
CustomOp/linalg::channelwiseQuantMatMul<31>138 contrib.epu.quadric_custom_op [-18980.2f, 553.5f] 16
/model/layers.34/Add_1 add [-7343.94f, 5351.69f] 16
/model/layers.35/input_layernorm/Mul_1 contrib.epu.rms_norm [-298.728f, 121.485f] 22
/model/layers.35/self_attn/q_proj/MatMul_smooth_mul multiply [-40.8404f, 9.94293f] 24
/model/layers.35/self_attn/k_proj/MatMul_smooth_mul multiply [-34.7046f, 8.17391f] 25
/model/layers.35/self_attn/v_proj/MatMul_smooth_mul multiply [-6.21749f, 4.66202f] 25
CustomOp/linalg::channelwiseQuantMatMul<36>143 contrib.epu.quadric_custom_op [-282.141f, 824.661f] 21
/model/layers.35/Add add [-7101.39f, 5072.41f] 16
/model/layers.35/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-601.076f, 47.2325f] 21
/model/layers.35/mlp/gate_proj/MatMul_smooth_mul multiply [-42.4844f, 4.64554f] 22
CustomOp/nn::gateProj<34, 24>141 contrib.epu.quadric_custom_op [-0.278465f, 109.277f] 24
/model/layers.35/mlp/up_proj/MatMul_smooth_mul multiply [-58.8737f, 4.981f] 22
CustomOp/linalg::channelwiseQuantMatMul<33>140 contrib.epu.quadric_custom_op [-159.195f, 162.484f] 23
/model/layers.35/mlp/Mul multiply [-1002.33f, 1755.35f] 16
/model/layers.35/mlp/down_proj/MatMul_smooth_mul multiply [-73.9084f, 97.6708f] 21
CustomOp/linalg::channelwiseQuantMatMul<29>142 contrib.epu.quadric_custom_op [-4812.36f, 1990.75f] 18
/model/layers.35/Add_1 add [-6990.34f, 2581.91f] 16
/model/norm/Mul_1 contrib.epu.rms_norm [-139.062f, 130.451f] 23
/Gather take [-93.9597f, 66.4554f] 23
/lm_head/MatMul_smooth_mul multiply [-5.11149f, 4.01418f] 24
CustomOp/linalg::channelwiseQuantMatMul<36>144 contrib.epu.quadric_custom_op [-15.0257f, 24.2741f] 26
2026-06-19 04:32 - INFO - epu - codegen - START====================build_cpu_runnable_fx_relay
2026-06-19 04:32 - INFO - epu - codegen - START=======================quantize_to_chimera_fx
2026-06-19 04:32 - INFO - epu - codegen - START=================================relay_to_tir
2026-06-19 04:32 - INFO - epu - codegen - START===========================relay_to_epu_relay
2026-06-19 04:32 - INFO - epu - codegen - START==============================adapt_and_order
2026-06-19 04:34 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-06-19 04:34 - INFO - epu - codegen - START=============================plan_lrm_virtual
2026-06-19 04:37 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-06-19 04:37 - INFO - epu - codegen - START===============================lrm_alloc_loop
2026-06-19 04:40 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-06-19 04:40 - INFO - epu - codegen - START================================lrm_splitting
2026-06-19 04:47 - INFO - epu - codegen - START==============================ext_split_relay
2026-06-19 04:52 - INFO - epu - codegen - START====================================build_tir
2026-06-19 04:52 - INFO - epu - chimera_job - Compilation of qwen3_custom_ops_seq1024_QC_P_1d7_16MB_4kB_128GBps_128GBps_16_OFF_x1_x1 successful
Compilation complete!
╒═════════════════════╤═════════════════════════════════════════════════════════════════════════╕
│ Module Name │ qwen3_custom_ops_seq1024_QC_P_1d7_16MB_4kB_128GBps_128GBps_16_OFF_x1_x1 │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────┤
│ ONNX File │ qwen3_custom_ops_seq1024.onnx │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────┤
│ Product Target │ QC-P │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────┤
│ Number of Cores │ 1 │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────┤
│ ISS Clock Frequency │ 1.700 │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────┤
│ L2M Size │ 16MB │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────┤
│ LRM Size │ 4kB │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────┤
│ External Read BW │ 128GBps │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────┤
│ External Write BW │ 128GBps │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────┤
│ MACS per PE │ 16 │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────┤
│ Max L2M │ 0.172MB │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────┤
│ Max LRM │ 0.609kB │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────┤
│ Max Temp Ext Bytes │ 0.583MB │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────┤
│ Network GMACs │ 7.870 │
╘═════════════════════╧═════════════════════════════════════════════════════════════════════════╛
╒═════╤════════╤══════════════════════════╤═══════════════════╤══════════════════════════╤═══════╕
│ │ Type │ Name │ shape │ type │ mse │
╞═════╪════════╪══════════════════════════╪═══════════════════╪══════════════════════════╪═══════╡
│ 0 │ Input │ input_ids │ [1, 1] │ tensor[int32] │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 1 │ Input │ attention_mask │ [1, 1, 1024] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 2 │ Input │ sin │ [1, 1, 128] │ tensor[FixedPoint32<30>] │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 3 │ Input │ cos │ [1, 1, 128] │ tensor[FixedPoint32<30>] │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 4 │ Input │ past_key_values.0.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 5 │ Input │ past_key_values.0.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 6 │ Input │ past_key_values.1.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 7 │ Input │ past_key_values.1.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 8 │ Input │ past_key_values.2.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 9 │ Input │ past_key_values.2.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 10 │ Input │ past_key_values.3.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 11 │ Input │ past_key_values.3.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 12 │ Input │ past_key_values.4.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 13 │ Input │ past_key_values.4.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 14 │ Input │ past_key_values.5.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 15 │ Input │ past_key_values.5.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 16 │ Input │ past_key_values.6.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 17 │ Input │ past_key_values.6.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 18 │ Input │ past_key_values.7.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 19 │ Input │ past_key_values.7.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 20 │ Input │ past_key_values.8.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 21 │ Input │ past_key_values.8.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 22 │ Input │ past_key_values.9.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 23 │ Input │ past_key_values.9.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 24 │ Input │ past_key_values.10.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 25 │ Input │ past_key_values.10.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 26 │ Input │ past_key_values.11.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 27 │ Input │ past_key_values.11.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 28 │ Input │ past_key_values.12.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 29 │ Input │ past_key_values.12.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 30 │ Input │ past_key_values.13.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 31 │ Input │ past_key_values.13.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 32 │ Input │ past_key_values.14.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 33 │ Input │ past_key_values.14.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 34 │ Input │ past_key_values.15.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 35 │ Input │ past_key_values.15.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 36 │ Input │ past_key_values.16.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 37 │ Input │ past_key_values.16.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 38 │ Input │ past_key_values.17.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 39 │ Input │ past_key_values.17.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 40 │ Input │ past_key_values.18.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 41 │ Input │ past_key_values.18.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 42 │ Input │ past_key_values.19.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 43 │ Input │ past_key_values.19.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 44 │ Input │ past_key_values.20.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 45 │ Input │ past_key_values.20.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 46 │ Input │ past_key_values.21.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 47 │ Input │ past_key_values.21.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 48 │ Input │ past_key_values.22.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 49 │ Input │ past_key_values.22.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 50 │ Input │ past_key_values.23.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 51 │ Input │ past_key_values.23.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 52 │ Input │ past_key_values.24.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 53 │ Input │ past_key_values.24.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 54 │ Input │ past_key_values.25.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 55 │ Input │ past_key_values.25.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 56 │ Input │ past_key_values.26.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 57 │ Input │ past_key_values.26.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 58 │ Input │ past_key_values.27.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 59 │ Input │ past_key_values.27.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 60 │ Input │ past_key_values.28.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 61 │ Input │ past_key_values.28.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 62 │ Input │ past_key_values.29.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 63 │ Input │ past_key_values.29.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 64 │ Input │ past_key_values.30.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 65 │ Input │ past_key_values.30.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 66 │ Input │ past_key_values.31.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 67 │ Input │ past_key_values.31.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 68 │ Input │ past_key_values.32.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 69 │ Input │ past_key_values.32.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 70 │ Input │ past_key_values.33.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 71 │ Input │ past_key_values.33.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 72 │ Input │ past_key_values.34.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 73 │ Input │ past_key_values.34.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 74 │ Input │ past_key_values.35.key │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 75 │ Input │ past_key_values.35.value │ [1, 8, 1023, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 76 │ Output │ logits │ [1, 151936] │ tensor[FixedPoint32<26>] │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 77 │ Output │ present.0.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 78 │ Output │ present.0.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 79 │ Output │ present.1.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 80 │ Output │ present.1.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 81 │ Output │ present.2.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 82 │ Output │ present.2.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 83 │ Output │ present.3.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 84 │ Output │ present.3.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 85 │ Output │ present.4.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 86 │ Output │ present.4.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 87 │ Output │ present.5.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 88 │ Output │ present.5.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 89 │ Output │ present.6.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 90 │ Output │ present.6.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 91 │ Output │ present.7.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 92 │ Output │ present.7.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 93 │ Output │ present.8.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 94 │ Output │ present.8.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 95 │ Output │ present.9.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 96 │ Output │ present.9.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 97 │ Output │ present.10.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 98 │ Output │ present.10.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 99 │ Output │ present.11.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 100 │ Output │ present.11.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 101 │ Output │ present.12.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 102 │ Output │ present.12.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 103 │ Output │ present.13.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 104 │ Output │ present.13.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 105 │ Output │ present.14.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 106 │ Output │ present.14.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 107 │ Output │ present.15.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 108 │ Output │ present.15.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 109 │ Output │ present.16.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 110 │ Output │ present.16.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 111 │ Output │ present.17.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 112 │ Output │ present.17.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 113 │ Output │ present.18.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 114 │ Output │ present.18.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 115 │ Output │ present.19.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 116 │ Output │ present.19.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 117 │ Output │ present.20.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 118 │ Output │ present.20.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 119 │ Output │ present.21.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 120 │ Output │ present.21.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 121 │ Output │ present.22.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 122 │ Output │ present.22.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 123 │ Output │ present.23.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 124 │ Output │ present.23.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 125 │ Output │ present.24.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 126 │ Output │ present.24.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 127 │ Output │ present.25.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 128 │ Output │ present.25.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 129 │ Output │ present.26.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 130 │ Output │ present.26.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 131 │ Output │ present.27.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 132 │ Output │ present.27.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 133 │ Output │ present.28.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 134 │ Output │ present.28.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 135 │ Output │ present.29.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 136 │ Output │ present.29.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 137 │ Output │ present.30.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 138 │ Output │ present.30.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 139 │ Output │ present.31.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 140 │ Output │ present.31.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 141 │ Output │ present.32.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 142 │ Output │ present.32.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 143 │ Output │ present.33.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 144 │ Output │ present.33.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 145 │ Output │ present.34.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 146 │ Output │ present.34.value │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 147 │ Output │ present.35.key │ [1, 8, 1024, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼───────────────────┼──────────────────────────┼───────┤
│ 148 │ Output │ present.35.value │ [1, 8, 1024, 128] │ n/a │ n/a │
╘═════╧════════╧══════════════════════════╧═══════════════════╧══════════════════════════╧═══════╛
Step 5: Compile C++ to Assembly and Run on ISS
Now we compile the CGC-generated C++ code to assembly using the Quadric LLVM compiler with the Chimera GPNPU backend, then execute it on the Instruction Set Simulator (ISS) in a 4-core multicore configuration.
Compilation Pipeline:
- C++ → Assembly: Quadric LLVM with Chimera GPNPU backend
- Execution: 4-core ISS with the following configuration:
- 4x QC-P cores
- 16 MACs per PE
- 2MB OCM per core
- 1 GHz clock frequency
- 24 GB/s memory bandwidth per core (96 GB/s total ÷ 4 cores)
- 256-bit DDR AXI width
Setup
The qwen3.cpp file wraps the CGC-generated code in a ready-to-use autoregressive runner for convenience. The body of the CGC-generated code can be found in the ccl_build folder, but has been replicated in qwen3_helpers.hpp. This handles all the complexity of token-by-token generation, KV cache management, and top-K sampling.
Copy Model Weights
First, we need to copy the model weights generated by the ChimeraJob from the ccl_build folder to the current working directory.
import shutil
from pathlib import Path
import glob
## Find const_tensor_data.bin recursively in ccl_build/qwen3_custom_ops_seq1024*
const_tensor_files = glob.glob(
"ccl_build/qwen3_custom_ops_seq1024*/**/const_tensor_data.bin", recursive=True
)
if not const_tensor_files:
print("Error: const_tensor_data.bin not found in ccl_build/qwen3_custom_ops_seq1024*")
else:
ccl_build_path = Path(const_tensor_files[0])
shutil.move(ccl_build_path, "const_tensor_data.bin")
print(f"✓ Moved const_tensor_data.bin from {ccl_build_path}")
✓ Moved const_tensor_data.bin from ccl_build/qwen3_custom_ops_seq1024_QC_P_1d7_16MB_4kB_128GBps_128GBps_16_OFF_x1_x1/build/const_tensor_data.bin
Create Prompt
Prepare the input prompt and tokenize it for inference using the QWEN3 tokenizer.
## Create virtual environment to install QWEN3 compatible transformers
!python3 -m venv qwen_transformers_env
!qwen_transformers_env/bin/pip install --extra-index-url https://download.pytorch.org/whl/cpu 'numpy<1.25' transformers==4.57.1 tokenizers==0.22.1 torch==2.6.0 --quiet
!qwen_transformers_env/bin/python3 prepare_inputs.py --prompt "The tallest" --new_tokens 1
[ 785 81617]
Prompt length: 2
Compile and Execute
The sdk source command compiles the C++ code using the Quadric LLVM toolchain and runs it on the ISS.
Key Parameters:
--target QC-P: Target Quadric Chimera Processor--num-cores 4: 4-core multicore configuration--ocm-size 2MB: 2MB on-chip memory per core--macs-per-pe 16: 16 multiply-accumulate units per PE--clock-freq-ghz 1: 1 GHz clock frequency--ext-read-bw/--ext-write-bw 24GBps: Memory bandwidth per core (96 GB/s total ÷ 4)--ddr-axi-width 256: 256-bit DDR AXI interface width
Compile Time: ~10 minutes
Runtime: ~30 minutes
%%bash
set -euo pipefail
sdk source -v qwen3.cpp \
--include-cgc-headers \
--target QC-P \
--num-cores 4 \
--ocm-size 2MB \
--macs-per-pe 16 \
--clock-freq-ghz 1 \
--ext-read-bw 24GBps \
--ext-write-bw 24GBps \
--ddr-axi-width 256 \
--quiet
2026-06-19 04:53 - DEBUG - sdk - cli - Executing command: cmake CMakeLists.txt -B /quadric/sdk-cli/examples/models/qwen/qwen3_8b/qwen3_QC-P_2MB_4kB_24GBps_24GBps/build -DNUM_GPNPUS=4 -DNUM_CORES=16 -DNUM_BORDERS=2 -DEPU_VERSION=2.0.0 -DQLLVM_ROOT_PATH=/quadric/llvm -DOCM_SIZE_KIBIBYTES=2048 -DNUM_PE_MACS=16 -DASSERT_MLS_WIDTH_LINE_ALIGN=ON -DHARDWARE_TARGET=OFF
2026-06-19 04:53 - DEBUG - sdk - cli - Executing command: make -j8
[SDK-CLI] : Executing on QC-P simulator
2026-06-19 04:56 - DEBUG - sdk - cli - Executing command: ./qwen3_host -c --ddrRdBwTotal 196608.0 --ddrWrBwTotal 196608.0 --ddrAxiWidth 256 --instMemDepth 1310720 --ocmSize 2097152 --cycleTimeNS 1.0 --no-check --ddrRdAvgPct 100 --ddrRdMaxPct 100 --ddrWrAvgPct 100 --ddrWrMaxPct 100 --postKernelFlowTimeoutCycles 4000000 --clusterSize 4 --numClusters 1
2026-06-19 05:08 - WARNING - epu - core - No profile.json found. Not able to show performance results.
2026-06-19 05:08 - INFO - epu - core - If you would like to see performance results, add profiling statements to your source code.
[SDK-CLI] : Execution completed.
View Multicore Profile
Display the performance profile from the ISS execution. This shows cycle counts for different operations (compute, data movement, MAC operations, etc.) and calculates the tokens per second throughput.
Note: The prompt "The tallest" contains 2 tokens. To calculate the actual tokens/sec throughput, multiply the "Executions/second" value by the number of tokens in your prompt:
Example: 10 Executions/second × 2 tokens = 20 tokens/sec
from tvm.contrib.epu.chimera_job import core
import glob
## Find and plot the profile from core 0
profile_file = glob.glob("qwen3_QC-P*/**/profile_core0.json", recursive=True)[0]
core._plot_profile_results(profile_file, clock_freq=1 * 1e9)
[SDK-CLI] : TotalCycles: 100,537,784
[SDK-CLI] : Executions/second: 10
compute : ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 18.135M
data_array : ▇▇▇▇▇▇▇ 7.149M
mac : ▇ 1.077M
data_ocm : ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 44.973M
data_external: ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 29.202M
{'compute': 18134515,
'data_array': 7148838,
'mac': 1077354,
'data_ocm': 44973034,
'data_external': 29202446,
'total': 100537784,
'ExtBytes': {'LOAD': 1923425800, 'STORE': 5671688}}
Decode Output Tokens
Now let's decode the generated tokens back to human-readable text using the QWEN3 tokenizer.
!qwen_transformers_env/bin/python3 decode_outputs.py
Token ids: [883]
Generated text: man
