NOTE: The Jupyter Notebook below is included in the Chimera SDK and can be run interactively by running the following CLI command:
$ quadric sdk notebook
From the Jupyter Notebook window in your browser, select the notebook named /quadric/sdk-cli/examples/models/qwen/qwen2.5-1.5b/qwen2.5.ipynb.
Running DeepSeek-R1-Distill-Qwen-1.5B on Quadric Hardware
This notebook demonstrates running DeepSeek-R1-Distill-Qwen-1.5B on Quadric hardware - a distilled version of DeepSeek's R1 reasoning model using the QWEN 2.5 1.5B architecture, combining strong reasoning capabilities with a compact size suitable for edge deployment.
Pipeline:
- Lowering the pre-quantized model to C++ using Chimera Graph Compiler (CGC)
- Compiling C++ to assembly using Quadric LLVM compiler
- Executing on Instruction Set Simulator (ISS) with QC-N hardware configuration
- Autoregressive inference with KV caching
Model Optimizations:
- INT8 asymmetric per-channel quantization
- Smooth quantization for improved activation distribution
- QK MatMul exclusion from quantization to preserve softmax accuracy
- Custom attention operators for efficient KV caching
Background: Model Quantization
The pre-quantized model was prepared using two key techniques:
1. Smooth Quantization
Activation outliers between nodes can cause quantization scales to become very large, reducing precision. Smooth quantization addresses this by inserting scaling operations into the ONNX graph before PTQ (Post-Training Quantization). This shifts quantization difficulty into the weights, smoothing activation outliers and preserving greater resolution in intermediate values.
2. QK MatMul Exclusion
The Query-Key matrix multiplication feeds into softmax, which is highly sensitive due to its exponential nature. Quantizing the QK MatMul can amplify small numerical differences and degrade accuracy. By excluding this operation from quantization, we maintain float32 precision where it matters most for model quality.
Step 0: Install Packages
!pip3 install -r ../../../requirements.txt -q
[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv[0m[33m
[0m
Step 1: Download Pre-Quantized DeepSeek R1 Distill Model
Download the INT8 quantized DeepSeek-R1-Distill-Qwen-1.5B model from S3. This includes:
qwen2.5_1.5b_int8_q.onnx- The quantized ONNX model with custom opsqwen2.5_1.5b_int8_q.tranges- Tensor range information for the quantized modela3bfb1d2-4d56-11f0-9e8c-eaea990809fc- Runtime data file
from urllib.request import urlretrieve
base_url = "https://sdk-cli-models.s3.amazonaws.com/"
files = [
"qwen2.5_1.5b_int8_q.onnx",
"qwen2.5_1.5b_int8_q.tranges",
"a3bfb1d2-4d56-11f0-9e8c-eaea990809fc",
]
for filename in files:
print(f"Downloading {filename}...")
urlretrieve(base_url + filename, filename)
print(f" ✓ Downloaded {filename}")
print("\nAll files downloaded successfully!")
Downloading qwen2.5_1.5b_int8_q.onnx...
✓ Downloaded qwen2.5_1.5b_int8_q.onnx
Downloading qwen2.5_1.5b_int8_q.tranges...
✓ Downloaded qwen2.5_1.5b_int8_q.tranges
Downloading a3bfb1d2-4d56-11f0-9e8c-eaea990809fc...
✓ Downloaded a3bfb1d2-4d56-11f0-9e8c-eaea990809fc
All files downloaded successfully!
Step 2: Fix Shapes for Autoregressive Execution
The downloaded ONNX model has dynamic shapes. We need to fix these shapes for autoregressive inference with a sequence length of 512 tokens.
Runtime: ~30-60 seconds
from fix_shapes import fix_shapes
## Input: downloaded quantized model with dynamic shapes
in_onnx_path = "qwen2.5_1.5b_int8_q.onnx"
tranges_path = "qwen2.5_1.5b_int8_q.tranges"
## Output: shape-fixed model for autoregressive inference
fixed_onnx_path = "qwen2.5_1.5b_int8_q_seq512.onnx"
seq_length = 512
print(f"Fixing shapes for sequence length {seq_length}...")
fix_shapes(in_onnx_path, fixed_onnx_path, seq_length)
print(f"✓ Fixed shapes saved to {fixed_onnx_path}")
Fixing shapes for sequence length 512...
✓ Fixed shapes saved to qwen2.5_1.5b_int8_q_seq512.onnx
Step 3: Replace Attention Blocks with Custom Ops
To maximize performance with KV caching, we replace the standard attention blocks with Quadric-optimized custom operators. This step:
- Identifies multi-headed attention patterns in the ONNX graph
- Replaces them with fused custom attention ops
- Optimizes memory access patterns for the Quadric hardware
Runtime: ~5-10 minutes
from match_att import multi_headed_attention_replacer
## Input: shape-fixed model
in_onnx_path = "qwen2.5_1.5b_int8_q_seq512.onnx"
## Output: model with custom attention ops
custom_op_onnx_path = "qwen2.5_1.5b_int8_q_seq512_custom_ops.onnx"
## QWEN 2.5 1.5B architecture parameters
num_heads = 12
embed_dim = 1536
seq_length = 512
print(f"Replacing attention blocks with custom ops...")
multi_headed_attention_replacer(in_onnx_path, custom_op_onnx_path, num_heads, embed_dim, seq_length)
print(f"✓ Custom ops model saved to {custom_op_onnx_path}")
Replacing attention blocks with custom ops...
✓ Custom ops model saved to qwen2.5_1.5b_int8_q_seq512_custom_ops.onnx
Step 4: Lower ONNX to C++ with Chimera Graph Compiler (CGC)
The Chimera Graph Compiler (CGC) converts the ONNX graph into optimized C++ code for the Quadric platform. This process:
- Analyzes the computational graph and schedules operations
- Generates memory-efficient code that fits within the 4MB OCM constraint
- Optimizes for the QC-N hardware configuration (8 MACs per PE)
Hardware Configuration:
- Product: QC-N (Quadric Chimera Neural)
- Target Language: QIL (Quadric Intermediate Language)
- OCM Size: 4MB on-chip memory
- MACs per PE: 8 multiply-accumulate units per processing element
Note: This compilation requires significant RAM and may take 20-30 minutes due to the model size (1.5B parameters).
import resource
## Increase stack size for large model compilation
resource.setrlimit(resource.RLIMIT_STACK, (32768 * 1024, 32768 * 1024))
from tvm.contrib.epu.chimera_job.chimera_job import ChimeraJob
from tvm.contrib.epu.chimera_job.hw_config import HWConfig
## Use the custom-op model
model_path = "qwen2.5_1.5b_int8_q_seq512_custom_ops.onnx"
trange_file = "qwen2.5_1.5b_int8_q.tranges"
## Configure hardware target
hw_config = HWConfig(
product="QC-N", # Quadric product type
ocm_size="4MB", # On-chip memory
macs_per_pe=8, # MACs per processing element
)
## Specify I/O tensors to ignore during compilation (KV cache management)
num_decoders = 28
io_to_ignore = ["attention_mask"]
for i in range(num_decoders):
io_to_ignore.extend(
[
f"present.{i}.key",
f"present.{i}.value",
f"past_key_values.{i}.key",
f"past_key_values.{i}.value",
]
)
## Create and run CGC compilation job
cgc_job = ChimeraJob(
model_path,
hw_config=hw_config,
trange_file=trange_file,
target_lang="QIL", # Quadric Intermediate Language
io_to_ignore=io_to_ignore,
)
print("Starting CGC compilation (this will take ~20-30 minutes)...")
cgc_job.compile()
print("\nCompilation complete!")
print(cgc_job)
Starting CGC compilation (this will take ~20-30 minutes)...
2026-07-18 12:19 - INFO - epu - chimera_job - START==================================onnx_ingest
2026-07-18 12:19 - INFO - epu - chimera_job - Numerical ranges provided
/usr/local/lib/python3.10/dist-packages/tvm/relay/frontend/onnx.py:6270: UserWarning: This protobuf of onnx model is too large (>2GB). Call check_model with model path instead.
warnings.warn(str(e))
2026-07-18 12:21 - INFO - epu - codegen - START===============================optimize_relay
2026-07-18 12:21 - INFO - epu - codegen - START====================quantize_to_cpu_runnable_fx
2026-07-18 12:21 - INFO - epu - fx -
Source name Op Output 0 Range Output 0 Frac Bits
------------------------------------------------------------------- ---------------------- ----------------------- --------------------
/model/embed_tokens/Gather contrib.epu.embedding [-0.330078f, 0.257812f] 31
/model/layers.0/input_layernorm/Mul_1 contrib.epu.rms_norm [-6.16894f, 5.36183f] 28
gemm_input_reshape_token_8 reshape [-6.16894f, 5.36183f] 28
/model/layers.0/self_attn/q_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-2.43474f, 1.84163f] 27
/model/layers.0/self_attn/k_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-1.05282f, 1.02692f] 27
/model/layers.0/self_attn/v_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-0.745194f, 0.869626f] 28
/model/layers.0/self_attn/Unsqueeze expand_dims [-1f, 1f] 30
/model/layers.0/self_attn/Slice strided_slice [-1f, 1f] 30
/model/layers.0/self_attn/Unsqueeze_1 expand_dims [-1f, 1f] 30
/model/layers.0/self_attn/Slice_2 strided_slice [-1f, 1f] 30
/model/layers.0/self_attn/Reshape_7_output_0_DequantizeLinear contrib.epu.dequantize [-3.65169f, 5.09632f] 28
/model/layers.0/self_attn/o_proj/MatMul_smooth_mul multiply [-1.14048f, 1.67202f] 27
/model/layers.0/self_attn/o_proj/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-11.8113f, 10.9742f] 27
/model/layers.0/Add add [-11.8796f, 10.8678f] 27
/model/layers.0/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-1.96867f, 3.41459f] 28
/model/layers.0/mlp/gate_proj/MatMul_smooth_mul multiply [-1.15413f, 1.61562f] 25
/model/layers.0/mlp/up_proj/MatMul_smooth_mul multiply [-0.933593f, 0.850626f] 25
/model/layers.0/mlp/Mul_output_0_DequantizeLinear contrib.epu.dequantize [-15.9686f, 13.4539f] 26
/model/layers.0/mlp/down_proj/MatMul_smooth_mul multiply [-13.2742f, 11.052f] 26
/model/layers.0/mlp/down_proj/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-13.2494f, 13.5752f] 27
/model/layers.0/Add_1 add [-21.9621f, 21.7684f] 26
/model/layers.1/input_layernorm/Mul_1 contrib.epu.rms_norm [-5.55017f, 3.63564f] 28
gemm_input_reshape_token_26 reshape [-5.55017f, 3.63564f] 28
/model/layers.1/self_attn/q_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-1.3183f, 1.00542f] 24
/model/layers.1/self_attn/k_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-1.12512f, 0.930116f] 24
/model/layers.1/self_attn/v_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-0.615321f, 0.67785f] 24
/model/layers.1/self_attn/Reshape_7_output_0_DequantizeLinear contrib.epu.dequantize [-1.55083f, 1.28428f] 30
/model/layers.1/self_attn/o_proj/MatMul_smooth_mul multiply [-0.761665f, 0.719532f] 29
/model/layers.1/self_attn/o_proj/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-4.23805f, 3.7083f] 28
/model/layers.1/Add add [-20.2735f, 21.3049f] 26
/model/layers.1/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-17.0948f, 71.6944f] 24
/model/layers.1/mlp/gate_proj/MatMul_smooth_mul multiply [-1.78584f, 5.41087f] 22
/model/layers.1/mlp/up_proj/MatMul_smooth_mul multiply [-2.71376f, 4.21087f] 22
/model/layers.1/mlp/Mul_output_0_DequantizeLinear contrib.epu.dequantize [-395.95f, 102.181f] 22
/model/layers.1/mlp/down_proj/MatMul_smooth_mul multiply [-25.7678f, 29.1186f] 20
/model/layers.1/mlp/down_proj/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-760.706f, 827.434f] 21
/model/layers.1/Add_1 add [-780.979f, 833.834f] 21
/model/layers.2/input_layernorm/Mul_1 contrib.epu.rms_norm [-4.87579f, 4.91903f] 28
gemm_input_reshape_token_44 reshape [-4.87579f, 4.91903f] 28
/model/layers.2/self_attn/q_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-1.46368f, 1.57376f] 28
/model/layers.2/self_attn/k_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-1.51925f, 1.62687f] 28
/model/layers.2/self_attn/v_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-0.998105f, 1.05973f] 28
/model/layers.2/self_attn/Reshape_7_output_0_DequantizeLinear contrib.epu.dequantize [-2.13828f, 3.52678f] 29
/model/layers.2/self_attn/o_proj/MatMul_smooth_mul multiply [-0.970765f, 1.86286f] 28
/model/layers.2/self_attn/o_proj/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-3.36552f, 3.36552f] 28
/model/layers.2/Add add [-780.121f, 833.834f] 21
/model/layers.2/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-20.8502f, 70.2273f] 24
/model/layers.2/mlp/gate_proj/MatMul_smooth_mul multiply [-1.99742f, 3.28327f] 21
/model/layers.2/mlp/up_proj/MatMul_smooth_mul multiply [-2.07808f, 3.60432f] 21
/model/layers.2/mlp/Mul_output_0_DequantizeLinear contrib.epu.dequantize [-1258.77f, 1248.86f] 20
/model/layers.2/mlp/down_proj/MatMul_smooth_mul multiply [-24.8041f, 25.7793f] 19
/model/layers.2/mlp/down_proj/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-1226.76f, 1187.19f] 20
/model/layers.2/Add_1 add [-2006.89f, 1961.93f] 20
/model/layers.3/input_layernorm/Mul_1 contrib.epu.rms_norm [-5.79104f, 6.42236f] 28
gemm_input_reshape_token_62 reshape [-5.79104f, 6.42236f] 28
/model/layers.3/self_attn/q_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-1.80061f, 1.98227f] 25
/model/layers.3/self_attn/k_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-1.45638f, 1.72616f] 25
/model/layers.3/self_attn/v_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-1.38451f, 1.64326f] 26
/model/layers.3/self_attn/Reshape_7_output_0_DequantizeLinear contrib.epu.dequantize [-1.88898f, 2.2213f] 29
/model/layers.3/self_attn/o_proj/MatMul_smooth_mul multiply [-0.924433f, 0.862205f] 28
/model/layers.3/self_attn/o_proj/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-3.5941f, 2.88938f] 28
/model/layers.3/Add add [-2007.84f, 1962.5f] 20
/model/layers.3/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-24.9679f, 48.8285f] 25
/model/layers.3/mlp/gate_proj/MatMul_smooth_mul multiply [-2.23f, 3.28765f] 22
/model/layers.3/mlp/up_proj/MatMul_smooth_mul multiply [-1.87903f, 2.56656f] 23
/model/layers.3/mlp/Mul_output_0_DequantizeLinear contrib.epu.dequantize [-23.5597f, 34.7917f] 25
/model/layers.3/mlp/down_proj/MatMul_smooth_mul multiply [-12.2371f, 16.5247f] 25
/model/layers.3/mlp/down_proj/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-18.848f, 16.1808f] 26
/model/layers.3/Add_1 add [-2026.68f, 1978.68f] 20
/model/layers.4/input_layernorm/Mul_1 contrib.epu.rms_norm [-5.27614f, 6.12898f] 28
gemm_input_reshape_token_80 reshape [-5.27614f, 6.12898f] 28
/model/layers.4/self_attn/q_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-1.61238f, 1.69382f] 27
/model/layers.4/self_attn/k_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-1.52416f, 1.65165f] 27
/model/layers.4/self_attn/v_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-1.24227f, 1.444f] 28
/model/layers.4/self_attn/Reshape_7_output_0_DequantizeLinear contrib.epu.dequantize [-2.43022f, 2.55073f] 29
/model/layers.4/self_attn/o_proj/MatMul_smooth_mul multiply [-1.27798f, 1.41695f] 28
/model/layers.4/self_attn/o_proj/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-3.24083f, 3.82077f] 28
/model/layers.4/Add add [-2026.51f, 1978.37f] 20
/model/layers.4/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-10.5427f, 19.1072f] 26
/model/layers.4/mlp/gate_proj/MatMul_smooth_mul multiply [-2.18714f, 2.92659f] 22
/model/layers.4/mlp/up_proj/MatMul_smooth_mul multiply [-1.78774f, 2.22912f] 22
/model/layers.4/mlp/Mul_output_0_DequantizeLinear contrib.epu.dequantize [-40.543f, 22.8055f] 25
/model/layers.4/mlp/down_proj/MatMul_smooth_mul multiply [-9.29053f, 7.83864f] 25
/model/layers.4/mlp/down_proj/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-58.8981f, 52.6983f] 24
/model/layers.4/Add_1 add [-2085.41f, 2031.07f] 19
/model/layers.5/input_layernorm/Mul_1 contrib.epu.rms_norm [-4.9416f, 5.99741f] 28
gemm_input_reshape_token_98 reshape [-4.9416f, 5.99741f] 28
/model/layers.5/self_attn/q_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-1.63345f, 1.43245f] 28
/model/layers.5/self_attn/k_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-1.60881f, 1.55023f] 28
/model/layers.5/self_attn/v_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-1.0755f, 1.08637f] 28
/model/layers.5/self_attn/Reshape_7_output_0_DequantizeLinear contrib.epu.dequantize [-3.09797f, 4.17703f] 28
/model/layers.5/self_attn/o_proj/MatMul_smooth_mul multiply [-1.72641f, 1.28594f] 27
/model/layers.5/self_attn/o_proj/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-5.04504f, 5.09402f] 28
/model/layers.5/Add add [-2080.32f, 2027.4f] 19
/model/layers.5/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-25.3382f, 58.1099f] 25
/model/layers.5/mlp/gate_proj/MatMul_smooth_mul multiply [-2.22005f, 3.19162f] 22
/model/layers.5/mlp/up_proj/MatMul_smooth_mul multiply [-2.30343f, 2.41094f] 22
/model/layers.5/mlp/Mul_output_0_DequantizeLinear contrib.epu.dequantize [-21.7299f, 25.2067f] 25
/model/layers.5/mlp/down_proj/MatMul_smooth_mul multiply [-10.7392f, 6.83383f] 25
/model/layers.5/mlp/down_proj/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-27.2843f, 20.25f] 26
/model/layers.5/Add_1 add [-2107.6f, 2047.65f] 19
/model/layers.6/input_layernorm/Mul_1 contrib.epu.rms_norm [-11.2136f, 10.0696f] 27
gemm_input_reshape_token_116 reshape [-11.2136f, 10.0696f] 27
/model/layers.6/self_attn/q_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-2.34615f, 1.94163f] 27
/model/layers.6/self_attn/k_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-1.83833f, 1.60455f] 27
/model/layers.6/self_attn/v_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-1.05462f, 1.1697f] 28
/model/layers.6/self_attn/Reshape_7_output_0_DequantizeLinear contrib.epu.dequantize [-2.78058f, 4.5681f] 28
/model/layers.6/self_attn/o_proj/MatMul_smooth_mul multiply [-1.88299f, 2.62248f] 27
/model/layers.6/self_attn/o_proj/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-3.34094f, 3.86296f] 28
/model/layers.6/Add add [-2107.39f, 2047.72f] 19
/model/layers.6/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-9.23895f, 13.0439f] 27
/model/layers.6/mlp/gate_proj/MatMul_smooth_mul multiply [-1.74524f, 2.39069f] 24
/model/layers.6/mlp/up_proj/MatMul_smooth_mul multiply [-1.31547f, 1.22218f] 24
/model/layers.6/mlp/Mul_output_0_DequantizeLinear contrib.epu.dequantize [-14.5292f, 10.4428f] 27
/model/layers.6/mlp/down_proj/MatMul_smooth_mul multiply [-4.86472f, 6.36353f] 26
/model/layers.6/mlp/down_proj/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-22.9716f, 15.9472f] 26
/model/layers.6/Add_1 add [-2130.37f, 2063.66f] 19
/model/layers.7/input_layernorm/Mul_1 contrib.epu.rms_norm [-5.42681f, 5.26166f] 28
gemm_input_reshape_token_134 reshape [-5.42681f, 5.26166f] 28
/model/layers.7/self_attn/q_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-1.80911f, 1.53873f] 28
/model/layers.7/self_attn/k_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-1.10949f, 1.20099f] 28
/model/layers.7/self_attn/v_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-1.01743f, 0.921791f] 28
/model/layers.7/self_attn/Reshape_7_output_0_DequantizeLinear contrib.epu.dequantize [-2.72976f, 2.21793f] 29
/model/layers.7/self_attn/o_proj/MatMul_smooth_mul multiply [-1.23394f, 1.50976f] 28
/model/layers.7/self_attn/o_proj/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-5.4746f, 7.09463f] 28
/model/layers.7/Add add [-2123.94f, 2061.21f] 19
/model/layers.7/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-9.01009f, 11.3839f] 27
/model/layers.7/mlp/gate_proj/MatMul_smooth_mul multiply [-1.41076f, 2.28368f] 24
/model/layers.7/mlp/up_proj/MatMul_smooth_mul multiply [-1.40965f, 1.49561f] 24
/model/layers.7/mlp/Mul_output_0_DequantizeLinear contrib.epu.dequantize [-14.7443f, 14.6291f] 27
/model/layers.7/mlp/down_proj/MatMul_smooth_mul multiply [-5.71543f, 5.93455f] 26
/model/layers.7/mlp/down_proj/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-25.7086f, 17.6348f] 26
/model/layers.7/Add_1 add [-2149.65f, 2078.84f] 19
/model/layers.8/input_layernorm/Mul_1 contrib.epu.rms_norm [-12.1917f, 9.54448f] 27
gemm_input_reshape_token_152 reshape [-12.1917f, 9.54448f] 27
/model/layers.8/self_attn/q_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-3.17668f, 2.65074f] 27
/model/layers.8/self_attn/k_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-2.26969f, 1.90303f] 27
/model/layers.8/self_attn/v_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-1.03709f, 0.897022f] 27
/model/layers.8/self_attn/Reshape_7_output_0_DequantizeLinear contrib.epu.dequantize [-2.60976f, 4.04195f] 28
/model/layers.8/self_attn/o_proj/MatMul_smooth_mul multiply [-1.50846f, 1.87256f] 27
/model/layers.8/self_attn/o_proj/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-4.37727f, 6.90261f] 28
/model/layers.8/Add add [-2145.55f, 2076.03f] 19
/model/layers.8/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-9.55142f, 11.3939f] 27
/model/layers.8/mlp/gate_proj/MatMul_smooth_mul multiply [-1.53432f, 2.0169f] 25
/model/layers.8/mlp/up_proj/MatMul_smooth_mul multiply [-1.22467f, 1.27836f] 25
/model/layers.8/mlp/Mul_output_0_DequantizeLinear contrib.epu.dequantize [-20.546f, 16.9728f] 26
/model/layers.8/mlp/down_proj/MatMul_smooth_mul multiply [-5.16228f, 10.1425f] 26
/model/layers.8/mlp/down_proj/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-28.6082f, 20.4344f] 25
/model/layers.8/Add_1 add [-2174.16f, 2096.47f] 19
/model/layers.9/input_layernorm/Mul_1 contrib.epu.rms_norm [-6.89598f, 7.67325f] 27
gemm_input_reshape_token_170 reshape [-6.89598f, 7.67325f] 27
/model/layers.9/self_attn/q_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-1.81533f, 1.7932f] 28
/model/layers.9/self_attn/k_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-1.24594f, 1.23868f] 28
/model/layers.9/self_attn/v_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-0.944457f, 1.02265f] 28
/model/layers.9/self_attn/Reshape_7_output_0_DequantizeLinear contrib.epu.dequantize [-3.17572f, 3.38921f] 29
/model/layers.9/self_attn/o_proj/MatMul_smooth_mul multiply [-1.84747f, 1.88092f] 28
/model/layers.9/self_attn/o_proj/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-8.37671f, 9.95722f] 27
/model/layers.9/Add add [-2164.2f, 2090.15f] 19
/model/layers.9/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-9.04102f, 9.53634f] 27
/model/layers.9/mlp/gate_proj/MatMul_smooth_mul multiply [-1.40424f, 1.72821f] 23
/model/layers.9/mlp/up_proj/MatMul_smooth_mul multiply [-1.04379f, 1.21108f] 23
/model/layers.9/mlp/Mul_output_0_DequantizeLinear contrib.epu.dequantize [-15.1117f, 16.5447f] 26
/model/layers.9/mlp/down_proj/MatMul_smooth_mul multiply [-6.16992f, 5.72944f] 26
/model/layers.9/mlp/down_proj/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-29.2058f, 16.583f] 26
/model/layers.9/Add_1 add [-2193.41f, 2106.73f] 19
/model/layers.10/input_layernorm/Mul_1 contrib.epu.rms_norm [-8.01522f, 6.93579f] 27
gemm_input_reshape_token_188 reshape [-8.01522f, 6.93579f] 27
/model/layers.10/self_attn/q_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-1.58721f, 1.63432f] 27
/model/layers.10/self_attn/k_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-1.6552f, 1.56848f] 28
/model/layers.10/self_attn/v_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-0.929301f, 0.923449f] 28
/model/layers.10/self_attn/Reshape_7_output_0_DequantizeLinear contrib.epu.dequantize [-3.21382f, 3.80411f] 28
/model/layers.10/self_attn/o_proj/MatMul_smooth_mul multiply [-1.63538f, 2.09304f] 28
/model/layers.10/self_attn/o_proj/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-7.25819f, 12.6273f] 27
/model/layers.10/Add add [-2181.58f, 2099.47f] 19
/model/layers.10/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-8.54981f, 8.63469f] 27
/model/layers.10/mlp/gate_proj/MatMul_smooth_mul multiply [-1.34082f, 2.03379f] 25
/model/layers.10/mlp/up_proj/MatMul_smooth_mul multiply [-1.47111f, 1.32283f] 26
/model/layers.10/mlp/Mul_output_0_DequantizeLinear contrib.epu.dequantize [-16.8417f, 9.21029f] 26
/model/layers.10/mlp/down_proj/MatMul_smooth_mul multiply [-5.07418f, 6.23921f] 26
/model/layers.10/mlp/down_proj/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-30.4041f, 17.7787f] 25
/model/layers.10/Add_1 add [-2211.98f, 2117.25f] 19
/model/layers.11/input_layernorm/Mul_1 contrib.epu.rms_norm [-7.01215f, 7.44462f] 27
gemm_input_reshape_token_206 reshape [-7.01215f, 7.44462f] 27
/model/layers.11/self_attn/q_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-1.70177f, 1.93335f] 27
/model/layers.11/self_attn/k_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-1.40419f, 1.46376f] 28
/model/layers.11/self_attn/v_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-1.2152f, 1.05015f] 28
/model/layers.11/self_attn/Reshape_7_output_0_DequantizeLinear contrib.epu.dequantize [-2.85657f, 3.94331f] 29
/model/layers.11/self_attn/o_proj/MatMul_smooth_mul multiply [-1.53292f, 2.2954f] 28
/model/layers.11/self_attn/o_proj/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-13.1138f, 18.9765f] 26
/model/layers.11/Add add [-2193.01f, 2105.22f] 19
/model/layers.11/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-9.45189f, 8.30149f] 27
/model/layers.11/mlp/gate_proj/MatMul_smooth_mul multiply [-1.59985f, 1.84719f] 24
/model/layers.11/mlp/up_proj/MatMul_smooth_mul multiply [-1.34874f, 1.56192f] 24
/model/layers.11/mlp/Mul_output_0_DequantizeLinear contrib.epu.dequantize [-7.94531f, 14.2785f] 27
/model/layers.11/mlp/down_proj/MatMul_smooth_mul multiply [-3.86514f, 3.06789f] 26
/model/layers.11/mlp/down_proj/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-35.6372f, 21.5001f] 25
/model/layers.11/Add_1 add [-2228.64f, 2126.72f] 19
/model/layers.12/input_layernorm/Mul_1 contrib.epu.rms_norm [-8.38008f, 7.96865f] 27
gemm_input_reshape_token_224 reshape [-8.38008f, 7.96865f] 27
/model/layers.12/self_attn/q_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-1.773f, 1.85649f] 27
/model/layers.12/self_attn/k_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-1.79468f, 1.50664f] 28
/model/layers.12/self_attn/v_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-1.33194f, 1.17326f] 28
/model/layers.12/self_attn/Reshape_7_output_0_DequantizeLinear contrib.epu.dequantize [-4.06158f, 4.33694f] 28
/model/layers.12/self_attn/o_proj/MatMul_smooth_mul multiply [-1.57039f, 1.88537f] 28
/model/layers.12/self_attn/o_proj/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-12.3192f, 11.6472f] 27
/model/layers.12/Add add [-2217.89f, 2120.33f] 19
/model/layers.12/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-8.5319f, 6.28941f] 27
/model/layers.12/mlp/gate_proj/MatMul_smooth_mul multiply [-1.47613f, 1.58071f] 25
/model/layers.12/mlp/up_proj/MatMul_smooth_mul multiply [-1.30581f, 1.3147f] 25
/model/layers.12/mlp/Mul_output_0_DequantizeLinear contrib.epu.dequantize [-9.45055f, 11.1132f] 27
/model/layers.12/mlp/down_proj/MatMul_smooth_mul multiply [-7.42455f, 7.76539f] 26
/model/layers.12/mlp/down_proj/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-22.8804f, 13.9825f] 26
/model/layers.12/Add_1 add [-2240.77f, 2134.31f] 19
/model/layers.13/input_layernorm/Mul_1 contrib.epu.rms_norm [-6.91517f, 6.2219f] 28
gemm_input_reshape_token_242 reshape [-6.91517f, 6.2219f] 28
/model/layers.13/self_attn/q_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-1.45512f, 1.53288f] 28
/model/layers.13/self_attn/k_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-1.54682f, 1.50705f] 28
/model/layers.13/self_attn/v_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-1.0777f, 1.06585f] 28
/model/layers.13/self_attn/Reshape_7_output_0_DequantizeLinear contrib.epu.dequantize [-4.60797f, 3.95998f] 28
/model/layers.13/self_attn/o_proj/MatMul_smooth_mul multiply [-1.5343f, 1.66657f] 28
/model/layers.13/self_attn/o_proj/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-11.4021f, 11.9675f] 27
/model/layers.13/Add add [-2231.73f, 2128.28f] 19
/model/layers.13/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-7.25405f, 5.74461f] 28
/model/layers.13/mlp/gate_proj/MatMul_smooth_mul multiply [-1.46365f, 1.30891f] 26
/model/layers.13/mlp/up_proj/MatMul_smooth_mul multiply [-1.14316f, 1.3143f] 26
/model/layers.13/mlp/Mul_output_0_DequantizeLinear contrib.epu.dequantize [-10.4464f, 13.6772f] 27
/model/layers.13/mlp/down_proj/MatMul_smooth_mul multiply [-4.7647f, 4.6456f] 26
/model/layers.13/mlp/down_proj/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-24.6669f, 14.1238f] 26
/model/layers.13/Add_1 add [-2256.39f, 2142.41f] 19
/model/layers.14/input_layernorm/Mul_1 contrib.epu.rms_norm [-16.7933f, 28.0791f] 26
gemm_input_reshape_token_260 reshape [-16.7933f, 28.0791f] 26
/model/layers.14/self_attn/q_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-2.23483f, 2.35467f] 26
/model/layers.14/self_attn/k_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-2.29288f, 3.83379f] 26
/model/layers.14/self_attn/v_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-1.33584f, 1.38551f] 26
/model/layers.14/self_attn/Reshape_7_output_0_DequantizeLinear contrib.epu.dequantize [-5.27901f, 3.678f] 28
/model/layers.14/self_attn/o_proj/MatMul_smooth_mul multiply [-3.25403f, 3.79997f] 26
/model/layers.14/self_attn/o_proj/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-9.43766f, 12.8937f] 26
/model/layers.14/Add add [-2248.02f, 2136.56f] 19
/model/layers.14/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-7.69488f, 5.67217f] 28
/model/layers.14/mlp/gate_proj/MatMul_smooth_mul multiply [-1.33701f, 1.46114f] 23
/model/layers.14/mlp/up_proj/MatMul_smooth_mul multiply [-1.32172f, 1.70264f] 23
/model/layers.14/mlp/Mul_output_0_DequantizeLinear contrib.epu.dequantize [-9.80894f, 8.65495f] 27
/model/layers.14/mlp/down_proj/MatMul_smooth_mul multiply [-5.35898f, 6.81198f] 27
/model/layers.14/mlp/down_proj/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-18.1672f, 9.65135f] 26
/model/layers.14/Add_1 add [-2263.35f, 2145.22f] 19
/model/layers.15/input_layernorm/Mul_1 contrib.epu.rms_norm [-9.71374f, 8.51807f] 27
gemm_input_reshape_token_278 reshape [-9.71374f, 8.51807f] 27
/model/layers.15/self_attn/q_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-1.9877f, 1.99683f] 27
/model/layers.15/self_attn/k_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-1.9342f, 1.85542f] 27
/model/layers.15/self_attn/v_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-1.0061f, 1.07851f] 27
/model/layers.15/self_attn/Reshape_7_output_0_DequantizeLinear contrib.epu.dequantize [-4.59019f, 4.26807f] 28
/model/layers.15/self_attn/o_proj/MatMul_smooth_mul multiply [-1.41938f, 2.23721f] 28
/model/layers.15/self_attn/o_proj/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-11.2972f, 14.5922f] 27
/model/layers.15/Add add [-2248.75f, 2135.1f] 19
/model/layers.15/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-6.73364f, 6.9635f] 28
/model/layers.15/mlp/gate_proj/MatMul_smooth_mul multiply [-1.63977f, 1.50611f] 26
/model/layers.15/mlp/up_proj/MatMul_smooth_mul multiply [-1.08762f, 1.32427f] 26
/model/layers.15/mlp/Mul_output_0_DequantizeLinear contrib.epu.dequantize [-7.56505f, 11.8713f] 27
/model/layers.15/mlp/down_proj/MatMul_smooth_mul multiply [-5.26988f, 4.5116f] 26
/model/layers.15/mlp/down_proj/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-23.6018f, 15.8574f] 26
/model/layers.15/Add_1 add [-2265.53f, 2145.42f] 19
/model/layers.16/input_layernorm/Mul_1 contrib.epu.rms_norm [-10.6589f, 12.5173f] 27
gemm_input_reshape_token_296 reshape [-10.6589f, 12.5173f] 27
/model/layers.16/self_attn/q_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-1.81157f, 1.7112f] 27
/model/layers.16/self_attn/k_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-2.01258f, 2.1495f] 27
/model/layers.16/self_attn/v_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-1.59285f, 1.42885f] 27
/model/layers.16/self_attn/Reshape_7_output_0_DequantizeLinear contrib.epu.dequantize [-4.17096f, 3.84253f] 28
/model/layers.16/self_attn/o_proj/MatMul_smooth_mul multiply [-1.83694f, 1.5825f] 28
/model/layers.16/self_attn/o_proj/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-11.7331f, 15.6025f] 27
/model/layers.16/Add add [-2249.93f, 2135.69f] 19
/model/layers.16/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-7.1051f, 6.93435f] 28
/model/layers.16/mlp/gate_proj/MatMul_smooth_mul multiply [-1.47986f, 2.0182f] 24
/model/layers.16/mlp/up_proj/MatMul_smooth_mul multiply [-1.23218f, 1.4029f] 24
/model/layers.16/mlp/Mul_output_0_DequantizeLinear contrib.epu.dequantize [-7.83379f, 8.67312f] 27
/model/layers.16/mlp/down_proj/MatMul_smooth_mul multiply [-3.54965f, 5.20902f] 26
/model/layers.16/mlp/down_proj/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-17.3144f, 15.4207f] 26
/model/layers.16/Add_1 add [-2262.65f, 2142.99f] 19
/model/layers.17/input_layernorm/Mul_1 contrib.epu.rms_norm [-10.0108f, 8.13536f] 27
gemm_input_reshape_token_314 reshape [-10.0108f, 8.13536f] 27
/model/layers.17/self_attn/q_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-2.15156f, 2.15921f] 27
/model/layers.17/self_attn/k_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-1.80749f, 1.88767f] 27
/model/layers.17/self_attn/v_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-1.05262f, 1.05002f] 28
/model/layers.17/self_attn/Reshape_7_output_0_DequantizeLinear contrib.epu.dequantize [-3.23456f, 3.18402f] 29
/model/layers.17/self_attn/o_proj/MatMul_smooth_mul multiply [-1.52963f, 1.15177f] 28
/model/layers.17/self_attn/o_proj/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-12.3627f, 9.7778f] 27
/model/layers.17/Add add [-2252.87f, 2136.92f] 19
/model/layers.17/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-6.18603f, 7.21669f] 27
/model/layers.17/mlp/gate_proj/MatMul_smooth_mul multiply [-1.33736f, 1.79799f] 25
/model/layers.17/mlp/up_proj/MatMul_smooth_mul multiply [-1.27325f, 1.62598f] 25
/model/layers.17/mlp/Mul_output_0_DequantizeLinear contrib.epu.dequantize [-17.7063f, 8.57209f] 26
/model/layers.17/mlp/down_proj/MatMul_smooth_mul multiply [-8.5319f, 6.1876f] 26
/model/layers.17/mlp/down_proj/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-14.8588f, 12.6532f] 27
/model/layers.17/Add_1 add [-2259.02f, 2140.87f] 19
/model/layers.18/input_layernorm/Mul_1 contrib.epu.rms_norm [-9.84233f, 8.46751f] 27
gemm_input_reshape_token_332 reshape [-9.84233f, 8.46751f] 27
/model/layers.18/self_attn/q_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-1.73369f, 1.98321f] 27
/model/layers.18/self_attn/k_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-1.84075f, 1.84709f] 28
/model/layers.18/self_attn/v_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-1.35609f, 1.29838f] 27
/model/layers.18/self_attn/Reshape_7_output_0_DequantizeLinear contrib.epu.dequantize [-4.04623f, 4.80254f] 28
/model/layers.18/self_attn/o_proj/MatMul_smooth_mul multiply [-1.58647f, 2.07393f] 28
/model/layers.18/self_attn/o_proj/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-15.8002f, 9.58816f] 26
/model/layers.18/Add add [-2253.75f, 2135.33f] 19
/model/layers.18/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-5.90104f, 7.02577f] 27
/model/layers.18/mlp/gate_proj/MatMul_smooth_mul multiply [-1.54129f, 1.62492f] 26
/model/layers.18/mlp/up_proj/MatMul_smooth_mul multiply [-1.36847f, 1.56336f] 26
/model/layers.18/mlp/Mul_output_0_DequantizeLinear contrib.epu.dequantize [-17.3382f, 13.4492f] 26
/model/layers.18/mlp/down_proj/MatMul_smooth_mul multiply [-6.30908f, 9.37604f] 25
/model/layers.18/mlp/down_proj/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-11.4439f, 13.8962f] 27
/model/layers.18/Add_1 add [-2257.72f, 2138.95f] 19
/model/layers.19/input_layernorm/Mul_1 contrib.epu.rms_norm [-11.4571f, 12.3059f] 27
gemm_input_reshape_token_350 reshape [-11.4571f, 12.3059f] 27
/model/layers.19/self_attn/q_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-1.92416f, 2.0737f] 27
/model/layers.19/self_attn/k_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-2.35204f, 3.00163f] 27
/model/layers.19/self_attn/v_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-1.27972f, 1.40155f] 27
/model/layers.19/self_attn/Reshape_7_output_0_DequantizeLinear contrib.epu.dequantize [-3.54796f, 3.75493f] 29
/model/layers.19/self_attn/o_proj/MatMul_smooth_mul multiply [-1.68304f, 1.92258f] 28
/model/layers.19/self_attn/o_proj/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-11.8455f, 12.0497f] 27
/model/layers.19/Add add [-2254.97f, 2135.89f] 19
/model/layers.19/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-7.02649f, 9.09748f] 27
/model/layers.19/mlp/gate_proj/MatMul_smooth_mul multiply [-1.53495f, 1.78837f] 25
/model/layers.19/mlp/up_proj/MatMul_smooth_mul multiply [-1.48996f, 1.6102f] 25
/model/layers.19/mlp/Mul_output_0_DequantizeLinear contrib.epu.dequantize [-22.1731f, 21.2569f] 26
/model/layers.19/mlp/down_proj/MatMul_smooth_mul multiply [-8.6078f, 9.2469f] 25
/model/layers.19/mlp/down_proj/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-25.8321f, 27.113f] 26
/model/layers.19/Add_1 add [-2261.16f, 2141.65f] 19
/model/layers.20/input_layernorm/Mul_1 contrib.epu.rms_norm [-10.3886f, 10.4753f] 27
gemm_input_reshape_token_368 reshape [-10.3886f, 10.4753f] 27
/model/layers.20/self_attn/q_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-1.91683f, 1.96073f] 27
/model/layers.20/self_attn/k_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-2.58277f, 2.66649f] 28
/model/layers.20/self_attn/v_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-1.26085f, 1.39758f] 28
/model/layers.20/self_attn/Reshape_7_output_0_DequantizeLinear contrib.epu.dequantize [-4.10082f, 5.31433f] 28
/model/layers.20/self_attn/o_proj/MatMul_smooth_mul multiply [-2.34641f, 3.19203f] 28
/model/layers.20/self_attn/o_proj/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-35.1506f, 30.7222f] 25
/model/layers.20/Add add [-2250.64f, 2133.63f] 19
/model/layers.20/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-9.19756f, 12.8515f] 27
/model/layers.20/mlp/gate_proj/MatMul_smooth_mul multiply [-1.56006f, 1.75395f] 23
/model/layers.20/mlp/up_proj/MatMul_smooth_mul multiply [-1.98689f, 1.67582f] 23
/model/layers.20/mlp/Mul_output_0_DequantizeLinear contrib.epu.dequantize [-34.8929f, 18.9709f] 25
/model/layers.20/mlp/down_proj/MatMul_smooth_mul multiply [-8.20101f, 9.49806f] 25
/model/layers.20/mlp/down_proj/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-18.8924f, 19.3495f] 26
/model/layers.20/Add_1 add [-2252.77f, 2136.37f] 19
/model/layers.21/input_layernorm/Mul_1 contrib.epu.rms_norm [-9.58123f, 9.53649f] 27
gemm_input_reshape_token_386 reshape [-9.58123f, 9.53649f] 27
/model/layers.21/self_attn/q_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-1.97868f, 1.97198f] 28
/model/layers.21/self_attn/k_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-1.79546f, 2.22244f] 28
/model/layers.21/self_attn/v_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-1.36314f, 1.47782f] 28
/model/layers.21/self_attn/Reshape_7_output_0_DequantizeLinear contrib.epu.dequantize [-3.75884f, 4.26226f] 28
/model/layers.21/self_attn/o_proj/MatMul_smooth_mul multiply [-2.96375f, 2.43026f] 28
/model/layers.21/self_attn/o_proj/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-15.71f, 19.554f] 26
/model/layers.21/Add add [-2248.26f, 2137.87f] 19
/model/layers.21/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-11.0205f, 14.3484f] 27
/model/layers.21/mlp/gate_proj/MatMul_smooth_mul multiply [-1.99114f, 1.95634f] 24
/model/layers.21/mlp/up_proj/MatMul_smooth_mul multiply [-1.60038f, 1.89907f] 24
/model/layers.21/mlp/Mul_output_0_DequantizeLinear contrib.epu.dequantize [-32.3889f, 37.5348f] 25
/model/layers.21/mlp/down_proj/MatMul_smooth_mul multiply [-14.5578f, 13.6219f] 25
/model/layers.21/mlp/down_proj/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-43.6967f, 51.4559f] 25
/model/layers.21/Add_1 add [-2247.45f, 2139.1f] 19
/model/layers.22/input_layernorm/Mul_1 contrib.epu.rms_norm [-8.65606f, 9.24928f] 27
gemm_input_reshape_token_404 reshape [-8.65606f, 9.24928f] 27
/model/layers.22/self_attn/q_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-1.73656f, 1.83084f] 28
/model/layers.22/self_attn/k_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-1.93147f, 2.01617f] 28
/model/layers.22/self_attn/v_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-2.05887f, 2.65952f] 28
/model/layers.22/self_attn/Reshape_7_output_0_DequantizeLinear contrib.epu.dequantize [-6.42015f, 5.4091f] 28
/model/layers.22/self_attn/o_proj/MatMul_smooth_mul multiply [-2.81149f, 2.92772f] 28
/model/layers.22/self_attn/o_proj/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-23.0105f, 18.1567f] 26
/model/layers.22/Add add [-2233.96f, 2122.02f] 19
/model/layers.22/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-11.7072f, 19.2388f] 26
/model/layers.22/mlp/gate_proj/MatMul_smooth_mul multiply [-1.84052f, 1.95147f] 24
/model/layers.22/mlp/up_proj/MatMul_smooth_mul multiply [-1.83295f, 2.40221f] 25
/model/layers.22/mlp/Mul_output_0_DequantizeLinear contrib.epu.dequantize [-45.9365f, 45.5776f] 25
/model/layers.22/mlp/down_proj/MatMul_smooth_mul multiply [-15.2396f, 13.3352f] 25
/model/layers.22/mlp/down_proj/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-34.4755f, 34.1833f] 25
/model/layers.22/Add_1 add [-2228.7f, 2115.59f] 19
/model/layers.23/input_layernorm/Mul_1 contrib.epu.rms_norm [-13.0917f, 13.3282f] 27
gemm_input_reshape_token_422 reshape [-13.0917f, 13.3282f] 27
/model/layers.23/self_attn/q_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-2.06462f, 2.18439f] 27
/model/layers.23/self_attn/k_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-2.20355f, 2.35417f] 27
/model/layers.23/self_attn/v_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-2.2084f, 2.29215f] 27
/model/layers.23/self_attn/Reshape_7_output_0_DequantizeLinear contrib.epu.dequantize [-6.47336f, 6.57693f] 28
/model/layers.23/self_attn/o_proj/MatMul_smooth_mul multiply [-3.58264f, 3.05603f] 28
/model/layers.23/self_attn/o_proj/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-25.9866f, 19.3289f] 26
/model/layers.23/Add add [-2224.84f, 2111.51f] 19
/model/layers.23/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-11.9734f, 26.6538f] 26
/model/layers.23/mlp/gate_proj/MatMul_smooth_mul multiply [-2.31132f, 2.19508f] 23
/model/layers.23/mlp/up_proj/MatMul_smooth_mul multiply [-1.89113f, 2.35949f] 23
/model/layers.23/mlp/Mul_output_0_DequantizeLinear contrib.epu.dequantize [-48.2795f, 47.13f] 25
/model/layers.23/mlp/down_proj/MatMul_smooth_mul multiply [-17.7429f, 16.7087f] 25
/model/layers.23/mlp/down_proj/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-44.8177f, 46.6697f] 25
/model/layers.23/Add_1 add [-2192.61f, 2077.81f] 19
/model/layers.24/input_layernorm/Mul_1 contrib.epu.rms_norm [-10.7199f, 11.0412f] 27
gemm_input_reshape_token_440 reshape [-10.7199f, 11.0412f] 27
/model/layers.24/self_attn/q_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-2.04763f, 2.05723f] 27
/model/layers.24/self_attn/k_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-2.10131f, 1.86701f] 27
/model/layers.24/self_attn/v_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-2.71978f, 2.95485f] 27
/model/layers.24/self_attn/Reshape_7_output_0_DequantizeLinear contrib.epu.dequantize [-7.19929f, 7.33774f] 27
/model/layers.24/self_attn/o_proj/MatMul_smooth_mul multiply [-3.33851f, 3.64716f] 27
/model/layers.24/self_attn/o_proj/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-17.4194f, 17.7147f] 26
/model/layers.24/Add add [-2191.28f, 2076.48f] 19
/model/layers.24/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-11.9233f, 28.5322f] 26
/model/layers.24/mlp/gate_proj/MatMul_smooth_mul multiply [-2.17947f, 2.34172f] 23
/model/layers.24/mlp/up_proj/MatMul_smooth_mul multiply [-1.74746f, 2.21065f] 23
/model/layers.24/mlp/Mul_output_0_DequantizeLinear contrib.epu.dequantize [-45.825f, 32.2207f] 25
/model/layers.24/mlp/down_proj/MatMul_smooth_mul multiply [-13.2913f, 10.1042f] 25
/model/layers.24/mlp/down_proj/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-69.786f, 69.786f] 24
/model/layers.24/Add_1 add [-2121.5f, 2007.31f] 19
/model/layers.25/input_layernorm/Mul_1 contrib.epu.rms_norm [-13.3404f, 10.584f] 27
gemm_input_reshape_token_458 reshape [-13.3404f, 10.584f] 27
/model/layers.25/self_attn/q_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-1.67736f, 1.66992f] 27
/model/layers.25/self_attn/k_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-1.73582f, 1.87265f] 27
/model/layers.25/self_attn/v_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-2.90016f, 3.11647f] 27
/model/layers.25/self_attn/Reshape_7_output_0_DequantizeLinear contrib.epu.dequantize [-8.73085f, 8.87055f] 27
/model/layers.25/self_attn/o_proj/MatMul_smooth_mul multiply [-4.3726f, 4.45093f] 27
/model/layers.25/self_attn/o_proj/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-30.9856f, 32.2555f] 25
/model/layers.25/Add add [-2101.43f, 1987.76f] 19
/model/layers.25/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-12.0494f, 29.539f] 26
/model/layers.25/mlp/gate_proj/MatMul_smooth_mul multiply [-2.03667f, 2.62454f] 23
/model/layers.25/mlp/up_proj/MatMul_smooth_mul multiply [-1.77741f, 2.23702f] 23
/model/layers.25/mlp/Mul_output_0_DequantizeLinear contrib.epu.dequantize [-94.0946f, 52.1931f] 24
/model/layers.25/mlp/down_proj/MatMul_smooth_mul multiply [-12.9914f, 12.5167f] 24
/model/layers.25/mlp/down_proj/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-239.356f, 237.425f] 23
/model/layers.25/Add_1 add [-1864.01f, 1748.4f] 19
/model/layers.26/input_layernorm/Mul_1 contrib.epu.rms_norm [-12.582f, 11.7604f] 27
gemm_input_reshape_token_476 reshape [-12.582f, 11.7604f] 27
/model/layers.26/self_attn/q_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-1.80837f, 1.91073f] 27
/model/layers.26/self_attn/k_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-1.93819f, 2.29129f] 27
/model/layers.26/self_attn/v_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-2.91941f, 2.83608f] 27
/model/layers.26/self_attn/Reshape_7_output_0_DequantizeLinear contrib.epu.dequantize [-10.105f, 10.3555f] 27
/model/layers.26/self_attn/o_proj/MatMul_smooth_mul multiply [-4.24043f, 3.45294f] 27
/model/layers.26/self_attn/o_proj/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-46.2343f, 49.1765f] 25
/model/layers.26/Add add [-1855.18f, 1742.1f] 19
/model/layers.26/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-14.0125f, 36.3901f] 25
/model/layers.26/mlp/gate_proj/MatMul_smooth_mul multiply [-1.86196f, 2.74673f] 24
/model/layers.26/mlp/up_proj/MatMul_smooth_mul multiply [-2.60639f, 2.69339f] 24
/model/layers.26/mlp/Mul_output_0_DequantizeLinear contrib.epu.dequantize [-920.916f, 575.572f] 21
/model/layers.26/mlp/down_proj/MatMul_smooth_mul multiply [-22.3793f, 17.804f] 21
/model/layers.26/mlp/down_proj/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-1050.47f, 869.917f] 20
/model/layers.26/Add_1 add [-985.265f, 905.007f] 19
/model/layers.27/input_layernorm/Mul_1 contrib.epu.rms_norm [-16.2217f, 13.974f] 26
gemm_input_reshape_token_494 reshape [-16.2217f, 13.974f] 26
/model/layers.27/self_attn/q_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-1.90407f, 1.80525f] 27
/model/layers.27/self_attn/k_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-2.13365f, 2.35895f] 27
/model/layers.27/self_attn/v_proj/MatMul/MatMulAddFusion_smooth_mul multiply [-3.66444f, 3.79492f] 27
/model/layers.27/self_attn/Reshape_7_output_0_DequantizeLinear contrib.epu.dequantize [-13.988f, 13.7694f] 27
/model/layers.27/self_attn/o_proj/MatMul_smooth_mul multiply [-5.61135f, 5.30639f] 27
/model/layers.27/self_attn/o_proj/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-136.439f, 141.312f] 23
/model/layers.27/Add add [-913.391f, 831.915f] 19
/model/layers.27/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-28.8839f, 48.9156f] 25
/model/layers.27/mlp/gate_proj/MatMul_smooth_mul multiply [-2.93575f, 3.74036f] 21
/model/layers.27/mlp/up_proj/MatMul_smooth_mul multiply [-2.78124f, 4.95287f] 21
/model/layers.27/mlp/Mul_output_0_DequantizeLinear contrib.epu.dequantize [-154.57f, 175.271f] 23
/model/layers.27/mlp/down_proj/MatMul_smooth_mul multiply [-26.9228f, 16.3997f] 23
/model/layers.27/mlp/down_proj/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-995.761f, 987.982f] 21
/model/layers.27/Add_1 add [-323.559f, 355.294f] 19
/model/norm/Mul_1 contrib.epu.rms_norm [-4.04675f, 2.95359f] 28
logits_DequantizeLinear contrib.epu.dequantize [-14.3316f, 22.8934f] 26
2026-07-18 12:21 - INFO - epu - codegen - START====================build_cpu_runnable_fx_relay
2026-07-18 12:21 - INFO - epu - codegen - START=======================quantize_to_chimera_fx
2026-07-18 12:23 - INFO - epu - codegen - START=================================relay_to_tir
2026-07-18 12:23 - INFO - epu - codegen - START===========================relay_to_epu_relay
2026-07-18 12:23 - INFO - epu - codegen - START==============================adapt_and_order
2026-07-18 12:25 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-07-18 12:25 - INFO - epu - codegen - START=============================plan_lrm_virtual
2026-07-18 12:28 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-07-18 12:28 - INFO - epu - codegen - START===============================lrm_alloc_loop
2026-07-18 12:32 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-07-18 12:32 - INFO - epu - codegen - START================================lrm_splitting
2026-07-18 12:40 - INFO - epu - codegen - START==============================ext_split_relay
2026-07-18 12:44 - INFO - epu - codegen - START====================================build_tir
2026-07-18 12:44 - INFO - epu - chimera_job - Compilation of qwen2_5_1_5b_int8_q_seq512_custom_ops_QC_N_1d7_4MB_4kB_128GBps_128GBps_8_OFF_x1_x1 successful
Compilation complete!
╒═════════════════════╤════════════════════════════════════════════════════════════════════════════════════╕
│ Module Name │ qwen2_5_1_5b_int8_q_seq512_custom_ops_QC_N_1d7_4MB_4kB_128GBps_128GBps_8_OFF_x1_x1 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤
│ ONNX File │ qwen2.5_1.5b_int8_q_seq512_custom_ops.onnx │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤
│ Product Target │ QC-N │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤
│ Number of Cores │ 1 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤
│ ISS Clock Frequency │ 1.700 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤
│ L2M Size │ 4MB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤
│ LRM Size │ 4kB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤
│ External Read BW │ 128GBps │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤
│ External Write BW │ 128GBps │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤
│ MACS per PE │ 8 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤
│ Max L2M │ 3.036MB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤
│ Max LRM │ 2.695kB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤
│ Max Temp Ext Bytes │ 0.006MB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤
│ Network GMACs │ │
╘═════════════════════╧════════════════════════════════════════════════════════════════════════════════════╛
╒═════╤════════╤══════════════════════════╤══════════════════╤══════════════════════════╤═══════╕
│ │ Type │ Name │ shape │ type │ mse │
╞═════╪════════╪══════════════════════════╪══════════════════╪══════════════════════════╪═══════╡
│ 0 │ Input │ input_ids │ [1, 1] │ tensor[int32] │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 1 │ Input │ sin │ [1, 1, 128] │ tensor[FixedPoint32<30>] │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 2 │ Input │ cos │ [1, 1, 128] │ tensor[FixedPoint32<30>] │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 3 │ Input │ attention_mask │ [1, 512] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 4 │ Input │ past_key_values.0.key │ [1, 2, 511, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 5 │ Input │ past_key_values.0.value │ [1, 2, 511, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 6 │ Input │ past_key_values.1.key │ [1, 2, 511, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 7 │ Input │ past_key_values.1.value │ [1, 2, 511, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 8 │ Input │ past_key_values.2.key │ [1, 2, 511, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 9 │ Input │ past_key_values.2.value │ [1, 2, 511, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 10 │ Input │ past_key_values.3.key │ [1, 2, 511, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 11 │ Input │ past_key_values.3.value │ [1, 2, 511, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 12 │ Input │ past_key_values.4.key │ [1, 2, 511, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 13 │ Input │ past_key_values.4.value │ [1, 2, 511, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 14 │ Input │ past_key_values.5.key │ [1, 2, 511, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 15 │ Input │ past_key_values.5.value │ [1, 2, 511, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 16 │ Input │ past_key_values.6.key │ [1, 2, 511, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 17 │ Input │ past_key_values.6.value │ [1, 2, 511, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 18 │ Input │ past_key_values.7.key │ [1, 2, 511, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 19 │ Input │ past_key_values.7.value │ [1, 2, 511, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 20 │ Input │ past_key_values.8.key │ [1, 2, 511, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 21 │ Input │ past_key_values.8.value │ [1, 2, 511, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 22 │ Input │ past_key_values.9.key │ [1, 2, 511, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 23 │ Input │ past_key_values.9.value │ [1, 2, 511, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 24 │ Input │ past_key_values.10.key │ [1, 2, 511, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 25 │ Input │ past_key_values.10.value │ [1, 2, 511, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 26 │ Input │ past_key_values.11.key │ [1, 2, 511, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 27 │ Input │ past_key_values.11.value │ [1, 2, 511, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 28 │ Input │ past_key_values.12.key │ [1, 2, 511, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 29 │ Input │ past_key_values.12.value │ [1, 2, 511, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 30 │ Input │ past_key_values.13.key │ [1, 2, 511, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 31 │ Input │ past_key_values.13.value │ [1, 2, 511, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 32 │ Input │ past_key_values.14.key │ [1, 2, 511, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 33 │ Input │ past_key_values.14.value │ [1, 2, 511, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 34 │ Input │ past_key_values.15.key │ [1, 2, 511, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 35 │ Input │ past_key_values.15.value │ [1, 2, 511, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 36 │ Input │ past_key_values.16.key │ [1, 2, 511, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 37 │ Input │ past_key_values.16.value │ [1, 2, 511, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 38 │ Input │ past_key_values.17.key │ [1, 2, 511, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 39 │ Input │ past_key_values.17.value │ [1, 2, 511, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 40 │ Input │ past_key_values.18.key │ [1, 2, 511, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 41 │ Input │ past_key_values.18.value │ [1, 2, 511, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 42 │ Input │ past_key_values.19.key │ [1, 2, 511, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 43 │ Input │ past_key_values.19.value │ [1, 2, 511, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 44 │ Input │ past_key_values.20.key │ [1, 2, 511, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 45 │ Input │ past_key_values.20.value │ [1, 2, 511, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 46 │ Input │ past_key_values.21.key │ [1, 2, 511, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 47 │ Input │ past_key_values.21.value │ [1, 2, 511, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 48 │ Input │ past_key_values.22.key │ [1, 2, 511, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 49 │ Input │ past_key_values.22.value │ [1, 2, 511, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 50 │ Input │ past_key_values.23.key │ [1, 2, 511, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 51 │ Input │ past_key_values.23.value │ [1, 2, 511, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 52 │ Input │ past_key_values.24.key │ [1, 2, 511, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 53 │ Input │ past_key_values.24.value │ [1, 2, 511, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 54 │ Input │ past_key_values.25.key │ [1, 2, 511, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 55 │ Input │ past_key_values.25.value │ [1, 2, 511, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 56 │ Input │ past_key_values.26.key │ [1, 2, 511, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 57 │ Input │ past_key_values.26.value │ [1, 2, 511, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 58 │ Input │ past_key_values.27.key │ [1, 2, 511, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 59 │ Input │ past_key_values.27.value │ [1, 2, 511, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 60 │ Output │ logits │ [1, 1, 151936] │ tensor[FixedPoint32<26>] │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 61 │ Output │ present.0.key │ [1, 2, 512, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 62 │ Output │ present.0.value │ [1, 2, 512, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 63 │ Output │ present.1.key │ [1, 2, 512, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 64 │ Output │ present.1.value │ [1, 2, 512, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 65 │ Output │ present.2.key │ [1, 2, 512, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 66 │ Output │ present.2.value │ [1, 2, 512, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 67 │ Output │ present.3.key │ [1, 2, 512, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 68 │ Output │ present.3.value │ [1, 2, 512, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 69 │ Output │ present.4.key │ [1, 2, 512, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 70 │ Output │ present.4.value │ [1, 2, 512, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 71 │ Output │ present.5.key │ [1, 2, 512, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 72 │ Output │ present.5.value │ [1, 2, 512, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 73 │ Output │ present.6.key │ [1, 2, 512, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 74 │ Output │ present.6.value │ [1, 2, 512, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 75 │ Output │ present.7.key │ [1, 2, 512, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 76 │ Output │ present.7.value │ [1, 2, 512, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 77 │ Output │ present.8.key │ [1, 2, 512, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 78 │ Output │ present.8.value │ [1, 2, 512, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 79 │ Output │ present.9.key │ [1, 2, 512, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 80 │ Output │ present.9.value │ [1, 2, 512, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 81 │ Output │ present.10.key │ [1, 2, 512, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 82 │ Output │ present.10.value │ [1, 2, 512, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 83 │ Output │ present.11.key │ [1, 2, 512, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 84 │ Output │ present.11.value │ [1, 2, 512, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 85 │ Output │ present.12.key │ [1, 2, 512, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 86 │ Output │ present.12.value │ [1, 2, 512, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 87 │ Output │ present.13.key │ [1, 2, 512, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 88 │ Output │ present.13.value │ [1, 2, 512, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 89 │ Output │ present.14.key │ [1, 2, 512, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 90 │ Output │ present.14.value │ [1, 2, 512, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 91 │ Output │ present.15.key │ [1, 2, 512, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 92 │ Output │ present.15.value │ [1, 2, 512, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 93 │ Output │ present.16.key │ [1, 2, 512, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 94 │ Output │ present.16.value │ [1, 2, 512, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 95 │ Output │ present.17.key │ [1, 2, 512, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 96 │ Output │ present.17.value │ [1, 2, 512, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 97 │ Output │ present.18.key │ [1, 2, 512, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 98 │ Output │ present.18.value │ [1, 2, 512, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 99 │ Output │ present.19.key │ [1, 2, 512, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 100 │ Output │ present.19.value │ [1, 2, 512, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 101 │ Output │ present.20.key │ [1, 2, 512, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 102 │ Output │ present.20.value │ [1, 2, 512, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 103 │ Output │ present.21.key │ [1, 2, 512, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 104 │ Output │ present.21.value │ [1, 2, 512, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 105 │ Output │ present.22.key │ [1, 2, 512, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 106 │ Output │ present.22.value │ [1, 2, 512, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 107 │ Output │ present.23.key │ [1, 2, 512, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 108 │ Output │ present.23.value │ [1, 2, 512, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 109 │ Output │ present.24.key │ [1, 2, 512, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 110 │ Output │ present.24.value │ [1, 2, 512, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 111 │ Output │ present.25.key │ [1, 2, 512, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 112 │ Output │ present.25.value │ [1, 2, 512, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 113 │ Output │ present.26.key │ [1, 2, 512, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 114 │ Output │ present.26.value │ [1, 2, 512, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 115 │ Output │ present.27.key │ [1, 2, 512, 128] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 116 │ Output │ present.27.value │ [1, 2, 512, 128] │ n/a │ n/a │
╘═════╧════════╧══════════════════════════╧══════════════════╧══════════════════════════╧═══════╛
Step 5: Compile C++ to Assembly and Run on ISS
Now we compile the CGC-generated C++ code to assembly using the Quadric LLVM compiler with the Chimera GPNPU backend, then execute it on the Instruction Set Simulator (ISS).
Compilation Pipeline:
- C++ → Assembly: Quadric LLVM with Chimera GPNPU backend
- Execution: ISS with QC-N configuration:
- QC-N processor
- 8 MACs per PE
- 4MB OCM
Setup
The qwen.cpp file wraps the CGC-generated code in an autoregressive runner. The body of the CGC-generated code can be found in the ccl_build folder and has been integrated into qwen3_helpers.hpp. This handles token-by-token generation, KV cache management, and sampling.
Copy Model Weights
First, we need to copy the model weights generated by the ChimeraJob from the ccl_build folder to the current working directory.
import shutil
from pathlib import Path
import glob
## Find const_tensor_data.bin recursively in ccl_build/qwen2.5_1.5b_int8_q_seq512_custom_ops*
const_tensor_files = glob.glob(
"ccl_build/qwen2_5_1_5b*/**/const_tensor_data.bin",
recursive=True,
)
if not const_tensor_files:
print("Error: const_tensor_data.bin not found in ccl_build/qwen2_5_1_5b*")
else:
ccl_build_path = Path(const_tensor_files[0])
shutil.move(ccl_build_path, "const_tensor_data.bin")
print(f"✓ Moved const_tensor_data.bin from {ccl_build_path}")
✓ Moved const_tensor_data.bin from ccl_build/qwen2_5_1_5b_int8_q_seq512_custom_ops_QC_N_1d7_4MB_4kB_128GBps_128GBps_8_OFF_x1_x1/build/const_tensor_data.bin
Create Prompt
Prepare the input prompt and tokenize it for inference using the DeepSeek-R1-Distill-Qwen-1.5B tokenizer.
!python3 prepare_inputs.py --prompt "On the other" --new_tokens 3
2026-07-18 12:44:49.869033: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2026-07-18 12:44:49.869278: I external/local_tsl/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2026-07-18 12:44:49.871229: I external/local_tsl/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2026-07-18 12:44:49.892242: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-07-18 12:44:50.400608: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
[1925 279 1008]
Prompt length: 3
Note: Update the following constants in qwen.cpp based on your prompt:
constexpr std::int32_t maxPromptLength = 512; // Maximum prompt length
constexpr std::int32_t newTokens = 1; // Number of tokens to generate
The prepare_inputs.py script will output the actual prompt length. For ISS runs, use short prompts and few tokens since each token takes 15-20 minutes to process.
Compile and Execute
The sdk source command compiles the C++ code using the Quadric LLVM toolchain and runs it on the ISS.
Key Parameters:
--target QC-N: Target Quadric Chimera Neural processor--ocm-size 4MB: 4MB on-chip memory--macs-per-pe 8: 8 multiply-accumulate units per PE
Compile Time: ~5-10 minutes
Runtime: Variable depending on prompt length and token count (15-20 min per token)
%%bash
set -euo pipefail
sdk source -v qwen.cpp \
--include-cgc-headers \
--target QC-N \
--ocm-size 4MB \
--macs-per-pe 8 \
--quiet
2026-07-18 12:44 - DEBUG - sdk - cli - Executing command: cmake CMakeLists.txt -B /quadric/sdk-cli/examples/models/qwen/qwen2.5-1.5b/qwen_QC-N_1d7_4MB_4kB_128GBps_128GBps_8_OFF_x1_x1/build -DNUM_GPNPUS=1 -DNUM_CORES=8 -DNUM_BORDERS=2 -DEPU_VERSION=2.0.0 -DQLLVM_ROOT_PATH=/quadric/llvm -DOCM_SIZE_KIBIBYTES=4096 -DNUM_PE_MACS=8 -DASSERT_MLS_WIDTH_LINE_ALIGN=ON -DHARDWARE_TARGET=OFF
2026-07-18 12:44 - DEBUG - sdk - cli - Executing command: make -j8
[SDK-CLI] : Executing on QC-N simulator
2026-07-18 12:47 - DEBUG - sdk - cli - Executing command: ./qwen_host -c --ddrRdBwTotal 1048576.0 --ddrWrBwTotal 1048576.0 --ddrAxiWidth 128 --instMemDepth 1310720 --ocmSize 4194304 --cycleTimeNS 0.5882352941176471 --no-check --ddrRdAvgPct 100 --ddrRdMaxPct 100 --ddrWrAvgPct 100 --ddrWrMaxPct 100 --postKernelFlowTimeoutCycles 4000000 --clusterSize 1 --numClusters 1
[SDK-CLI] : TotalCycles: 679,820,114
[SDK-CLI] : Executions/second: 3
compute : ▇ 10.704M
data_array : ▏ 2.253M
mac : ▏ 7.863M
data_ocm : ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 240.478M
data_external: ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇���▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 418.522M
[SDK-CLI] : Execution completed.
Decode Output Tokens
Once the ISS run is complete, decode the generated tokens back to human-readable text using the DeepSeek-R1-Distill tokenizer.
For our example prompt "On the other", the DeepSeek R1 distilled model should predict tokens like " hand, if" demonstrating its reasoning capabilities.
!python3 decode_outputs.py
Token ids: [96662]
Generated text: /Open
from tvm.contrib.epu.chimera_job import core
import glob
## Find and plot the profile from the ISS run
profile_file = glob.glob("qwen_QC-N*/**/profile.json", recursive=True)
if profile_file:
core._plot_profile_results(profile_file[0], clock_freq=1 * 1e9)
else:
print("Profile file not found. Make sure the ISS run completed successfully.")
[SDK-CLI] : TotalCycles: 679,820,114
[SDK-CLI] : Executions/second: 1
compute : ▇ 10.704M
data_array : ▏ 2.253M
mac : ▏ 7.863M
data_ocm : ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 240.478M
data_external: ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 418.522M
How to Read the Profiler Output
The profiler shows the total cycle count for the entire inference run. To calculate per-token performance:
Example: For a 3-token prompt:
- Total cycles: ~680M cycles
- At 1 GHz clock frequency: ~680ms total runtime
- Per-token latency: 680ms ÷ 3 tokens = 226 ms/token
- Throughput: ~4.4 tokens/sec on QC-N (1 TOPs)
Understanding autoregressive execution: The model runs autoregressively - each token must be generated sequentially, with each generation requiring a full forward pass through the network. The cycles shown represent all these sequential passes combined.
Key insight: The cycle count scales with the number of tokens generated. For longer sequences, divide the total cycles by your token count to get per-token performance.
View Performance Profile
Display the performance profile from the ISS execution. This shows cycle counts for different operations and calculates the tokens per second throughput.
'%3e%3crect%20width='512'%20height='512'%20rx='60'%3e%3c/rect%3e%3cpath%20fill='%23fff'%20d='M355.904%20100H408.832L293.2%20232.16L429.232%20412H322.72L239.296%20302.928L143.84%20412H90.8805L214.56%20270.64L84.0645%20100H193.28L268.688%20199.696L355.904%20100ZM337.328%20380.32H366.656L177.344%20130.016H145.872L337.328%20380.32Z'%3e%3c/path%3e%3c/g%3e%3cdefs%3e%3cclipPath%20id='clip0_84_15698'%3e%3crect%20width='512'%20height='512'%20fill='%23fff'%3e%3c/rect%3e%3c/clipPath%3e%3c/defs%3e%3c/svg%3e)
