NOTE: The Jupyter Notebook below is included in the Chimera SDK and can be run interactively by running the following CLI command:
$ quadric sdk notebook
From the Jupyter Notebook window in your browser, select the notebook named /quadric/sdk-cli/examples/models/detr/encoder/detr_encoder.ipynb.
DETR Transformer Encoder on Chimera GPNPU
This notebook compiles and validates the DETR transformer encoder on Quadric hardware. The encoder (6 identical layers of self-attention + FFN) compiles fully natively using CGC — no custom ops required.
Why does this compile natively?
CGC recognises the multi-head attention pattern and maps it to the SDK’s built-in nn::multiheadAttentionHead kernel, which tiles the 360-token sequence across an 18×20 PE grid. LayerNorm and the FFN (256→2048→256) are handled by standard nn::layer_norm and matrixMulMatrix kernels.
Model: DETR Transformer Encoder (ResNet-50 backbone variant, 6 layers, d_model=256, 8 heads, FFN dim=2048, seq_len=360)
1. Setup
import onnx
from tvm.contrib.epu.chimera_job.chimera_job import ChimeraJob
from tvm.contrib.epu.chimera_job.hw_config import HWConfig
from tvm.contrib.epu.chimera_job.quantize import quadric_quantize
from tvm.contrib.epu.onnx_util import cut_onnx
ENCODER_ONNX = "onnx/detr_3_transformer_encoder.onnx"
OUTPUT_DIR = "onnx"
2. Quantize
Quantize the float32 encoder to symmetric INT8 using the QOperator format. Softmax and LayerNorm are excluded from quantization (kept in float) since they require higher precision for numerical stability.
Note: This step uses synthetic (random) calibration data. The purpose of this notebook is to demonstrate that CGC can compile and run the DETR encoder natively and numerically match ORT — not to produce a production-quality quantized model. For deployment, replace
synthetic_input=Truewith a representative calibration dataset.
result = quadric_quantize(
ENCODER_ONNX,
num_images=1,
synthetic_input=True,
output_folder=OUTPUT_DIR,
)
quantized_model = result.qmodel_path
tranges_file = result.tranges_path
print(f"Quantized model: {quantized_model}")
print(f"Tensor ranges: {tranges_file}")
2026-06-19 03:53 - INFO - epu - quantize - Generating synthetic data
2026-06-19 03:53 - INFO - epu - quantize - Optimized model to opset
2026-06-19 03:53 - INFO - epu - quantize - Saved optimized model to detr_3_transformer_encoder_float32_opt.onnx
2026-06-19 03:53 - INFO - epu - quantize - Input shapes: [1, 360, 256]. Input names: src
2026-06-19 03:53 - INFO - epu - quantize - Input shapes: [1, 360, 256]. Input names: pos_embed
2026-06-19 03:53 - INFO - epu - quantize - Output shapes: [[1, 360, 256]]. Output names: ['encoder_memory']
2026-06-19 03:53 - DEBUG - epu - quantize - Full exclusion set for quantization: ['Softmax', 'Sigmoid', 'QuadricCustomOp']
2026-06-19 03:53 - DEBUG - epu - quantize - excl_nodes ['/encoder/layers.0/self_attn/Softmax', '/encoder/layers.1/self_attn/Softmax', '/encoder/layers.2/self_attn/Softmax', '/encoder/layers.3/self_attn/Softmax', '/encoder/layers.4/self_attn/Softmax', '/encoder/layers.5/self_attn/Softmax', '/encoder/layers.0/self_attn_layer_norm/ReduceMean', '/encoder/layers.0/self_attn_layer_norm/Sub', '/encoder/layers.0/self_attn_layer_norm/Constant_output_0', '/encoder/layers.0/self_attn_layer_norm/Pow', '/encoder/layers.0/self_attn_layer_norm/ReduceMean_1', '/encoder/layers.0/self_attn_layer_norm/Constant_1_output_0', '/encoder/layers.0/self_attn_layer_norm/Add', '/encoder/layers.0/self_attn_layer_norm/Sqrt', '/encoder/layers.0/self_attn_layer_norm/Div', 'encoder.layers.0.self_attn_layer_norm.weight', '/encoder/layers.0/self_attn_layer_norm/Mul', 'encoder.layers.0.self_attn_layer_norm.bias', '/encoder/layers.0/self_attn_layer_norm/Add_1', '/encoder/layers.0/final_layer_norm/ReduceMean', '/encoder/layers.0/final_layer_norm/Sub', '/encoder/layers.0/self_attn_layer_norm/Constant_output_0', '/encoder/layers.0/final_layer_norm/Pow', '/encoder/layers.0/final_layer_norm/ReduceMean_1', '/encoder/layers.0/self_attn_layer_norm/Constant_1_output_0', '/encoder/layers.0/final_layer_norm/Add', '/encoder/layers.0/final_layer_norm/Sqrt', '/encoder/layers.0/final_layer_norm/Div', 'encoder.layers.0.final_layer_norm.weight', '/encoder/layers.0/final_layer_norm/Mul', 'encoder.layers.0.final_layer_norm.bias', '/encoder/layers.0/final_layer_norm/Add_1', '/encoder/layers.1/self_attn_layer_norm/ReduceMean', '/encoder/layers.1/self_attn_layer_norm/Sub', '/encoder/layers.0/self_attn_layer_norm/Constant_output_0', '/encoder/layers.1/self_attn_layer_norm/Pow', '/encoder/layers.1/self_attn_layer_norm/ReduceMean_1', '/encoder/layers.0/self_attn_layer_norm/Constant_1_output_0', '/encoder/layers.1/self_attn_layer_norm/Add', '/encoder/layers.1/self_attn_layer_norm/Sqrt', '/encoder/layers.1/self_attn_layer_norm/Div', 'encoder.layers.1.self_attn_layer_norm.weight', '/encoder/layers.1/self_attn_layer_norm/Mul', 'encoder.layers.1.self_attn_layer_norm.bias', '/encoder/layers.1/self_attn_layer_norm/Add_1', '/encoder/layers.1/final_layer_norm/ReduceMean', '/encoder/layers.1/final_layer_norm/Sub', '/encoder/layers.0/self_attn_layer_norm/Constant_output_0', '/encoder/layers.1/final_layer_norm/Pow', '/encoder/layers.1/final_layer_norm/ReduceMean_1', '/encoder/layers.0/self_attn_layer_norm/Constant_1_output_0', '/encoder/layers.1/final_layer_norm/Add', '/encoder/layers.1/final_layer_norm/Sqrt', '/encoder/layers.1/final_layer_norm/Div', 'encoder.layers.1.final_layer_norm.weight', '/encoder/layers.1/final_layer_norm/Mul', 'encoder.layers.1.final_layer_norm.bias', '/encoder/layers.1/final_layer_norm/Add_1', '/encoder/layers.2/self_attn_layer_norm/ReduceMean', '/encoder/layers.2/self_attn_layer_norm/Sub', '/encoder/layers.0/self_attn_layer_norm/Constant_output_0', '/encoder/layers.2/self_attn_layer_norm/Pow', '/encoder/layers.2/self_attn_layer_norm/ReduceMean_1', '/encoder/layers.0/self_attn_layer_norm/Constant_1_output_0', '/encoder/layers.2/self_attn_layer_norm/Add', '/encoder/layers.2/self_attn_layer_norm/Sqrt', '/encoder/layers.2/self_attn_layer_norm/Div', 'encoder.layers.2.self_attn_layer_norm.weight', '/encoder/layers.2/self_attn_layer_norm/Mul', 'encoder.layers.2.self_attn_layer_norm.bias', '/encoder/layers.2/self_attn_layer_norm/Add_1', '/encoder/layers.2/final_layer_norm/ReduceMean', '/encoder/layers.2/final_layer_norm/Sub', '/encoder/layers.0/self_attn_layer_norm/Constant_output_0', '/encoder/layers.2/final_layer_norm/Pow', '/encoder/layers.2/final_layer_norm/ReduceMean_1', '/encoder/layers.0/self_attn_layer_norm/Constant_1_output_0', '/encoder/layers.2/final_layer_norm/Add', '/encoder/layers.2/final_layer_norm/Sqrt', '/encoder/layers.2/final_layer_norm/Div', 'encoder.layers.2.final_layer_norm.weight', '/encoder/layers.2/final_layer_norm/Mul', 'encoder.layers.2.final_layer_norm.bias', '/encoder/layers.2/final_layer_norm/Add_1', '/encoder/layers.3/self_attn_layer_norm/ReduceMean', '/encoder/layers.3/self_attn_layer_norm/Sub', '/encoder/layers.0/self_attn_layer_norm/Constant_output_0', '/encoder/layers.3/self_attn_layer_norm/Pow', '/encoder/layers.3/self_attn_layer_norm/ReduceMean_1', '/encoder/layers.0/self_attn_layer_norm/Constant_1_output_0', '/encoder/layers.3/self_attn_layer_norm/Add', '/encoder/layers.3/self_attn_layer_norm/Sqrt', '/encoder/layers.3/self_attn_layer_norm/Div', 'encoder.layers.3.self_attn_layer_norm.weight', '/encoder/layers.3/self_attn_layer_norm/Mul', 'encoder.layers.3.self_attn_layer_norm.bias', '/encoder/layers.3/self_attn_layer_norm/Add_1', '/encoder/layers.3/final_layer_norm/ReduceMean', '/encoder/layers.3/final_layer_norm/Sub', '/encoder/layers.0/self_attn_layer_norm/Constant_output_0', '/encoder/layers.3/final_layer_norm/Pow', '/encoder/layers.3/final_layer_norm/ReduceMean_1', '/encoder/layers.0/self_attn_layer_norm/Constant_1_output_0', '/encoder/layers.3/final_layer_norm/Add', '/encoder/layers.3/final_layer_norm/Sqrt', '/encoder/layers.3/final_layer_norm/Div', 'encoder.layers.3.final_layer_norm.weight', '/encoder/layers.3/final_layer_norm/Mul', 'encoder.layers.3.final_layer_norm.bias', '/encoder/layers.3/final_layer_norm/Add_1', '/encoder/layers.4/self_attn_layer_norm/ReduceMean', '/encoder/layers.4/self_attn_layer_norm/Sub', '/encoder/layers.0/self_attn_layer_norm/Constant_output_0', '/encoder/layers.4/self_attn_layer_norm/Pow', '/encoder/layers.4/self_attn_layer_norm/ReduceMean_1', '/encoder/layers.0/self_attn_layer_norm/Constant_1_output_0', '/encoder/layers.4/self_attn_layer_norm/Add', '/encoder/layers.4/self_attn_layer_norm/Sqrt', '/encoder/layers.4/self_attn_layer_norm/Div', 'encoder.layers.4.self_attn_layer_norm.weight', '/encoder/layers.4/self_attn_layer_norm/Mul', 'encoder.layers.4.self_attn_layer_norm.bias', '/encoder/layers.4/self_attn_layer_norm/Add_1', '/encoder/layers.4/final_layer_norm/ReduceMean', '/encoder/layers.4/final_layer_norm/Sub', '/encoder/layers.0/self_attn_layer_norm/Constant_output_0', '/encoder/layers.4/final_layer_norm/Pow', '/encoder/layers.4/final_layer_norm/ReduceMean_1', '/encoder/layers.0/self_attn_layer_norm/Constant_1_output_0', '/encoder/layers.4/final_layer_norm/Add', '/encoder/layers.4/final_layer_norm/Sqrt', '/encoder/layers.4/final_layer_norm/Div', 'encoder.layers.4.final_layer_norm.weight', '/encoder/layers.4/final_layer_norm/Mul', 'encoder.layers.4.final_layer_norm.bias', '/encoder/layers.4/final_layer_norm/Add_1', '/encoder/layers.5/self_attn_layer_norm/ReduceMean', '/encoder/layers.5/self_attn_layer_norm/Sub', '/encoder/layers.0/self_attn_layer_norm/Constant_output_0', '/encoder/layers.5/self_attn_layer_norm/Pow', '/encoder/layers.5/self_attn_layer_norm/ReduceMean_1', '/encoder/layers.0/self_attn_layer_norm/Constant_1_output_0', '/encoder/layers.5/self_attn_layer_norm/Add', '/encoder/layers.5/self_attn_layer_norm/Sqrt', '/encoder/layers.5/self_attn_layer_norm/Div', 'encoder.layers.5.self_attn_layer_norm.weight', '/encoder/layers.5/self_attn_layer_norm/Mul', 'encoder.layers.5.self_attn_layer_norm.bias', '/encoder/layers.5/self_attn_layer_norm/Add_1', '/encoder/layers.5/final_layer_norm/ReduceMean', '/encoder/layers.5/final_layer_norm/Sub', '/encoder/layers.0/self_attn_layer_norm/Constant_output_0', '/encoder/layers.5/final_layer_norm/Pow', '/encoder/layers.5/final_layer_norm/ReduceMean_1', '/encoder/layers.0/self_attn_layer_norm/Constant_1_output_0', '/encoder/layers.5/final_layer_norm/Add', '/encoder/layers.5/final_layer_norm/Sqrt', '/encoder/layers.5/final_layer_norm/Div', 'encoder.layers.5.final_layer_norm.weight', '/encoder/layers.5/final_layer_norm/Mul', 'encoder.layers.5.final_layer_norm.bias', '/encoder/layers.5/final_layer_norm/Add_1', '/encoder/layers.0/self_attn_layer_norm/ReduceMean', '/encoder/layers.0/self_attn_layer_norm/Sub', '/encoder/layers.0/self_attn_layer_norm/Constant_output_0', '/encoder/layers.0/self_attn_layer_norm/Pow', '/encoder/layers.0/self_attn_layer_norm/ReduceMean_1', '/encoder/layers.0/self_attn_layer_norm/Constant_1_output_0', '/encoder/layers.0/self_attn_layer_norm/Add', '/encoder/layers.0/self_attn_layer_norm/Sqrt', '/encoder/layers.0/self_attn_layer_norm/Div', '/encoder/layers.0/final_layer_norm/ReduceMean', '/encoder/layers.0/final_layer_norm/Sub', '/encoder/layers.0/self_attn_layer_norm/Constant_output_0', '/encoder/layers.0/final_layer_norm/Pow', '/encoder/layers.0/final_layer_norm/ReduceMean_1', '/encoder/layers.0/self_attn_layer_norm/Constant_1_output_0', '/encoder/layers.0/final_layer_norm/Add', '/encoder/layers.0/final_layer_norm/Sqrt', '/encoder/layers.0/final_layer_norm/Div', '/encoder/layers.1/self_attn_layer_norm/ReduceMean', '/encoder/layers.1/self_attn_layer_norm/Sub', '/encoder/layers.0/self_attn_layer_norm/Constant_output_0', '/encoder/layers.1/self_attn_layer_norm/Pow', '/encoder/layers.1/self_attn_layer_norm/ReduceMean_1', '/encoder/layers.0/self_attn_layer_norm/Constant_1_output_0', '/encoder/layers.1/self_attn_layer_norm/Add', '/encoder/layers.1/self_attn_layer_norm/Sqrt', '/encoder/layers.1/self_attn_layer_norm/Div', '/encoder/layers.1/final_layer_norm/ReduceMean', '/encoder/layers.1/final_layer_norm/Sub', '/encoder/layers.0/self_attn_layer_norm/Constant_output_0', '/encoder/layers.1/final_layer_norm/Pow', '/encoder/layers.1/final_layer_norm/ReduceMean_1', '/encoder/layers.0/self_attn_layer_norm/Constant_1_output_0', '/encoder/layers.1/final_layer_norm/Add', '/encoder/layers.1/final_layer_norm/Sqrt', '/encoder/layers.1/final_layer_norm/Div', '/encoder/layers.2/self_attn_layer_norm/ReduceMean', '/encoder/layers.2/self_attn_layer_norm/Sub', '/encoder/layers.0/self_attn_layer_norm/Constant_output_0', '/encoder/layers.2/self_attn_layer_norm/Pow', '/encoder/layers.2/self_attn_layer_norm/ReduceMean_1', '/encoder/layers.0/self_attn_layer_norm/Constant_1_output_0', '/encoder/layers.2/self_attn_layer_norm/Add', '/encoder/layers.2/self_attn_layer_norm/Sqrt', '/encoder/layers.2/self_attn_layer_norm/Div', '/encoder/layers.2/final_layer_norm/ReduceMean', '/encoder/layers.2/final_layer_norm/Sub', '/encoder/layers.0/self_attn_layer_norm/Constant_output_0', '/encoder/layers.2/final_layer_norm/Pow', '/encoder/layers.2/final_layer_norm/ReduceMean_1', '/encoder/layers.0/self_attn_layer_norm/Constant_1_output_0', '/encoder/layers.2/final_layer_norm/Add', '/encoder/layers.2/final_layer_norm/Sqrt', '/encoder/layers.2/final_layer_norm/Div', '/encoder/layers.3/self_attn_layer_norm/ReduceMean', '/encoder/layers.3/self_attn_layer_norm/Sub', '/encoder/layers.0/self_attn_layer_norm/Constant_output_0', '/encoder/layers.3/self_attn_layer_norm/Pow', '/encoder/layers.3/self_attn_layer_norm/ReduceMean_1', '/encoder/layers.0/self_attn_layer_norm/Constant_1_output_0', '/encoder/layers.3/self_attn_layer_norm/Add', '/encoder/layers.3/self_attn_layer_norm/Sqrt', '/encoder/layers.3/self_attn_layer_norm/Div', '/encoder/layers.3/final_layer_norm/ReduceMean', '/encoder/layers.3/final_layer_norm/Sub', '/encoder/layers.0/self_attn_layer_norm/Constant_output_0', '/encoder/layers.3/final_layer_norm/Pow', '/encoder/layers.3/final_layer_norm/ReduceMean_1', '/encoder/layers.0/self_attn_layer_norm/Constant_1_output_0', '/encoder/layers.3/final_layer_norm/Add', '/encoder/layers.3/final_layer_norm/Sqrt', '/encoder/layers.3/final_layer_norm/Div', '/encoder/layers.4/self_attn_layer_norm/ReduceMean', '/encoder/layers.4/self_attn_layer_norm/Sub', '/encoder/layers.0/self_attn_layer_norm/Constant_output_0', '/encoder/layers.4/self_attn_layer_norm/Pow', '/encoder/layers.4/self_attn_layer_norm/ReduceMean_1', '/encoder/layers.0/self_attn_layer_norm/Constant_1_output_0', '/encoder/layers.4/self_attn_layer_norm/Add', '/encoder/layers.4/self_attn_layer_norm/Sqrt', '/encoder/layers.4/self_attn_layer_norm/Div', '/encoder/layers.4/final_layer_norm/ReduceMean', '/encoder/layers.4/final_layer_norm/Sub', '/encoder/layers.0/self_attn_layer_norm/Constant_output_0', '/encoder/layers.4/final_layer_norm/Pow', '/encoder/layers.4/final_layer_norm/ReduceMean_1', '/encoder/layers.0/self_attn_layer_norm/Constant_1_output_0', '/encoder/layers.4/final_layer_norm/Add', '/encoder/layers.4/final_layer_norm/Sqrt', '/encoder/layers.4/final_layer_norm/Div', '/encoder/layers.5/self_attn_layer_norm/ReduceMean', '/encoder/layers.5/self_attn_layer_norm/Sub', '/encoder/layers.0/self_attn_layer_norm/Constant_output_0', '/encoder/layers.5/self_attn_layer_norm/Pow', '/encoder/layers.5/self_attn_layer_norm/ReduceMean_1', '/encoder/layers.0/self_attn_layer_norm/Constant_1_output_0', '/encoder/layers.5/self_attn_layer_norm/Add', '/encoder/layers.5/self_attn_layer_norm/Sqrt', '/encoder/layers.5/self_attn_layer_norm/Div', '/encoder/layers.5/final_layer_norm/ReduceMean', '/encoder/layers.5/final_layer_norm/Sub', '/encoder/layers.0/self_attn_layer_norm/Constant_output_0', '/encoder/layers.5/final_layer_norm/Pow', '/encoder/layers.5/final_layer_norm/ReduceMean_1', '/encoder/layers.0/self_attn_layer_norm/Constant_1_output_0', '/encoder/layers.5/final_layer_norm/Add', '/encoder/layers.5/final_layer_norm/Sqrt', '/encoder/layers.5/final_layer_norm/Div']
2026-06-19 03:53 - INFO - epu - quantize - Quantization started...
WARNING:root:Please use QuantFormat.QDQ for activation type QInt8 and weight type QInt8. Or it will lead to bad performance on x64.
2026-06-19 03:53 - INFO - epu - quantize - Quantization done succesfully!
2026-06-19 03:53 - INFO - epu - quantize - ONNX full precision model size: 30.18 MB
2026-06-19 03:53 - INFO - epu - quantize - ONNX quantized model size: 7.71 MB
2026-06-19 03:53 - INFO - epu - quantize - Saved quantized model to onnx/detr_3_transformer_encoder_opt_sym_int8_q.onnx
2026-06-19 03:53 - INFO - epu - quantize - Saved shape inferenced model to onnx/detr_3_transformer_encoder_opt_sym_int8_q.onnx
2026-06-19 03:53 - INFO - epu - quantize - Checking for remaining FLOAT/FLOAT16 types.
2026-06-19 03:53 - INFO - epu - quantize - Model still has FLOAT/FLOAT16 types. Creating ranges for floating point tensors using calibration data
2026-06-19 03:53 - INFO - epu - quantize - Saved tensor ranges to onnx/detr_3_transformer_encoder_opt_sym_int8_q.onnx.tranges
Quantized model: onnx/detr_3_transformer_encoder_opt_sym_int8_q.onnx
Tensor ranges: onnx/detr_3_transformer_encoder_opt_sym_int8_q.onnx.tranges
3. Compile with CGC (Chimera Graph Compiler)
Compile the quantized encoder to ASM targeting QC-U with ISS validation enabled. CGC handles all 6 encoder layers natively:
| Block | CGC kernel | Notes |
|---|---|---|
| Self-attention | nn::multiheadAttentionHead | 8 heads, 360 tokens mapped to 18×20 PE grid |
| LayerNorm | nn::layer_norm | Float precision |
| FFN | matrixMulMatrix + conv2d | 256→2048→256 |
hw_config = HWConfig(
product="QC-U",
ocm_size="8MB",
lrm_size="4kB",
macs_per_pe=16,
clock_freq_ghz=1.56,
ext_rd_bw="64GBps",
ext_wr_bw="64GBps",
num_cores=1,
)
cgc_job = ChimeraJob(
quantized_model,
hw_config=hw_config,
trange_file=tranges_file,
target_lang="ASM",
validate_iss=True,
)
print(f"Compiling {quantized_model} ...")
cgc_job.compile()
print("Compilation successful!")
print(cgc_job)
Compiling onnx/detr_3_transformer_encoder_opt_sym_int8_q.onnx ...
2026-06-19 03:53 - INFO - epu - chimera_job - START==================================onnx_ingest
2026-06-19 03:53 - INFO - epu - chimera_job - Numerical ranges provided
2026-06-19 03:53 - INFO - epu - codegen - START===============================optimize_relay
2026-06-19 03:53 - INFO - epu - codegen - START====================quantize_to_cpu_runnable_fx
2026-06-19 03:53 - INFO - epu - fx -
Source name Op Output 0 Range Output 0 Frac Bits
------------------------------------------------------------ ---------------------- -------------------------- --------------------
/encoder/layers.0/self_attn/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-0.961714f, 0.96947f] 31
/encoder/layers.0/self_attn/Softmax nn.softmax [0.00104144f, 0.00702261f] 31
/encoder/layers.0/Add_output_0_DequantizeLinear contrib.epu.dequantize [-1.35529f, 1.42662f] 30
/encoder/layers.0/self_attn_layer_norm/Add_1 nn.layer_norm [-7.88192f, 7.72645f] 28
/encoder/layers.0/Add_1_output_0_DequantizeLinear contrib.epu.dequantize [-15.6543f, 15.7905f] 26
/encoder/layers.0/final_layer_norm/Add_1 nn.layer_norm [-7.15572f, 5.43056f] 28
/encoder/layers.1/self_attn/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-12.3522f, 14.0065f] 27
/encoder/layers.1/self_attn/Softmax nn.softmax [1.72072e-10f, 0.99018f] 27
/encoder/layers.1/Add_output_0_DequantizeLinear contrib.epu.dequantize [-7.67558f, 5.57679f] 28
/encoder/layers.1/self_attn_layer_norm/Add_1 nn.layer_norm [-11.9921f, 10.5237f] 27
/encoder/layers.1/Add_1_output_0_DequantizeLinear contrib.epu.dequantize [-29.7313f, 17.8892f] 25
/encoder/layers.1/final_layer_norm/Add_1 nn.layer_norm [-7.15995f, 4.61846f] 28
/encoder/layers.2/self_attn/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-8.92486f, 10.3562f] 27
/encoder/layers.2/self_attn/Softmax nn.softmax [1.54682e-07f, 0.752262f] 27
/encoder/layers.2/Add_output_0_DequantizeLinear contrib.epu.dequantize [-7.13891f, 4.79646f] 28
/encoder/layers.2/self_attn_layer_norm/Add_1 nn.layer_norm [-13.0621f, 9.97753f] 27
/encoder/layers.2/Add_1_output_0_DequantizeLinear contrib.epu.dequantize [-52.7643f, 25.9025f] 25
/encoder/layers.2/final_layer_norm/Add_1 nn.layer_norm [-6.45745f, 4.95792f] 28
/encoder/layers.3/self_attn/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-7.16091f, 7.10497f] 28
/encoder/layers.3/self_attn/Softmax nn.softmax [5.97861e-08f, 0.63221f] 28
/encoder/layers.3/Add_output_0_DequantizeLinear contrib.epu.dequantize [-5.1206f, 4.76056f] 28
/encoder/layers.3/self_attn_layer_norm/Add_1 nn.layer_norm [-13.2036f, 10.7333f] 27
/encoder/layers.3/Add_1_output_0_DequantizeLinear contrib.epu.dequantize [-73.8637f, 73.2867f] 24
/encoder/layers.3/final_layer_norm/Add_1 nn.layer_norm [-5.744f, 6.03491f] 28
/encoder/layers.4/self_attn/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-8.92979f, 11.3408f] 27
/encoder/layers.4/self_attn/Softmax nn.softmax [6.19007e-09f, 0.68639f] 27
/encoder/layers.4/Add_output_0_DequantizeLinear contrib.epu.dequantize [-5.63272f, 5.58871f] 28
/encoder/layers.4/self_attn_layer_norm/Add_1 nn.layer_norm [-16.908f, 16.6125f] 26
/encoder/layers.4/Add_1_output_0_DequantizeLinear contrib.epu.dequantize [-27.488f, 27.2732f] 26
/encoder/layers.4/final_layer_norm/Add_1 nn.layer_norm [-5.9278f, 5.35991f] 28
/encoder/layers.5/self_attn/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-7.76563f, 8.21863f] 27
/encoder/layers.5/self_attn/Softmax nn.softmax [4.69318e-07f, 0.382338f] 27
/encoder/layers.5/Add_output_0_DequantizeLinear contrib.epu.dequantize [-5.51653f, 5.12865f] 28
/encoder/layers.5/self_attn_layer_norm/Add_1 nn.layer_norm [-13.8806f, 10.8122f] 27
/encoder/layers.5/Add_1_output_0_DequantizeLinear contrib.epu.dequantize [-15.8453f, 12.6471f] 26
/encoder/layers.5/final_layer_norm/Add_1 nn.layer_norm [-4.23313f, 4.54449f] -
2026-06-19 03:53 - INFO - epu - codegen - START====================build_cpu_runnable_fx_relay
2026-06-19 03:53 - INFO - epu - codegen - START=======================quantize_to_chimera_fx
2026-06-19 03:53 - INFO - epu - codegen - START=================================relay_to_tir
2026-06-19 03:53 - INFO - epu - codegen - START===========================relay_to_epu_relay
2026-06-19 03:53 - INFO - epu - codegen - START==============================adapt_and_order
2026-06-19 03:53 - INFO - epu - mac_counter -
2026-06-19 03:53 - INFO - epu - mac_counter - ============================================================
2026-06-19 03:53 - INFO - epu - mac_counter - MAC Operation Count Summary
2026-06-19 03:53 - INFO - epu - mac_counter - ============================================================
2026-06-19 03:53 - INFO - epu - mac_counter - conv2d: 47,185,920 ops (23,592,960 MACs) - /encoder/layers.0/self_attn/out_proj/MatMul_quant
2026-06-19 03:53 - INFO - epu - mac_counter - conv2d: 47,185,920 ops (23,592,960 MACs) - /encoder/layers.1/self_attn/out_proj/MatMul_quant
2026-06-19 03:53 - INFO - epu - mac_counter - conv2d: 47,185,920 ops (23,592,960 MACs) - /encoder/layers.2/self_attn/out_proj/MatMul_quant
2026-06-19 03:53 - INFO - epu - mac_counter - conv2d: 47,185,920 ops (23,592,960 MACs) - /encoder/layers.3/self_attn/out_proj/MatMul_quant
2026-06-19 03:53 - INFO - epu - mac_counter - conv2d: 47,185,920 ops (23,592,960 MACs) - /encoder/layers.4/self_attn/out_proj/MatMul_quant
2026-06-19 03:53 - INFO - epu - mac_counter - conv2d: 47,185,920 ops (23,592,960 MACs) - /encoder/layers.5/self_attn/out_proj/MatMul_quant
2026-06-19 03:53 - INFO - epu - mac_counter - ------------------------------------------------------------
2026-06-19 03:53 - INFO - epu - mac_counter - Total: 283,115,520 ops (141,557,760 MACs)
2026-06-19 03:53 - INFO - epu - mac_counter - ============================================================
2026-06-19 03:53 - INFO - epu - mac_counter -
2026-06-19 03:53 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-06-19 03:53 - INFO - epu - codegen - START=============================plan_lrm_virtual
2026-06-19 03:54 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-06-19 03:54 - INFO - epu - codegen - START===============================lrm_alloc_loop
2026-06-19 03:55 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-06-19 03:55 - INFO - epu - codegen - START================================lrm_splitting
2026-06-19 03:56 - INFO - epu - codegen - START==============================ext_split_relay
2026-06-19 03:56 - INFO - epu - codegen - START====================================build_tir
2026-06-19 03:57 - INFO - epu - chimera_job - Compilation of detr_3_transformer_encoder_opt_sym_int8_q_QC_U_1d56_8MB_4kB_64GBps_64GBps_16_OFF_x1_x1 successful
Compilation successful!
╒═════════════════════╤════════════════════════════════════════════════════════════════════════════════════════╕
│ Module Name │ detr_3_transformer_encoder_opt_sym_int8_q_QC_U_1d56_8MB_4kB_64GBps_64GBps_16_OFF_x1_x1 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ ONNX File │ onnx/detr_3_transformer_encoder_opt_sym_int8_q.onnx │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ Product Target │ QC-U │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ Number of Cores │ 1 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ ISS Clock Frequency │ 1.560 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ L2M Size │ 8MB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ LRM Size │ 4kB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ External Read BW │ 64GBps │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ External Write BW │ 64GBps │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ MACS per PE │ 16 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ Max L2M │ 6.475MB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ Max LRM │ 0.250kB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ Max Temp Ext Bytes │ 0.000MB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ Network GMACs │ 3.229 │
╘═════════════════════╧════════════════════════════════════════════════════════════════════════════════════════╛
╒════╤════════╤════════════════╤═══════════════╤══════════════════════════╤═══════╕
│ │ Type │ Name │ shape │ type │ mse │
╞════╪════════╪════════════════╪═══════════════╪══════════════════════════╪═══════╡
│ 0 │ Input │ src │ [1, 360, 256] │ tensor[FixedPoint32<30>] │ n/a │
├────┼────────┼────────────────┼───────────────┼──────────────────────────┼───────┤
│ 1 │ Input │ pos_embed │ [1, 360, 256] │ tensor[FixedPoint32<29>] │ n/a │
├────┼────────┼────────────────┼───────────────┼──────────────────────────┼───────┤
│ 2 │ Output │ encoder_memory │ [1, 360, 256] │ tensor[FixedPoint32<28>] │ n/a │
╘════╧════════╧════════════════╧═══════════════╧══════════════════════════╧═══════╛
4. Validate: ISS vs ORT
Compare the ISS (hardware simulator) output against the ONNX Runtime reference to verify numerical correctness of the compiled model.
validation_result = cgc_job.validate_ort_iss()
print(cgc_job)
print(validation_result)
2026-06-19 03:57 - INFO - epu - iss_testing - Found tranges for input: <tvm.contrib.epu.interval.Interval object at 0x7a4db58c17e0>
2026-06-19 03:57 - INFO - epu - iss_testing - Found tranges for input: <tvm.contrib.epu.interval.Interval object at 0x7a4f0effca90>
2026-06-19 03:57 - INFO - epu - iss_testing - Found tranges for input: <tvm.contrib.epu.interval.Interval object at 0x7a4db58c17e0>
2026-06-19 03:57 - INFO - epu - iss_testing - Found tranges for input: <tvm.contrib.epu.interval.Interval object at 0x7a4db58c17e0>
2026-06-19 03:57 - INFO - epu - iss_testing - Started Executing Onnxruntime...
2026-06-19 03:57 - INFO - epu - iss_testing - Done 0:00:00.319419
2026-06-19 03:57 - INFO - epu - iss_testing - Found tranges for input: <tvm.contrib.epu.interval.Interval object at 0x7a4db58c17e0>
2026-06-19 03:57 - INFO - epu - iss_testing - Found tranges for input: <tvm.contrib.epu.interval.Interval object at 0x7a4f0effca90>
FILM 79/79: 100%|███████████████████████████████████████████████████| 79/79 [01:49<00:00, 1.39s/it]
2026-06-19 03:59 - WARNING - epu - iss_testing - Node was skipped due to being multi-output: /encoder/layers.1/self_attn/MatMul_1_quant
2026-06-19 03:59 - WARNING - epu - iss_testing - Node was skipped due to being multi-output: /encoder/layers.2/self_attn/MatMul_1_quant
2026-06-19 03:59 - WARNING - epu - iss_testing - Node was skipped due to being multi-output: /encoder/layers.0/self_attn/MatMul_1_quant
2026-06-19 03:59 - WARNING - epu - iss_testing - Node was skipped due to being multi-output: /encoder/layers.3/self_attn/MatMul_1_quant
2026-06-19 03:59 - WARNING - epu - iss_testing - Node was skipped due to being multi-output: /encoder/layers.4/self_attn/MatMul_1_quant
2026-06-19 03:59 - WARNING - epu - iss_testing - Node was skipped due to being multi-output: /encoder/layers.5/self_attn/MatMul_1_quant
2026-06-19 03:59 - INFO - epu - iss_testing -
======================================================================
ISS validation results (quantized model, rtol=0.1, atol=-1)
======================================================================
pos_embed_QuantizeLinear:0 (PASS, Node, int8)
Bitwise Mismatches 0 / 92160 (0.0000%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0
PSNR inf dB
Max Abs Err 0
Max Abs Err / ε 0
Max Err Loc (0, 0, 0)
ORT @ max err 56
ISS @ max err 56
src_QuantizeLinear:0 (PASS, Node, int8)
Bitwise Mismatches 1 / 92160 (0.0011%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0.00329404
PSNR 97.78 dB
Max Abs Err 1
Max Abs Err / ε 1
Max Err Loc (0, 48, 202)
ORT @ max err 62
ISS @ max err 63
/encoder/layers.0/self_attn/Add_quant:0 (PASS, Node, int8)
Bitwise Mismatches 1 / 92160 (0.0011%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0.00329404
PSNR 97.78 dB
Max Abs Err 1
Max Abs Err / ε 1
Max Err Loc (0, 48, 202)
ORT @ max err 55
ISS @ max err 56
/encoder/layers.0/self_attn/MatMul_1_quant:0 (FAIL, Node, int8)
Bitwise Mismatches 33769 / 92160 (36.6417%)
Mismatches > Tol 30146 / 92160 (32.7105%)
Thresholds rtol=0.1, atol=-1
RMSE 79.2027
PSNR 10.16 dB
Max Abs Err 255
Max Abs Err / ε 255
Max Err Loc (3, 22, 17)
ORT @ max err 127
ISS @ max err -128
/encoder/layers.0/Add_output_0_DequantizeLinear:0 (PASS, Node, custom[qfp.30]32)
Bitwise Mismatches 3466 / 92160 (3.7609%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0.00351644
PSNR 54.16 dB
Max Abs Err 0.106996
Max Abs Err / ε 1.14886e+08
Max Err Loc (0, 57, 248)
ORT @ max err 0.54686964
ISS @ max err 0.4398734
/encoder/layers.0/self_attn_layer_norm/Add_1:0 (PASS, Node, custom[qfp.28]32)
Bitwise Mismatches 92157 / 92160 (99.9967%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0.0109622
PSNR 59.17 dB
Max Abs Err 0.479279
Max Abs Err / ε 1.28655e+08
Max Err Loc (0, 258, 0)
ORT @ max err 3.2069468
ISS @ max err 2.7276678
/encoder/layers.0/self_attn_layer_norm/Add_1_output_0_QuantizeLinear:0 (PASS, Node, int8)
Bitwise Mismatches 2720 / 92160 (2.9514%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0.212536
PSNR 61.58 dB
Max Abs Err 7
Max Abs Err / ε 7
Max Err Loc (0, 258, 0)
ORT @ max err 48
ISS @ max err 41
/encoder/layers.0/fc1/MatMul_quant:0 (PASS, Node, int8)
Bitwise Mismatches 114303 / 737280 (15.5033%)
Mismatches > Tol 0 / 737280 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0.395465
PSNR 56.19 dB
Max Abs Err 2
Max Abs Err / ε 2
Max Err Loc (0, 22, 26)
ORT @ max err -16
ISS @ max err -14
[ERROR] /encoder/layers.0/activation_fn/Relu:0
Error Skipping validation for /encoder/layers.0/activation_fn/Relu:0 dtype mismatch between ISS int8 and ORT float32
/encoder/layers.0/fc2/MatMul_quant:0 (PASS, Node, int8)
Bitwise Mismatches 21397 / 92160 (23.2172%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0.495652
PSNR 54.23 dB
Max Abs Err 5
Max Abs Err / ε 5
Max Err Loc (0, 22, 41)
ORT @ max err 91
ISS @ max err 96
/encoder/layers.0/Add_1_output_0_DequantizeLinear:0 (PASS, Node, custom[qfp.26]32)
Bitwise Mismatches 14581 / 92160 (15.8214%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0.0558308
PSNR 53.63 dB
Max Abs Err 0.408375
Max Abs Err / ε 2.74056e+07
Max Err Loc (0, 347, 231)
ORT @ max err -10.345478
ISS @ max err -10.753853
/encoder/layers.0/final_layer_norm/Add_1:0 (PASS, Node, custom[qfp.28]32)
Bitwise Mismatches 92118 / 92160 (99.9544%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0.0367947
PSNR 49.00 dB
Max Abs Err 0.295418
Max Abs Err / ε 7.93006e+07
Max Err Loc (0, 198, 248)
ORT @ max err -1.3612862
ISS @ max err -1.6567041
/encoder/layers.0/final_layer_norm/Add_1_output_0_QuantizeLinear:0 (PASS, Node, int8)
Bitwise Mismatches 18541 / 92160 (20.1183%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0.694863
PSNR 51.29 dB
Max Abs Err 5
Max Abs Err / ε 5
Max Err Loc (0, 57, 248)
ORT @ max err 9
ISS @ max err 4
/encoder/layers.1/self_attn/Add_quant:0 (PASS, Node, int8)
Bitwise Mismatches 18509 / 92160 (20.0836%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0.693698
PSNR 51.31 dB
Max Abs Err 5
Max Abs Err / ε 5
Max Err Loc (0, 57, 248)
ORT @ max err 24
ISS @ max err 19
/encoder/layers.1/self_attn/MatMul_1_quant:0 (FAIL, Node, int8)
Bitwise Mismatches 90569 / 92160 (98.2737%)
Mismatches > Tol 60313 / 92160 (65.4438%)
Thresholds rtol=0.1, atol=-1
RMSE 24.1854
PSNR 20.46 dB
Max Abs Err 127
Max Abs Err / ε 127
Max Err Loc (6, 252, 18)
ORT @ max err 38
ISS @ max err -89
/encoder/layers.1/Add_output_0_DequantizeLinear:0 (PASS, Node, custom[qfp.28]32)
Bitwise Mismatches 41242 / 92160 (44.7504%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0.0533656
PSNR 46.50 dB
Max Abs Err 0.359793
Max Abs Err / ε 9.65812e+07
Max Err Loc (0, 309, 66)
ORT @ max err -0.11993096
ISS @ max err 0.23986192
/encoder/layers.1/self_attn_layer_norm/Add_1:0 (PASS, Node, custom[qfp.27]32)
Bitwise Mismatches 92159 / 92160 (99.9989%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0.0514112
PSNR 51.19 dB
Max Abs Err 0.328067
Max Abs Err / ε 4.40324e+07
Max Err Loc (0, 198, 248)
ORT @ max err -0.7491393
ISS @ max err -1.0772061
/encoder/layers.1/self_attn_layer_norm/Add_1_output_0_QuantizeLinear:0 (PASS, Node, int8)
Bitwise Mismatches 29487 / 92160 (31.9954%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0.642371
PSNR 51.98 dB
Max Abs Err 4
Max Abs Err / ε 4
Max Err Loc (0, 27, 64)
ORT @ max err 18
ISS @ max err 14
/encoder/layers.1/fc1/MatMul_quant:0 (PASS, Node, int8)
Bitwise Mismatches 482902 / 737280 (65.4978%)
Mismatches > Tol 0 / 737280 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 1.17592
PSNR 46.72 dB
Max Abs Err 6
Max Abs Err / ε 6
Max Err Loc (0, 110, 497)
ORT @ max err -58
ISS @ max err -52
[ERROR] /encoder/layers.1/activation_fn/Relu:0
Error Skipping validation for /encoder/layers.1/activation_fn/Relu:0 dtype mismatch between ISS int8 and ORT float32
/encoder/layers.1/fc2/MatMul_quant:0 (PASS, Node, int8)
Bitwise Mismatches 11130 / 92160 (12.0768%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0.34797
PSNR 57.30 dB
Max Abs Err 3
Max Abs Err / ε 3
Max Err Loc (0, 263, 142)
ORT @ max err -4
ISS @ max err -7
/encoder/layers.1/Add_1_output_0_DequantizeLinear:0 (PASS, Node, custom[qfp.25]32)
Bitwise Mismatches 19912 / 92160 (21.6059%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0.122279
PSNR 49.33 dB
Max Abs Err 0.503921
Max Abs Err / ε 1.69088e+07
Max Err Loc (0, 24, 41)
ORT @ max err 11.338197
ISS @ max err 10.834276
/encoder/layers.1/final_layer_norm/Add_1:0 (PASS, Node, custom[qfp.28]32)
Bitwise Mismatches 92122 / 92160 (99.9588%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0.076319
PSNR 41.85 dB
Max Abs Err 0.469571
Max Abs Err / ε 1.2605e+08
Max Err Loc (0, 112, 42)
ORT @ max err -0.07682519
ISS @ max err -0.5463965
/encoder/layers.1/final_layer_norm/Add_1_output_0_QuantizeLinear:0 (PASS, Node, int8)
Bitwise Mismatches 26281 / 92160 (28.5167%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 1.42577
PSNR 45.05 dB
Max Abs Err 9
Max Abs Err / ε 9
Max Err Loc (0, 112, 42)
ORT @ max err -1
ISS @ max err -10
/encoder/layers.2/self_attn/Add_quant:0 (PASS, Node, int8)
Bitwise Mismatches 26259 / 92160 (28.4928%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 1.42253
PSNR 45.07 dB
Max Abs Err 9
Max Abs Err / ε 9
Max Err Loc (0, 112, 42)
ORT @ max err 2
ISS @ max err -7
/encoder/layers.2/self_attn/MatMul_1_quant:0 (FAIL, Node, int8)
Bitwise Mismatches 89636 / 92160 (97.2613%)
Mismatches > Tol 53325 / 92160 (57.8613%)
Thresholds rtol=0.1, atol=-1
RMSE 18.1951
PSNR 22.93 dB
Max Abs Err 129
Max Abs Err / ε 129
Max Err Loc (1, 264, 17)
ORT @ max err -63
ISS @ max err 66
/encoder/layers.2/Add_output_0_DequantizeLinear:0 (PASS, Node, custom[qfp.28]32)
Bitwise Mismatches 48235 / 92160 (52.3383%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0.0854692
PSNR 41.10 dB
Max Abs Err 0.557728
Max Abs Err / ε 1.49714e+08
Max Err Loc (0, 233, 108)
ORT @ max err -1.2270007
ISS @ max err -0.66927314
/encoder/layers.2/self_attn_layer_norm/Add_1:0 (PASS, Node, custom[qfp.27]32)
Bitwise Mismatches 92160 / 92160 (100.0000%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0.110867
PSNR 44.52 dB
Max Abs Err 0.82206
Max Abs Err / ε 1.10335e+08
Max Err Loc (0, 112, 42)
ORT @ max err 0.45826334
ISS @ max err -0.36379716
/encoder/layers.2/self_attn_layer_norm/Add_1_output_0_QuantizeLinear:0 (PASS, Node, int8)
Bitwise Mismatches 43618 / 92160 (47.3286%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 1.28462
PSNR 45.96 dB
Max Abs Err 9
Max Abs Err / ε 9
Max Err Loc (0, 112, 42)
ORT @ max err 5
ISS @ max err -4
/encoder/layers.2/fc1/MatMul_quant:0 (PASS, Node, int8)
Bitwise Mismatches 578760 / 737280 (78.4993%)
Mismatches > Tol 0 / 737280 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 1.90935
PSNR 42.51 dB
Max Abs Err 12
Max Abs Err / ε 12
Max Err Loc (0, 123, 1428)
ORT @ max err -23
ISS @ max err -11
[ERROR] /encoder/layers.2/activation_fn/Relu:0
Error Skipping validation for /encoder/layers.2/activation_fn/Relu:0 dtype mismatch between ISS int8 and ORT float32
/encoder/layers.2/fc2/MatMul_quant:0 (PASS, Node, int8)
Bitwise Mismatches 20876 / 92160 (22.6519%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0.505718
PSNR 54.05 dB
Max Abs Err 8
Max Abs Err / ε 8
Max Err Loc (0, 67, 142)
ORT @ max err -17
ISS @ max err -9
/encoder/layers.2/Add_1_output_0_DequantizeLinear:0 (PASS, Node, custom[qfp.25]32)
Bitwise Mismatches 28084 / 92160 (30.4731%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0.289726
PSNR 47.42 dB
Max Abs Err 3.8374
Max Abs Err / ε 1.28762e+08
Max Err Loc (0, 67, 142)
ORT @ max err -8.154483
ISS @ max err -4.317079
/encoder/layers.2/final_layer_norm/Add_1:0 (FAIL, Node, custom[qfp.28]32)
Bitwise Mismatches 92122 / 92160 (99.9588%)
Mismatches > Tol 135 / 92160 (0.1465%)
Thresholds rtol=0.1, atol=-1
RMSE 0.125928
PSNR 36.64 dB
Max Abs Err 1.04652
Max Abs Err / ε 2.80922e+08
Max Err Loc (0, 112, 42)
ORT @ max err 0.22258396
ISS @ max err -0.82393265
/encoder/layers.2/final_layer_norm/Add_1_output_0_QuantizeLinear:0 (FAIL, Node, int8)
Bitwise Mismatches 45789 / 92160 (49.6842%)
Mismatches > Tol 252 / 92160 (0.2734%)
Thresholds rtol=0.1, atol=-1
RMSE 3.42787
PSNR 37.43 dB
Max Abs Err 28
Max Abs Err / ε 28
Max Err Loc (0, 112, 42)
ORT @ max err 6
ISS @ max err -22
/encoder/layers.3/self_attn/Add_quant:0 (FAIL, Node, int8)
Bitwise Mismatches 44556 / 92160 (48.3464%)
Mismatches > Tol 227 / 92160 (0.2463%)
Thresholds rtol=0.1, atol=-1
RMSE 3.20317
PSNR 38.02 dB
Max Abs Err 26
Max Abs Err / ε 26
Max Err Loc (0, 112, 42)
ORT @ max err 10
ISS @ max err -16
/encoder/layers.3/self_attn/MatMul_1_quant:0 (FAIL, Node, int8)
Bitwise Mismatches 70271 / 92160 (76.2489%)
Mismatches > Tol 46013 / 92160 (49.9273%)
Thresholds rtol=0.1, atol=-1
RMSE 14.9774
PSNR 24.62 dB
Max Abs Err 89
Max Abs Err / ε 89
Max Err Loc (5, 276, 22)
ORT @ max err 23
ISS @ max err -66
/encoder/layers.3/Add_output_0_DequantizeLinear:0 (FAIL, Node, custom[qfp.28]32)
Bitwise Mismatches 67436 / 92160 (73.1727%)
Mismatches > Tol 172 / 92160 (0.1866%)
Thresholds rtol=0.1, atol=-1
RMSE 0.133232
PSNR 36.32 dB
Max Abs Err 1.00012
Max Abs Err / ε 2.68467e+08
Max Err Loc (0, 112, 42)
ORT @ max err 0.24002816
ISS @ max err -0.76008916
/encoder/layers.3/self_attn_layer_norm/Add_1:0 (FAIL, Node, custom[qfp.27]32)
Bitwise Mismatches 92160 / 92160 (100.0000%)
Mismatches > Tol 16 / 92160 (0.0174%)
Thresholds rtol=0.1, atol=-1
RMSE 0.178997
PSNR 41.03 dB
Max Abs Err 1.8073
Max Abs Err / ε 2.42572e+08
Max Err Loc (0, 112, 42)
ORT @ max err 1.0772173
ISS @ max err -0.7300843
/encoder/layers.3/self_attn_layer_norm/Add_1_output_0_QuantizeLinear:0 (FAIL, Node, int8)
Bitwise Mismatches 61278 / 92160 (66.4909%)
Mismatches > Tol 96 / 92160 (0.1042%)
Thresholds rtol=0.1, atol=-1
RMSE 2.83473
PSNR 39.08 dB
Max Abs Err 28
Max Abs Err / ε 28
Max Err Loc (0, 112, 42)
ORT @ max err 17
ISS @ max err -11
/encoder/layers.3/fc1/MatMul_quant:0 (FAIL, Node, int8)
Bitwise Mismatches 634862 / 737280 (86.1087%)
Mismatches > Tol 107 / 737280 (0.0145%)
Thresholds rtol=0.1, atol=-1
RMSE 3.13227
PSNR 38.21 dB
Max Abs Err 24
Max Abs Err / ε 24
Max Err Loc (0, 67, 1226)
ORT @ max err 27
ISS @ max err 3
[ERROR] /encoder/layers.3/activation_fn/Relu:0
Error Skipping validation for /encoder/layers.3/activation_fn/Relu:0 dtype mismatch between ISS int8 and ORT float32
/encoder/layers.3/fc2/MatMul_quant:0 (FAIL, Node, int8)
Bitwise Mismatches 24535 / 92160 (26.6222%)
Mismatches > Tol 5 / 92160 (0.0054%)
Thresholds rtol=0.1, atol=-1
RMSE 0.68146
PSNR 51.46 dB
Max Abs Err 27
Max Abs Err / ε 27
Max Err Loc (0, 223, 93)
ORT @ max err -76
ISS @ max err -49
/encoder/layers.3/Add_1_output_0_DequantizeLinear:0 (FAIL, Node, custom[qfp.24]32)
Bitwise Mismatches 33772 / 92160 (36.6450%)
Mismatches > Tol 3 / 92160 (0.0033%)
Thresholds rtol=0.1, atol=-1
RMSE 0.444631
PSNR 49.54 dB
Max Abs Err 13.8495
Max Abs Err / ε 2.32355e+08
Max Err Loc (0, 223, 93)
ORT @ max err -47.318962
ISS @ max err -33.46951
/encoder/layers.3/final_layer_norm/Add_1:0 (FAIL, Node, custom[qfp.28]32)
Bitwise Mismatches 92160 / 92160 (100.0000%)
Mismatches > Tol 468 / 92160 (0.5078%)
Thresholds rtol=0.1, atol=-1
RMSE 0.161073
PSNR 36.48 dB
Max Abs Err 1.34101
Max Abs Err / ε 3.59974e+08
Max Err Loc (0, 67, 231)
ORT @ max err -0.8965663
ISS @ max err -2.237573
/encoder/layers.3/final_layer_norm/Add_1_output_0_QuantizeLinear:0 (FAIL, Node, int8)
Bitwise Mismatches 49536 / 92160 (53.7500%)
Mismatches > Tol 572 / 92160 (0.6207%)
Thresholds rtol=0.1, atol=-1
RMSE 3.78958
PSNR 36.56 dB
Max Abs Err 31
Max Abs Err / ε 31
Max Err Loc (0, 67, 231)
ORT @ max err -21
ISS @ max err -52
/encoder/layers.4/self_attn/Add_quant:0 (FAIL, Node, int8)
Bitwise Mismatches 48235 / 92160 (52.3383%)
Mismatches > Tol 388 / 92160 (0.4210%)
Thresholds rtol=0.1, atol=-1
RMSE 3.51736
PSNR 37.21 dB
Max Abs Err 28
Max Abs Err / ε 28
Max Err Loc (0, 67, 231)
ORT @ max err -6
ISS @ max err -34
/encoder/layers.4/self_attn/MatMul_1_quant:0 (FAIL, Node, int8)
Bitwise Mismatches 89062 / 92160 (96.6385%)
Mismatches > Tol 36940 / 92160 (40.0825%)
Thresholds rtol=0.1, atol=-1
RMSE 14.4608
PSNR 24.93 dB
Max Abs Err 105
Max Abs Err / ε 105
Max Err Loc (6, 305, 6)
ORT @ max err -4
ISS @ max err 101
/encoder/layers.4/Add_output_0_DequantizeLinear:0 (FAIL, Node, custom[qfp.28]32)
Bitwise Mismatches 71665 / 92160 (77.7615%)
Mismatches > Tol 516 / 92160 (0.5599%)
Thresholds rtol=0.1, atol=-1
RMSE 0.168584
PSNR 36.36 dB
Max Abs Err 1.45218
Max Abs Err / ε 3.89818e+08
Max Err Loc (0, 46, 147)
ORT @ max err -0.04400559
ISS @ max err 1.4081789
/encoder/layers.4/self_attn_layer_norm/Add_1:0 (FAIL, Node, custom[qfp.26]32)
Bitwise Mismatches 92158 / 92160 (99.9978%)
Mismatches > Tol 2 / 92160 (0.0022%)
Thresholds rtol=0.1, atol=-1
RMSE 0.212122
PSNR 43.46 dB
Max Abs Err 2.21631
Max Abs Err / ε 1.48734e+08
Max Err Loc (0, 46, 47)
ORT @ max err -10.862047
ISS @ max err -8.6457405
/encoder/layers.4/self_attn_layer_norm/Add_1_output_0_QuantizeLinear:0 (FAIL, Node, int8)
Bitwise Mismatches 54554 / 92160 (59.1949%)
Mismatches > Tol 10 / 92160 (0.0109%)
Thresholds rtol=0.1, atol=-1
RMSE 1.96877
PSNR 42.25 dB
Max Abs Err 20
Max Abs Err / ε 20
Max Err Loc (0, 46, 47)
ORT @ max err -99
ISS @ max err -79
/encoder/layers.4/fc1/MatMul_quant:0 (FAIL, Node, int8)
Bitwise Mismatches 629148 / 737280 (85.3337%)
Mismatches > Tol 588 / 737280 (0.0798%)
Thresholds rtol=0.1, atol=-1
RMSE 3.43749
PSNR 37.41 dB
Max Abs Err 45
Max Abs Err / ε 45
Max Err Loc (0, 68, 1442)
ORT @ max err -52
ISS @ max err -7
[ERROR] /encoder/layers.4/activation_fn/Relu:0
Error Skipping validation for /encoder/layers.4/activation_fn/Relu:0 dtype mismatch between ISS int8 and ORT float32
/encoder/layers.4/fc2/MatMul_quant:0 (FAIL, Node, int8)
Bitwise Mismatches 45716 / 92160 (49.6050%)
Mismatches > Tol 39 / 92160 (0.0423%)
Thresholds rtol=0.1, atol=-1
RMSE 1.26705
PSNR 46.07 dB
Max Abs Err 47
Max Abs Err / ε 47
Max Err Loc (0, 183, 114)
ORT @ max err 106
ISS @ max err 59
/encoder/layers.4/Add_1_output_0_DequantizeLinear:0 (FAIL, Node, custom[qfp.26]32)
Bitwise Mismatches 52829 / 92160 (57.3231%)
Mismatches > Tol 8 / 92160 (0.0087%)
Thresholds rtol=0.1, atol=-1
RMSE 0.282933
PSNR 45.46 dB
Max Abs Err 5.79824
Max Abs Err / ε 3.89113e+08
Max Err Loc (0, 183, 114)
ORT @ max err 25.984707
ISS @ max err 20.186466
/encoder/layers.4/final_layer_norm/Add_1:0 (FAIL, Node, custom[qfp.28]32)
Bitwise Mismatches 92160 / 92160 (100.0000%)
Mismatches > Tol 144 / 92160 (0.1562%)
Thresholds rtol=0.1, atol=-1
RMSE 0.128682
PSNR 38.21 dB
Max Abs Err 1.25153
Max Abs Err / ε 3.35955e+08
Max Err Loc (0, 67, 231)
ORT @ max err -0.32758018
ISS @ max err -1.5791091
/encoder/layers.4/final_layer_norm/Add_1_output_0_QuantizeLinear:0 (FAIL, Node, int8)
Bitwise Mismatches 54992 / 92160 (59.6701%)
Mismatches > Tol 172 / 92160 (0.1866%)
Thresholds rtol=0.1, atol=-1
RMSE 2.98804
PSNR 38.62 dB
Max Abs Err 28
Max Abs Err / ε 28
Max Err Loc (0, 67, 231)
ORT @ max err -8
ISS @ max err -36
/encoder/layers.5/self_attn/Add_quant:0 (FAIL, Node, int8)
Bitwise Mismatches 53908 / 92160 (58.4939%)
Mismatches > Tol 125 / 92160 (0.1356%)
Thresholds rtol=0.1, atol=-1
RMSE 2.72479
PSNR 39.42 dB
Max Abs Err 25
Max Abs Err / ε 25
Max Err Loc (0, 67, 231)
ORT @ max err 6
ISS @ max err -19
/encoder/layers.5/self_attn/MatMul_1_quant:0 (FAIL, Node, int8)
Bitwise Mismatches 80215 / 92160 (87.0388%)
Mismatches > Tol 46951 / 92160 (50.9451%)
Thresholds rtol=0.1, atol=-1
RMSE 15.2052
PSNR 24.49 dB
Max Abs Err 100
Max Abs Err / ε 100
Max Err Loc (3, 80, 0)
ORT @ max err -41
ISS @ max err 59
/encoder/layers.5/Add_output_0_DequantizeLinear:0 (FAIL, Node, custom[qfp.28]32)
Bitwise Mismatches 67245 / 92160 (72.9655%)
Mismatches > Tol 181 / 92160 (0.1964%)
Thresholds rtol=0.1, atol=-1
RMSE 0.132061
PSNR 37.88 dB
Max Abs Err 1.24984
Max Abs Err / ε 3.35501e+08
Max Err Loc (0, 67, 231)
ORT @ max err -0.21548934
ISS @ max err -1.4653275
/encoder/layers.5/self_attn_layer_norm/Add_1:0 (FAIL, Node, custom[qfp.27]32)
Bitwise Mismatches 92160 / 92160 (100.0000%)
Mismatches > Tol 37 / 92160 (0.0401%)
Thresholds rtol=0.1, atol=-1
RMSE 0.195842
PSNR 41.80 dB
Max Abs Err 2.49028
Max Abs Err / ε 3.34239e+08
Max Err Loc (0, 232, 217)
ORT @ max err 0.4670541
ISS @ max err 2.957331
/encoder/layers.5/self_attn_layer_norm/Add_1_output_0_QuantizeLinear:0 (FAIL, Node, int8)
Bitwise Mismatches 57849 / 92160 (62.7702%)
Mismatches > Tol 49 / 92160 (0.0532%)
Thresholds rtol=0.1, atol=-1
RMSE 2.02433
PSNR 42.01 dB
Max Abs Err 25
Max Abs Err / ε 25
Max Err Loc (0, 67, 231)
ORT @ max err -6
ISS @ max err -31
/encoder/layers.5/fc1/MatMul_quant:0 (FAIL, Node, int8)
Bitwise Mismatches 612193 / 737280 (83.0340%)
Mismatches > Tol 172 / 737280 (0.0233%)
Thresholds rtol=0.1, atol=-1
RMSE 2.85078
PSNR 39.03 dB
Max Abs Err 25
Max Abs Err / ε 25
Max Err Loc (0, 46, 967)
ORT @ max err -15
ISS @ max err 10
[ERROR] /encoder/layers.5/activation_fn/Relu:0
Error Skipping validation for /encoder/layers.5/activation_fn/Relu:0 dtype mismatch between ISS int8 and ORT float32
/encoder/layers.5/fc2/MatMul_quant:0 (FAIL, Node, int8)
Bitwise Mismatches 34856 / 92160 (37.8212%)
Mismatches > Tol 95 / 92160 (0.1031%)
Thresholds rtol=0.1, atol=-1
RMSE 0.796838
PSNR 50.10 dB
Max Abs Err 14
Max Abs Err / ε 14
Max Err Loc (0, 67, 142)
ORT @ max err -25
ISS @ max err -11
/encoder/layers.5/Add_1_output_0_DequantizeLinear:0 (FAIL, Node, custom[qfp.26]32)
Bitwise Mismatches 53900 / 92160 (58.4852%)
Mismatches > Tol 21 / 92160 (0.0228%)
Thresholds rtol=0.1, atol=-1
RMSE 0.215049
PSNR 42.04 dB
Max Abs Err 2.61665
Max Abs Err / ε 1.756e+08
Max Err Loc (0, 232, 217)
ORT @ max err 0.29073894
ISS @ max err 2.9073894
/encoder/layers.5/final_layer_norm/Add_1:0 (FAIL, Node, custom[qfp.28]32)
Bitwise Mismatches 92160 / 92160 (100.0000%)
Mismatches > Tol 709 / 92160 (0.7693%)
Thresholds rtol=0.1, atol=-1
RMSE 0.116281
PSNR 37.20 dB
Max Abs Err 1.34301
Max Abs Err / ε 3.60511e+08
Max Err Loc (0, 232, 217)
ORT @ max err 0.25954124
ISS @ max err 1.6025488
encoder_memory (FAIL, Output, custom[qfp.28]32)
Bitwise Mismatches 92160 / 92160 (100.0000%)
Mismatches > Tol 709 / 92160 (0.7693%)
Thresholds rtol=0.1, atol=-1
RMSE 0.116281
PSNR 37.20 dB
Max Abs Err 1.34301
Max Abs Err / ε 3.60511e+08
Max Err Loc (0, 232, 217)
ORT @ max err 0.25954124
ISS @ max err 1.6025488
68 entries
Differences detected between ort and iss
╒═════════════════════╤════════════════════════════════════════════════════════════════════════════════════════╕
│ Module Name │ detr_3_transformer_encoder_opt_sym_int8_q_QC_U_1d56_8MB_4kB_64GBps_64GBps_16_OFF_x1_x1 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ ONNX File │ onnx/detr_3_transformer_encoder_opt_sym_int8_q.onnx │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ Product Target │ QC-U │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ Number of Cores │ 1 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ ISS Clock Frequency │ 1.560 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ L2M Size │ 8MB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ LRM Size │ 4kB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ External Read BW │ 64GBps │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ External Write BW │ 64GBps │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ MACS per PE │ 16 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ Max L2M │ 6.475MB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ Max LRM │ 0.250kB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ Max Temp Ext Bytes │ 0.000MB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ Network GMACs │ 3.229 │
╘═════════════════════╧════════════════════════════════════════════════════════════════════════════════════════╛
╒════╤════════╤════════════════╤═══════════════╤══════════════════════════╤═══════╕
│ │ Type │ Name │ shape │ type │ mse │
╞════╪════════╪════════════════╪═══════════════╪══════════════════════════╪═══════╡
│ 0 │ Input │ src │ [1, 360, 256] │ tensor[FixedPoint32<30>] │ n/a │
├────┼────────┼────────────────┼───────────────┼──────────────────────────┼───────┤
│ 1 │ Input │ pos_embed │ [1, 360, 256] │ tensor[FixedPoint32<29>] │ n/a │
├────┼────────┼────────────────┼───────────────┼──────────────────────────┼───────┤
│ 2 │ Output │ encoder_memory │ [1, 360, 256] │ tensor[FixedPoint32<28>] │ n/a │
╘════╧════════╧════════════════╧═══════════════╧══════════════════════════╧═══════╛
Post-ISS Report 1.56 GHz ***
Fully placed-and-routed gate simulation:
╒══════════════════════════════════╤═════════╕
│ Latency (ms) │ 5.56 │
├──────────────────────────────────┼─────────┤
│ FPS │ 179.84 │
├──────────────────────────────────┼─────────┤
│ Average Power @ 3nm SSGNP (mW) │ 1152.60 │
├──────────────────────────────────┼─────────┤
│ FPS per Watt @ 3nm SSGNP (FPS/W) │ 156.03 │
├──────────────────────────────────┼─────────┤
│ Ext Rd Bytes (MB) │ 13.07 │
├──────────────────────────────────┼─────────┤
│ Ext Wr Bytes (MB) │ 0.35 │
├──────────────────────────────────┼─────────┤
│ Avg Ext Rd BW (GBps) │ 2.29 │
├──────────────────────────────────┼─────────┤
│ Avg Ext Wr BW (GBps) │ 0.06 │
├──────────────────────────────────┼─────────┤
│ MAC Utilization │ 2.27% │
╘══════════════════════════════════╧═════════╛
*** Data generated using 7nm SSGNP gatesim and scaled to 3nm
[SDK-CLI] : TotalCycles: 8,674,246
[SDK-CLI] : Executions/second: 179.84
compute : ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 1.85M
data_array : ▇▇ 253.084K
mac : ▇▇▇▇ 489.695K
data_external: ▏ 25.294K
data_ocm : ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 6.007M
for more information check run directory: /quadric/sdk-cli/examples/models/detr/encoder/ccl_build/detr_3_transformer_encoder_opt_sym_int8_q_QC_U_1d56_8MB_4kB_64GBps_64GBps_16_OFF_x1_x1/build
[{'S': 'PASS', 'Type': 'Node', 'Name': 'pos_embed_QuantizeLinear:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 0, 'Mismatch %': 0.0, 'RMSE': 0.0, 'PSNR (dB)': inf, 'Max Abs Err': 0.0, 'Max Abs Err Loc': (0, 0, 0), 'ORT[loc]': 56, 'ISS[loc]': 56, 'Max Abs Err / ε': 0.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Node', 'Name': 'src_QuantizeLinear:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 1, 'Mismatch %': 0.0010850694444444445, 'RMSE': 0.0032940392293420617, 'PSNR (dB)': 97.77622826947047, 'Max Abs Err': 1.0, 'Max Abs Err Loc': (0, 48, 202), 'ORT[loc]': 62, 'ISS[loc]': 63, 'Max Abs Err / ε': 1.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.0/self_attn/Add_quant:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 1, 'Mismatch %': 0.0010850694444444445, 'RMSE': 0.0032940392293420617, 'PSNR (dB)': 97.77622826947047, 'Max Abs Err': 1.0, 'Max Abs Err Loc': (0, 48, 202), 'ORT[loc]': 55, 'ISS[loc]': 56, 'Max Abs Err / ε': 1.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.0/self_attn/MatMul_1_quant:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 33769, 'Mismatch %': 36.64171006944444, 'RMSE': 79.20266225259923, 'PSNR (dB)': 10.156008011691945, 'Max Abs Err': 255.0, 'Max Abs Err Loc': (3, 22, 17), 'ORT[loc]': 127, 'ISS[loc]': -128, 'Max Abs Err / ε': 255.0, 'Mismatches Above Tol': 30146, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.0/Add_output_0_DequantizeLinear:0', 'dtype': 'custom[qfp.30]32', 'total': 92160, 'Bitwise Mismatches': 3466, 'Mismatch %': 3.7608506944444446, 'RMSE': 0.0035164389752694735, 'PSNR (dB)': 54.15999706157728, 'Max Abs Err': 0.10699623823165894, 'Max Abs Err Loc': (0, 57, 248), 'ORT[loc]': 0.54686964, 'ISS[loc]': 0.4398734, 'Max Abs Err / ε': 114886336.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.0/self_attn_layer_norm/Add_1:0', 'dtype': 'custom[qfp.28]32', 'total': 92160, 'Bitwise Mismatches': 92157, 'Mismatch %': 99.99674479166667, 'RMSE': 0.010962241636961013, 'PSNR (dB)': 59.17012431855249, 'Max Abs Err': 0.4792790412902832, 'Max Abs Err Loc': (0, 258, 0), 'ORT[loc]': 3.2069468, 'ISS[loc]': 2.7276678, 'Max Abs Err / ε': 128655488.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.0/self_attn_layer_norm/Add_1_output_0_QuantizeLinear:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 2720, 'Mismatch %': 2.951388888888889, 'RMSE': 0.21253574045845142, 'PSNR (dB)': 61.5821641606027, 'Max Abs Err': 7.0, 'Max Abs Err Loc': (0, 258, 0), 'ORT[loc]': 48, 'ISS[loc]': 41, 'Max Abs Err / ε': 7.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.0/fc1/MatMul_quant:0', 'dtype': 'int8', 'total': 737280, 'Bitwise Mismatches': 114303, 'Mismatch %': 15.503336588541666, 'RMSE': 0.39546480926194094, 'PSNR (dB)': 56.18864673814042, 'Max Abs Err': 2.0, 'Max Abs Err Loc': (0, 22, 26), 'ORT[loc]': -16, 'ISS[loc]': -14, 'Max Abs Err / ε': 2.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'SKIPPED', 'Type': 'Node', 'Name': '/encoder/layers.0/activation_fn/Relu:0', 'dtype': 'int8', 'Mismatches Above Tol': None, 'total': 0, 'rtol': 0.1, 'atol': -1, 'Comment': 'Skipping validation for /encoder/layers.0/activation_fn/Relu:0 dtype mismatch between ISS int8 and ORT float32'}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.0/fc2/MatMul_quant:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 21397, 'Mismatch %': 23.21723090277778, 'RMSE': 0.4956516648985118, 'PSNR (dB)': 54.22727222235987, 'Max Abs Err': 5.0, 'Max Abs Err Loc': (0, 22, 41), 'ORT[loc]': 91, 'ISS[loc]': 96, 'Max Abs Err / ε': 5.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.0/Add_1_output_0_DequantizeLinear:0', 'dtype': 'custom[qfp.26]32', 'total': 92160, 'Bitwise Mismatches': 14581, 'Mismatch %': 15.821397569444445, 'RMSE': 0.05583077092480372, 'PSNR (dB)': 53.63059185175636, 'Max Abs Err': 0.4083747863769531, 'Max Abs Err Loc': (0, 347, 231), 'ORT[loc]': -10.345478, 'ISS[loc]': -10.753853, 'Max Abs Err / ε': 27405568.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.0/final_layer_norm/Add_1:0', 'dtype': 'custom[qfp.28]32', 'total': 92160, 'Bitwise Mismatches': 92118, 'Mismatch %': 99.95442708333333, 'RMSE': 0.036794696152054954, 'PSNR (dB)': 48.995217981414896, 'Max Abs Err': 0.2954179048538208, 'Max Abs Err Loc': (0, 198, 248), 'ORT[loc]': -1.3612862, 'ISS[loc]': -1.6567041, 'Max Abs Err / ε': 79300640.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.0/final_layer_norm/Add_1_output_0_QuantizeLinear:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 18541, 'Mismatch %': 20.118272569444443, 'RMSE': 0.6948627212542697, 'PSNR (dB)': 51.292823352578736, 'Max Abs Err': 5.0, 'Max Abs Err Loc': (0, 57, 248), 'ORT[loc]': 9, 'ISS[loc]': 4, 'Max Abs Err / ε': 5.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.1/self_attn/Add_quant:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 18509, 'Mismatch %': 20.08355034722222, 'RMSE': 0.6936983839657309, 'PSNR (dB)': 51.3073899532447, 'Max Abs Err': 5.0, 'Max Abs Err Loc': (0, 57, 248), 'ORT[loc]': 24, 'ISS[loc]': 19, 'Max Abs Err / ε': 5.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.1/self_attn/MatMul_1_quant:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 90569, 'Mismatch %': 98.27365451388889, 'RMSE': 24.185394593265258, 'PSNR (dB)': 20.459740059672598, 'Max Abs Err': 127.0, 'Max Abs Err Loc': (6, 252, 18), 'ORT[loc]': 38, 'ISS[loc]': -89, 'Max Abs Err / ε': 127.0, 'Mismatches Above Tol': 60313, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.1/Add_output_0_DequantizeLinear:0', 'dtype': 'custom[qfp.28]32', 'total': 92160, 'Bitwise Mismatches': 41242, 'Mismatch %': 44.75043402777778, 'RMSE': 0.05336564479763275, 'PSNR (dB)': 46.495948215173755, 'Max Abs Err': 0.35979288071393967, 'Max Abs Err Loc': (0, 309, 66), 'ORT[loc]': -0.11993096, 'ISS[loc]': 0.23986192, 'Max Abs Err / ε': 96581166.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.1/self_attn_layer_norm/Add_1:0', 'dtype': 'custom[qfp.27]32', 'total': 92160, 'Bitwise Mismatches': 92159, 'Mismatch %': 99.99891493055556, 'RMSE': 0.051411211350449106, 'PSNR (dB)': 51.18846236528153, 'Max Abs Err': 0.3280668258666992, 'Max Abs Err Loc': (0, 198, 248), 'ORT[loc]': -0.7491393, 'ISS[loc]': -1.0772061, 'Max Abs Err / ε': 44032384.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.1/self_attn_layer_norm/Add_1_output_0_QuantizeLinear:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 29487, 'Mismatch %': 31.995442708333332, 'RMSE': 0.6423714338509907, 'PSNR (dB)': 51.97507921472144, 'Max Abs Err': 4.0, 'Max Abs Err Loc': (0, 27, 64), 'ORT[loc]': 18, 'ISS[loc]': 14, 'Max Abs Err / ε': 4.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.1/fc1/MatMul_quant:0', 'dtype': 'int8', 'total': 737280, 'Bitwise Mismatches': 482902, 'Mismatch %': 65.49777560763889, 'RMSE': 1.1759172182825932, 'PSNR (dB)': 46.72326861789876, 'Max Abs Err': 6.0, 'Max Abs Err Loc': (0, 110, 497), 'ORT[loc]': -58, 'ISS[loc]': -52, 'Max Abs Err / ε': 6.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'SKIPPED', 'Type': 'Node', 'Name': '/encoder/layers.1/activation_fn/Relu:0', 'dtype': 'int8', 'Mismatches Above Tol': None, 'total': 0, 'rtol': 0.1, 'atol': -1, 'Comment': 'Skipping validation for /encoder/layers.1/activation_fn/Relu:0 dtype mismatch between ISS int8 and ORT float32'}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.1/fc2/MatMul_quant:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 11130, 'Mismatch %': 12.076822916666666, 'RMSE': 0.3479696815895827, 'PSNR (dB)': 57.29997549365264, 'Max Abs Err': 3.0, 'Max Abs Err Loc': (0, 263, 142), 'ORT[loc]': -4, 'ISS[loc]': -7, 'Max Abs Err / ε': 3.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.1/Add_1_output_0_DequantizeLinear:0', 'dtype': 'custom[qfp.25]32', 'total': 92160, 'Bitwise Mismatches': 19912, 'Mismatch %': 21.60590277777778, 'RMSE': 0.12227850704451271, 'PSNR (dB)': 49.325393395642635, 'Max Abs Err': 0.5039205551147461, 'Max Abs Err Loc': (0, 24, 41), 'ORT[loc]': 11.338197, 'ISS[loc]': 10.834276, 'Max Abs Err / ε': 16908768.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.1/final_layer_norm/Add_1:0', 'dtype': 'custom[qfp.28]32', 'total': 92160, 'Bitwise Mismatches': 92122, 'Mismatch %': 99.95876736111111, 'RMSE': 0.07631895860524494, 'PSNR (dB)': 41.854628723618525, 'Max Abs Err': 0.4695713073015213, 'Max Abs Err Loc': (0, 112, 42), 'ORT[loc]': -0.07682519, 'ISS[loc]': -0.5463965, 'Max Abs Err / ε': 126049588.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.1/final_layer_norm/Add_1_output_0_QuantizeLinear:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 26281, 'Mismatch %': 28.516710069444443, 'RMSE': 1.425767337260887, 'PSNR (dB)': 45.049830382902115, 'Max Abs Err': 9.0, 'Max Abs Err Loc': (0, 112, 42), 'ORT[loc]': -1, 'ISS[loc]': -10, 'Max Abs Err / ε': 9.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.2/self_attn/Add_quant:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 26259, 'Mismatch %': 28.492838541666668, 'RMSE': 1.4225330436993955, 'PSNR (dB)': 45.06955634242117, 'Max Abs Err': 9.0, 'Max Abs Err Loc': (0, 112, 42), 'ORT[loc]': 2, 'ISS[loc]': -7, 'Max Abs Err / ε': 9.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.2/self_attn/MatMul_1_quant:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 89636, 'Mismatch %': 97.26128472222223, 'RMSE': 18.195100609722296, 'PSNR (dB)': 22.931714381515782, 'Max Abs Err': 129.0, 'Max Abs Err Loc': (1, 264, 17), 'ORT[loc]': -63, 'ISS[loc]': 66, 'Max Abs Err / ε': 129.0, 'Mismatches Above Tol': 53325, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.2/Add_output_0_DequantizeLinear:0', 'dtype': 'custom[qfp.28]32', 'total': 92160, 'Bitwise Mismatches': 48235, 'Mismatch %': 52.33832465277778, 'RMSE': 0.0854691979618338, 'PSNR (dB)': 41.10323504305215, 'Max Abs Err': 0.557727575302124, 'Max Abs Err Loc': (0, 233, 108), 'ORT[loc]': -1.2270007, 'ISS[loc]': -0.66927314, 'Max Abs Err / ε': 149713856.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.2/self_attn_layer_norm/Add_1:0', 'dtype': 'custom[qfp.27]32', 'total': 92160, 'Bitwise Mismatches': 92160, 'Mismatch %': 100.0, 'RMSE': 0.11086678463655064, 'PSNR (dB)': 44.52346697027962, 'Max Abs Err': 0.8220604956150055, 'Max Abs Err Loc': (0, 112, 42), 'ORT[loc]': 0.45826334, 'ISS[loc]': -0.36379716, 'Max Abs Err / ε': 110335092.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.2/self_attn_layer_norm/Add_1_output_0_QuantizeLinear:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 43618, 'Mismatch %': 47.32855902777778, 'RMSE': 1.2846203976164408, 'PSNR (dB)': 45.955307336669165, 'Max Abs Err': 9.0, 'Max Abs Err Loc': (0, 112, 42), 'ORT[loc]': 5, 'ISS[loc]': -4, 'Max Abs Err / ε': 9.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.2/fc1/MatMul_quant:0', 'dtype': 'int8', 'total': 737280, 'Bitwise Mismatches': 578760, 'Mismatch %': 78.49934895833333, 'RMSE': 1.909351487057444, 'PSNR (dB)': 42.51308593301847, 'Max Abs Err': 12.0, 'Max Abs Err Loc': (0, 123, 1428), 'ORT[loc]': -23, 'ISS[loc]': -11, 'Max Abs Err / ε': 12.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'SKIPPED', 'Type': 'Node', 'Name': '/encoder/layers.2/activation_fn/Relu:0', 'dtype': 'int8', 'Mismatches Above Tol': None, 'total': 0, 'rtol': 0.1, 'atol': -1, 'Comment': 'Skipping validation for /encoder/layers.2/activation_fn/Relu:0 dtype mismatch between ISS int8 and ORT float32'}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.2/fc2/MatMul_quant:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 20876, 'Mismatch %': 22.65190972222222, 'RMSE': 0.5057181705807648, 'PSNR (dB)': 54.052632444227235, 'Max Abs Err': 8.0, 'Max Abs Err Loc': (0, 67, 142), 'ORT[loc]': -17, 'ISS[loc]': -9, 'Max Abs Err / ε': 8.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.2/Add_1_output_0_DequantizeLinear:0', 'dtype': 'custom[qfp.25]32', 'total': 92160, 'Bitwise Mismatches': 28084, 'Mismatch %': 30.47309027777778, 'RMSE': 0.28972638342300394, 'PSNR (dB)': 47.42495589057783, 'Max Abs Err': 3.8374037742614746, 'Max Abs Err Loc': (0, 67, 142), 'ORT[loc]': -8.154483, 'ISS[loc]': -4.317079, 'Max Abs Err / ε': 128761904.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.2/final_layer_norm/Add_1:0', 'dtype': 'custom[qfp.28]32', 'total': 92160, 'Bitwise Mismatches': 92122, 'Mismatch %': 99.95876736111111, 'RMSE': 0.12592781711651668, 'PSNR (dB)': 36.63740438762495, 'Max Abs Err': 1.0465166121721268, 'Max Abs Err Loc': (0, 112, 42), 'ORT[loc]': 0.22258396, 'ISS[loc]': -0.82393265, 'Max Abs Err / ε': 280922164.0, 'Mismatches Above Tol': 135, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.2/final_layer_norm/Add_1_output_0_QuantizeLinear:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 45789, 'Mismatch %': 49.684244791666664, 'RMSE': 3.427870033818377, 'PSNR (dB)': 37.43031665909064, 'Max Abs Err': 28.0, 'Max Abs Err Loc': (0, 112, 42), 'ORT[loc]': 6, 'ISS[loc]': -22, 'Max Abs Err / ε': 28.0, 'Mismatches Above Tol': 252, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.3/self_attn/Add_quant:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 44556, 'Mismatch %': 48.346354166666664, 'RMSE': 3.2031681907657195, 'PSNR (dB)': 38.01920874796131, 'Max Abs Err': 26.0, 'Max Abs Err Loc': (0, 112, 42), 'ORT[loc]': 10, 'ISS[loc]': -16, 'Max Abs Err / ε': 26.0, 'Mismatches Above Tol': 227, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.3/self_attn/MatMul_1_quant:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 70271, 'Mismatch %': 76.24891493055556, 'RMSE': 14.977388194143048, 'PSNR (dB)': 24.62208188315458, 'Max Abs Err': 89.0, 'Max Abs Err Loc': (5, 276, 22), 'ORT[loc]': 23, 'ISS[loc]': -66, 'Max Abs Err / ε': 89.0, 'Mismatches Above Tol': 46013, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.3/Add_output_0_DequantizeLinear:0', 'dtype': 'custom[qfp.28]32', 'total': 92160, 'Bitwise Mismatches': 67436, 'Mismatch %': 73.17274305555556, 'RMSE': 0.13323231085659043, 'PSNR (dB)': 36.31915750741629, 'Max Abs Err': 1.0001173168420792, 'Max Abs Err Loc': (0, 112, 42), 'ORT[loc]': 0.24002816, 'ISS[loc]': -0.76008916, 'Max Abs Err / ε': 268466948.0, 'Mismatches Above Tol': 172, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.3/self_attn_layer_norm/Add_1:0', 'dtype': 'custom[qfp.27]32', 'total': 92160, 'Bitwise Mismatches': 92160, 'Mismatch %': 100.0, 'RMSE': 0.17899663998389395, 'PSNR (dB)': 41.033232169250354, 'Max Abs Err': 1.807301640510559, 'Max Abs Err Loc': (0, 112, 42), 'ORT[loc]': 1.0772173, 'ISS[loc]': -0.7300843, 'Max Abs Err / ε': 242571920.0, 'Mismatches Above Tol': 16, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.3/self_attn_layer_norm/Add_1_output_0_QuantizeLinear:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 61278, 'Mismatch %': 66.49088541666667, 'RMSE': 2.8347269856783974, 'PSNR (dB)': 39.08057884726023, 'Max Abs Err': 28.0, 'Max Abs Err Loc': (0, 112, 42), 'ORT[loc]': 17, 'ISS[loc]': -11, 'Max Abs Err / ε': 28.0, 'Mismatches Above Tol': 96, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.3/fc1/MatMul_quant:0', 'dtype': 'int8', 'total': 737280, 'Bitwise Mismatches': 634862, 'Mismatch %': 86.10866970486111, 'RMSE': 3.1322723540309183, 'PSNR (dB)': 38.21361326184066, 'Max Abs Err': 24.0, 'Max Abs Err Loc': (0, 67, 1226), 'ORT[loc]': 27, 'ISS[loc]': 3, 'Max Abs Err / ε': 24.0, 'Mismatches Above Tol': 107, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'SKIPPED', 'Type': 'Node', 'Name': '/encoder/layers.3/activation_fn/Relu:0', 'dtype': 'int8', 'Mismatches Above Tol': None, 'total': 0, 'rtol': 0.1, 'atol': -1, 'Comment': 'Skipping validation for /encoder/layers.3/activation_fn/Relu:0 dtype mismatch between ISS int8 and ORT float32'}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.3/fc2/MatMul_quant:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 24535, 'Mismatch %': 26.622178819444443, 'RMSE': 0.6814602122158955, 'PSNR (dB)': 51.461993525427204, 'Max Abs Err': 27.0, 'Max Abs Err Loc': (0, 223, 93), 'ORT[loc]': -76, 'ISS[loc]': -49, 'Max Abs Err / ε': 27.0, 'Mismatches Above Tol': 5, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.3/Add_1_output_0_DequantizeLinear:0', 'dtype': 'custom[qfp.24]32', 'total': 92160, 'Bitwise Mismatches': 33772, 'Mismatch %': 36.64496527777778, 'RMSE': 0.44463061974788654, 'PSNR (dB)': 49.53667947342991, 'Max Abs Err': 13.84945297241211, 'Max Abs Err Loc': (0, 223, 93), 'ORT[loc]': -47.318962, 'ISS[loc]': -33.46951, 'Max Abs Err / ε': 232355264.0, 'Mismatches Above Tol': 3, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.3/final_layer_norm/Add_1:0', 'dtype': 'custom[qfp.28]32', 'total': 92160, 'Bitwise Mismatches': 92160, 'Mismatch %': 100.0, 'RMSE': 0.16107349502844664, 'PSNR (dB)': 36.47780422244566, 'Max Abs Err': 1.3410066366195679, 'Max Abs Err Loc': (0, 67, 231), 'ORT[loc]': -0.8965663, 'ISS[loc]': -2.237573, 'Max Abs Err / ε': 359973728.0, 'Mismatches Above Tol': 468, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.3/final_layer_norm/Add_1_output_0_QuantizeLinear:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 49536, 'Mismatch %': 53.75, 'RMSE': 3.789575602429351, 'PSNR (dB)': 36.55899209450705, 'Max Abs Err': 31.0, 'Max Abs Err Loc': (0, 67, 231), 'ORT[loc]': -21, 'ISS[loc]': -52, 'Max Abs Err / ε': 31.0, 'Mismatches Above Tol': 572, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.4/self_attn/Add_quant:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 48235, 'Mismatch %': 52.33832465277778, 'RMSE': 3.5173629963250406, 'PSNR (dB)': 37.206459801248776, 'Max Abs Err': 28.0, 'Max Abs Err Loc': (0, 67, 231), 'ORT[loc]': -6, 'ISS[loc]': -34, 'Max Abs Err / ε': 28.0, 'Mismatches Above Tol': 388, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.4/self_attn/MatMul_1_quant:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 89062, 'Mismatch %': 96.63845486111111, 'RMSE': 14.46084985004051, 'PSNR (dB)': 24.926927273246843, 'Max Abs Err': 105.0, 'Max Abs Err Loc': (6, 305, 6), 'ORT[loc]': -4, 'ISS[loc]': 101, 'Max Abs Err / ε': 105.0, 'Mismatches Above Tol': 36940, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.4/Add_output_0_DequantizeLinear:0', 'dtype': 'custom[qfp.28]32', 'total': 92160, 'Bitwise Mismatches': 71665, 'Mismatch %': 77.76150173611111, 'RMSE': 0.16858437013044208, 'PSNR (dB)': 36.36182177739782, 'Max Abs Err': 1.4521845169365406, 'Max Abs Err Loc': (0, 46, 147), 'ORT[loc]': -0.04400559, 'ISS[loc]': 1.4081789, 'Max Abs Err / ε': 389817813.0, 'Mismatches Above Tol': 516, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.4/self_attn_layer_norm/Add_1:0', 'dtype': 'custom[qfp.26]32', 'total': 92160, 'Bitwise Mismatches': 92158, 'Mismatch %': 99.99782986111111, 'RMSE': 0.21212153873243678, 'PSNR (dB)': 43.46497657098254, 'Max Abs Err': 2.216306686401367, 'Max Abs Err Loc': (0, 46, 47), 'ORT[loc]': -10.862047, 'ISS[loc]': -8.6457405, 'Max Abs Err / ε': 148733824.0, 'Mismatches Above Tol': 2, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.4/self_attn_layer_norm/Add_1_output_0_QuantizeLinear:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 54554, 'Mismatch %': 59.19487847222222, 'RMSE': 1.9687692900289537, 'PSNR (dB)': 42.24690708112088, 'Max Abs Err': 20.0, 'Max Abs Err Loc': (0, 46, 47), 'ORT[loc]': -99, 'ISS[loc]': -79, 'Max Abs Err / ε': 20.0, 'Mismatches Above Tol': 10, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.4/fc1/MatMul_quant:0', 'dtype': 'int8', 'total': 737280, 'Bitwise Mismatches': 629148, 'Mismatch %': 85.33365885416667, 'RMSE': 3.4374863872836525, 'PSNR (dB)': 37.405983868650466, 'Max Abs Err': 45.0, 'Max Abs Err Loc': (0, 68, 1442), 'ORT[loc]': -52, 'ISS[loc]': -7, 'Max Abs Err / ε': 45.0, 'Mismatches Above Tol': 588, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'SKIPPED', 'Type': 'Node', 'Name': '/encoder/layers.4/activation_fn/Relu:0', 'dtype': 'int8', 'Mismatches Above Tol': None, 'total': 0, 'rtol': 0.1, 'atol': -1, 'Comment': 'Skipping validation for /encoder/layers.4/activation_fn/Relu:0 dtype mismatch between ISS int8 and ORT float32'}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.4/fc2/MatMul_quant:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 45716, 'Mismatch %': 49.60503472222222, 'RMSE': 1.2670538059696685, 'PSNR (dB)': 46.07490245324303, 'Max Abs Err': 47.0, 'Max Abs Err Loc': (0, 183, 114), 'ORT[loc]': 106, 'ISS[loc]': 59, 'Max Abs Err / ε': 47.0, 'Mismatches Above Tol': 39, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.4/Add_1_output_0_DequantizeLinear:0', 'dtype': 'custom[qfp.26]32', 'total': 92160, 'Bitwise Mismatches': 52829, 'Mismatch %': 57.32313368055556, 'RMSE': 0.28293274816196795, 'PSNR (dB)': 45.45892395892494, 'Max Abs Err': 5.798240661621094, 'Max Abs Err Loc': (0, 183, 114), 'ORT[loc]': 25.984707, 'ISS[loc]': 20.186466, 'Max Abs Err / ε': 389113344.0, 'Mismatches Above Tol': 8, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.4/final_layer_norm/Add_1:0', 'dtype': 'custom[qfp.28]32', 'total': 92160, 'Bitwise Mismatches': 92160, 'Mismatch %': 100.0, 'RMSE': 0.12868189798183288, 'PSNR (dB)': 38.21064148629445, 'Max Abs Err': 1.2515288889408112, 'Max Abs Err Loc': (0, 67, 231), 'ORT[loc]': -0.32758018, 'ISS[loc]': -1.5791091, 'Max Abs Err / ε': 335954728.0, 'Mismatches Above Tol': 144, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.4/final_layer_norm/Add_1_output_0_QuantizeLinear:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 54992, 'Mismatch %': 59.670138888888886, 'RMSE': 2.9880385817409088, 'PSNR (dB)': 38.623079592337966, 'Max Abs Err': 28.0, 'Max Abs Err Loc': (0, 67, 231), 'ORT[loc]': -8, 'ISS[loc]': -36, 'Max Abs Err / ε': 28.0, 'Mismatches Above Tol': 172, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.5/self_attn/Add_quant:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 53908, 'Mismatch %': 58.493923611111114, 'RMSE': 2.724793729451012, 'PSNR (dB)': 39.424130985063414, 'Max Abs Err': 25.0, 'Max Abs Err Loc': (0, 67, 231), 'ORT[loc]': 6, 'ISS[loc]': -19, 'Max Abs Err / ε': 25.0, 'Mismatches Above Tol': 125, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.5/self_attn/MatMul_1_quant:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 80215, 'Mismatch %': 87.03884548611111, 'RMSE': 15.20522478219521, 'PSNR (dB)': 24.490946712702623, 'Max Abs Err': 100.0, 'Max Abs Err Loc': (3, 80, 0), 'ORT[loc]': -41, 'ISS[loc]': 59, 'Max Abs Err / ε': 100.0, 'Mismatches Above Tol': 46951, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.5/Add_output_0_DequantizeLinear:0', 'dtype': 'custom[qfp.28]32', 'total': 92160, 'Bitwise Mismatches': 67245, 'Mismatch %': 72.96549479166667, 'RMSE': 0.13206080326887668, 'PSNR (dB)': 37.87786218950436, 'Max Abs Err': 1.2498381584882736, 'Max Abs Err Loc': (0, 67, 231), 'ORT[loc]': -0.21548934, 'ISS[loc]': -1.4653275, 'Max Abs Err / ε': 335500876.0, 'Mismatches Above Tol': 181, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.5/self_attn_layer_norm/Add_1:0', 'dtype': 'custom[qfp.27]32', 'total': 92160, 'Bitwise Mismatches': 92160, 'Mismatch %': 100.0, 'RMSE': 0.19584225776468145, 'PSNR (dB)': 41.802185346770116, 'Max Abs Err': 2.4902768433094025, 'Max Abs Err Loc': (0, 232, 217), 'ORT[loc]': 0.4670541, 'ISS[loc]': 2.957331, 'Max Abs Err / ε': 334239300.0, 'Mismatches Above Tol': 37, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.5/self_attn_layer_norm/Add_1_output_0_QuantizeLinear:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 57849, 'Mismatch %': 62.770182291666664, 'RMSE': 2.024333845333104, 'PSNR (dB)': 42.005160883763054, 'Max Abs Err': 25.0, 'Max Abs Err Loc': (0, 67, 231), 'ORT[loc]': -6, 'ISS[loc]': -31, 'Max Abs Err / ε': 25.0, 'Mismatches Above Tol': 49, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.5/fc1/MatMul_quant:0', 'dtype': 'int8', 'total': 737280, 'Bitwise Mismatches': 612193, 'Mismatch %': 83.03398980034723, 'RMSE': 2.850781618727329, 'PSNR (dB)': 39.031524610888425, 'Max Abs Err': 25.0, 'Max Abs Err Loc': (0, 46, 967), 'ORT[loc]': -15, 'ISS[loc]': 10, 'Max Abs Err / ε': 25.0, 'Mismatches Above Tol': 172, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'SKIPPED', 'Type': 'Node', 'Name': '/encoder/layers.5/activation_fn/Relu:0', 'dtype': 'int8', 'Mismatches Above Tol': None, 'total': 0, 'rtol': 0.1, 'atol': -1, 'Comment': 'Skipping validation for /encoder/layers.5/activation_fn/Relu:0 dtype mismatch between ISS int8 and ORT float32'}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.5/fc2/MatMul_quant:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 34856, 'Mismatch %': 37.82118055555556, 'RMSE': 0.7968375535863979, 'PSNR (dB)': 50.10340773964771, 'Max Abs Err': 14.0, 'Max Abs Err Loc': (0, 67, 142), 'ORT[loc]': -25, 'ISS[loc]': -11, 'Max Abs Err / ε': 14.0, 'Mismatches Above Tol': 95, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.5/Add_1_output_0_DequantizeLinear:0', 'dtype': 'custom[qfp.26]32', 'total': 92160, 'Bitwise Mismatches': 53900, 'Mismatch %': 58.48524305555556, 'RMSE': 0.2150491332409611, 'PSNR (dB)': 42.03554255919762, 'Max Abs Err': 2.6166504621505737, 'Max Abs Err Loc': (0, 232, 217), 'ORT[loc]': 0.29073894, 'ISS[loc]': 2.9073894, 'Max Abs Err / ε': 175600440.0, 'Mismatches Above Tol': 21, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.5/final_layer_norm/Add_1:0', 'dtype': 'custom[qfp.28]32', 'total': 92160, 'Bitwise Mismatches': 92160, 'Mismatch %': 100.0, 'RMSE': 0.11628144746683536, 'PSNR (dB)': 37.197754785928325, 'Max Abs Err': 1.3430075943470001, 'Max Abs Err Loc': (0, 232, 217), 'ORT[loc]': 0.25954124, 'ISS[loc]': 1.6025488, 'Max Abs Err / ε': 360510856.0, 'Mismatches Above Tol': 709, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Output', 'Name': 'encoder_memory', 'dtype': 'custom[qfp.28]32', 'total': 92160, 'Bitwise Mismatches': 92160, 'Mismatch %': 100.0, 'RMSE': 0.11628144746683536, 'PSNR (dB)': 37.197754785928325, 'Max Abs Err': 1.3430075943470001, 'Max Abs Err Loc': (0, 232, 217), 'ORT[loc]': 0.25954124, 'ISS[loc]': 1.6025488, 'Max Abs Err / ε': 360510856.0, 'Mismatches Above Tol': 709, 'rtol': 0.1, 'atol': -1, 'Comment': ''}]
5. Single-Layer Extraction (optional)
Extract, quantize, compile, and validate a single encoder layer in isolation. Change LAYER to target any of the 6 layers (0–5). This reuses the hw_config and imports from the cells above — run sections 1–3 first.
| Layer | Input edges | Output edge |
|---|---|---|
| 0 | src, pos_embed | /encoder/layers.0/final_layer_norm/Add_1_output_0 |
| 1–4 | prev layer output, pos_embed | /encoder/layers.{N}/final_layer_norm/Add_1_output_0 |
| 5 | prev layer output, pos_embed | encoder_memory (graph output) |
LAYER = 0
if LAYER == 0:
in_edges = ["src", "pos_embed"]
else:
in_edges = [f"/encoder/layers.{LAYER - 1}/final_layer_norm/Add_1_output_0", "pos_embed"]
out_edges = (
["encoder_memory"]
if LAYER == 5
else [f"/encoder/layers.{LAYER}/final_layer_norm/Add_1_output_0"]
)
layer_onnx = f"{OUTPUT_DIR}/encoder_layer_{LAYER}.onnx"
onnx.save(
cut_onnx(onnx.load(ENCODER_ONNX), cut_before=in_edges, cut_after=out_edges),
layer_onnx,
)
layer_q = quadric_quantize(layer_onnx, num_images=1, synthetic_input=True, output_folder=OUTPUT_DIR)
layer_job = ChimeraJob(
layer_q.qmodel_path,
hw_config=hw_config,
trange_file=layer_q.tranges_path,
target_lang="ASM",
validate_iss=True,
)
layer_job.compile()
print(f"Layer {LAYER} compiled!")
print(layer_job)
2026-06-19 03:59 - INFO - epu - quantize - Generating synthetic data
2026-06-19 03:59 - INFO - epu - quantize - Optimized model to opset
2026-06-19 03:59 - INFO - epu - quantize - Saved optimized model to encoder_layer_0_float32_opt.onnx
2026-06-19 03:59 - INFO - epu - quantize - Input shapes: [1, 360, 256]. Input names: src
2026-06-19 03:59 - INFO - epu - quantize - Input shapes: [1, 360, 256]. Input names: pos_embed
2026-06-19 03:59 - INFO - epu - quantize - Output shapes: [[1, 360, 256]]. Output names: ['/encoder/layers.0/final_layer_norm/Add_1_output_0']
2026-06-19 03:59 - INFO - epu - quantize - Quantization started...
WARNING:root:Please use QuantFormat.QDQ for activation type QInt8 and weight type QInt8. Or it will lead to bad performance on x64.
2026-06-19 03:59 - INFO - epu - quantize - Quantization done succesfully!
2026-06-19 03:59 - INFO - epu - quantize - ONNX full precision model size: 5.03 MB
2026-06-19 03:59 - INFO - epu - quantize - ONNX quantized model size: 1.29 MB
2026-06-19 03:59 - INFO - epu - quantize - Saved quantized model to onnx/encoder_layer_0_opt_sym_int8_q.onnx
2026-06-19 03:59 - INFO - epu - quantize - Saved shape inferenced model to onnx/encoder_layer_0_opt_sym_int8_q.onnx
2026-06-19 03:59 - INFO - epu - quantize - Checking for remaining FLOAT/FLOAT16 types.
2026-06-19 03:59 - INFO - epu - quantize - Model still has FLOAT/FLOAT16 types. Creating ranges for floating point tensors using calibration data
2026-06-19 03:59 - INFO - epu - quantize - Saved tensor ranges to onnx/encoder_layer_0_opt_sym_int8_q.onnx.tranges
2026-06-19 03:59 - INFO - epu - chimera_job - START==================================onnx_ingest
2026-06-19 03:59 - INFO - epu - chimera_job - Numerical ranges provided
2026-06-19 03:59 - INFO - epu - codegen - START===============================optimize_relay
2026-06-19 03:59 - INFO - epu - codegen - START====================quantize_to_cpu_runnable_fx
2026-06-19 03:59 - INFO - epu - fx -
Source name Op Output 0 Range Output 0 Frac Bits
------------------------------------------------------------ ---------------------- --------------------------- --------------------
/encoder/layers.0/self_attn/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-1.05423f, 1.08851f] 30
/encoder/layers.0/self_attn/Softmax nn.softmax [0.000948724f, 0.00762514f] 30
/encoder/layers.0/Add_output_0_DequantizeLinear contrib.epu.dequantize [-1.48326f, 1.44618f] 30
/encoder/layers.0/self_attn_layer_norm/Add_1 nn.layer_norm [-7.76233f, 7.38209f] 28
/encoder/layers.0/Add_1_output_0_DequantizeLinear contrib.epu.dequantize [-15.0478f, 13.844f] 27
/encoder/layers.0/final_layer_norm/Add_1 nn.layer_norm [-6.51991f, 5.00955f] -
2026-06-19 03:59 - INFO - epu - codegen - START====================build_cpu_runnable_fx_relay
2026-06-19 03:59 - INFO - epu - codegen - START=======================quantize_to_chimera_fx
2026-06-19 03:59 - INFO - epu - codegen - START=================================relay_to_tir
2026-06-19 03:59 - INFO - epu - codegen - START===========================relay_to_epu_relay
2026-06-19 03:59 - INFO - epu - codegen - START==============================adapt_and_order
2026-06-19 03:59 - INFO - epu - mac_counter -
2026-06-19 03:59 - INFO - epu - mac_counter - ============================================================
2026-06-19 03:59 - INFO - epu - mac_counter - MAC Operation Count Summary
2026-06-19 03:59 - INFO - epu - mac_counter - ============================================================
2026-06-19 03:59 - INFO - epu - mac_counter - conv2d: 47,185,920 ops (23,592,960 MACs) - /encoder/layers.0/self_attn/out_proj/MatMul_quant
2026-06-19 03:59 - INFO - epu - mac_counter - ------------------------------------------------------------
2026-06-19 03:59 - INFO - epu - mac_counter - Total: 47,185,920 ops (23,592,960 MACs)
2026-06-19 03:59 - INFO - epu - mac_counter - ============================================================
2026-06-19 03:59 - INFO - epu - mac_counter -
2026-06-19 03:59 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-06-19 03:59 - INFO - epu - codegen - START=============================plan_lrm_virtual
2026-06-19 03:59 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-06-19 03:59 - INFO - epu - codegen - START===============================lrm_alloc_loop
2026-06-19 03:59 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-06-19 03:59 - INFO - epu - codegen - START================================lrm_splitting
2026-06-19 03:59 - INFO - epu - codegen - START==============================ext_split_relay
2026-06-19 03:59 - INFO - epu - codegen - START====================================build_tir
2026-06-19 04:00 - INFO - epu - chimera_job - Compilation of encoder_layer_0_opt_sym_int8_q_QC_U_1d56_8MB_4kB_64GBps_64GBps_16_OFF_x1_x1 successful
Layer 0 compiled!
╒═════════════════════╤═════════════════════════════════════════════════════════════════════════════╕
│ Module Name │ encoder_layer_0_opt_sym_int8_q_QC_U_1d56_8MB_4kB_64GBps_64GBps_16_OFF_x1_x1 │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────────┤
│ ONNX File │ onnx/encoder_layer_0_opt_sym_int8_q.onnx │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────────┤
│ Product Target │ QC-U │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────────┤
│ Number of Cores │ 1 │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────────┤
│ ISS Clock Frequency │ 1.560 │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────────┤
│ L2M Size │ 8MB │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────────┤
│ LRM Size │ 4kB │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────────┤
│ External Read BW │ 64GBps │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────────┤
│ External Write BW │ 64GBps │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────────┤
│ MACS per PE │ 16 │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────────┤
│ Max L2M │ 6.188MB │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────────┤
│ Max LRM │ 0.250kB │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────────┤
│ Max Temp Ext Bytes │ 0.000MB │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────────┤
│ Network GMACs │ 0.538 │
╘═════════════════════╧═════════════════════════════════════════════════════════════════════════════╛
╒════╤════════╤═══════════════════════════════════════════════════╤═══════════════╤══════════════════════════╤═══════╕
│ │ Type │ Name │ shape │ type │ mse │
╞════╪════════╪═══════════════════════════════════════════════════╪═══════════════╪══════════════════════════╪═══════╡
│ 0 │ Input │ src │ [1, 360, 256] │ tensor[FixedPoint32<29>] │ n/a │
├────┼────────┼───────────────────────────────────────────────────┼───────────────┼──────────────────────────┼───────┤
│ 1 │ Input │ pos_embed │ [1, 360, 256] │ tensor[FixedPoint32<30>] │ n/a │
├────┼────────┼───────────────────────────────────────────────────┼───────────────┼──────────────────────────┼───────┤
│ 2 │ Output │ /encoder/layers.0/final_layer_norm/Add_1_output_0 │ [1, 360, 256] │ tensor[FixedPoint32<28>] │ n/a │
╘════╧════════╧═══════════════════════════════════════════════════╧═══════════════╧══════════════════════════╧═══════╛
Validate the extracted layer against ORT:
layer_validation = layer_job.validate_ort_iss()
print(layer_validation)
2026-06-19 04:00 - INFO - epu - iss_testing - Found tranges for input: <tvm.contrib.epu.interval.Interval object at 0x7a4da82dace0>
2026-06-19 04:00 - INFO - epu - iss_testing - Found tranges for input: <tvm.contrib.epu.interval.Interval object at 0x7a4da82d85e0>
2026-06-19 04:00 - INFO - epu - iss_testing - Found tranges for input: <tvm.contrib.epu.interval.Interval object at 0x7a4da82db190>
2026-06-19 04:00 - INFO - epu - iss_testing - Found tranges for input: <tvm.contrib.epu.interval.Interval object at 0x7a4da82db190>
2026-06-19 04:00 - INFO - epu - iss_testing - Started Executing Onnxruntime...
2026-06-19 04:00 - INFO - epu - iss_testing - Done 0:00:00.067388
2026-06-19 04:00 - INFO - epu - iss_testing - Found tranges for input: <tvm.contrib.epu.interval.Interval object at 0x7a4da82db190>
2026-06-19 04:00 - INFO - epu - iss_testing - Found tranges for input: <tvm.contrib.epu.interval.Interval object at 0x7a4da82db190>
FILM 14/14: 100%|███████████████████████████████████████████████████| 14/14 [00:20<00:00, 1.50s/it]
2026-06-19 04:00 - WARNING - epu - iss_testing - Node was skipped due to being multi-output: /encoder/layers.0/self_attn/MatMul_1_quant
2026-06-19 04:00 - INFO - epu - iss_testing -
======================================================================
ISS validation results (quantized model, rtol=0.1, atol=-1)
======================================================================
pos_embed_QuantizeLinear:0 (PASS, Node, int8)
Bitwise Mismatches 2 / 92160 (0.0022%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0.00465847
PSNR 94.77 dB
Max Abs Err 1
Max Abs Err / ε 1
Max Err Loc (0, 315, 122)
ORT @ max err 60
ISS @ max err 59
src_QuantizeLinear:0 (PASS, Node, int8)
Bitwise Mismatches 1 / 92160 (0.0011%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0.00329404
PSNR 97.78 dB
Max Abs Err 1
Max Abs Err / ε 1
Max Err Loc (0, 249, 161)
ORT @ max err 18
ISS @ max err 19
/encoder/layers.0/self_attn/Add_quant:0 (PASS, Node, int8)
Bitwise Mismatches 2 / 92160 (0.0022%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0.00465847
PSNR 94.77 dB
Max Abs Err 1
Max Abs Err / ε 1
Max Err Loc (0, 315, 122)
ORT @ max err 68
ISS @ max err 67
/encoder/layers.0/self_attn/MatMul_1_quant:0 (FAIL, Node, int8)
Bitwise Mismatches 36878 / 92160 (40.0152%)
Mismatches > Tol 32909 / 92160 (35.7086%)
Thresholds rtol=0.1, atol=-1
RMSE 86.9583
PSNR 9.34 dB
Max Abs Err 255
Max Abs Err / ε 255
Max Err Loc (3, 22, 16)
ORT @ max err 127
ISS @ max err -128
/encoder/layers.0/Add_output_0_DequantizeLinear:0 (PASS, Node, custom[qfp.30]32)
Bitwise Mismatches 3512 / 92160 (3.8108%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0.00542457
PSNR 50.44 dB
Max Abs Err 0.123605
Max Abs Err / ε 1.3272e+08
Max Err Loc (0, 3, 248)
ORT @ max err 0.7292718
ISS @ max err 0.6056664
/encoder/layers.0/self_attn_layer_norm/Add_1:0 (PASS, Node, custom[qfp.28]32)
Bitwise Mismatches 92157 / 92160 (99.9967%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0.0171058
PSNR 54.85 dB
Max Abs Err 0.525744
Max Abs Err / ε 1.41128e+08
Max Err Loc (0, 120, 0)
ORT @ max err 3.9197855
ISS @ max err 3.3940415
/encoder/layers.0/self_attn_layer_norm/Add_1_output_0_QuantizeLinear:0 (PASS, Node, int8)
Bitwise Mismatches 3695 / 92160 (4.0093%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0.31048
PSNR 58.29 dB
Max Abs Err 8
Max Abs Err / ε 8
Max Err Loc (0, 120, 0)
ORT @ max err 62
ISS @ max err 54
/encoder/layers.0/fc1/MatMul_quant:0 (PASS, Node, int8)
Bitwise Mismatches 153741 / 737280 (20.8525%)
Mismatches > Tol 0 / 737280 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0.466082
PSNR 54.76 dB
Max Abs Err 3
Max Abs Err / ε 3
Max Err Loc (0, 14, 387)
ORT @ max err -7
ISS @ max err -4
[ERROR] /encoder/layers.0/activation_fn/Relu:0
Error Skipping validation for /encoder/layers.0/activation_fn/Relu:0 dtype mismatch between ISS int8 and ORT float32
/encoder/layers.0/fc2/MatMul_quant:0 (PASS, Node, int8)
Bitwise Mismatches 22731 / 92160 (24.6647%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0.51861
PSNR 53.83 dB
Max Abs Err 6
Max Abs Err / ε 6
Max Err Loc (0, 345, 41)
ORT @ max err 115
ISS @ max err 121
/encoder/layers.0/Add_1_output_0_DequantizeLinear:0 (PASS, Node, custom[qfp.27]32)
Bitwise Mismatches 18776 / 92160 (20.3733%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0.0587958
PSNR 54.32 dB
Max Abs Err 0.601913
Max Abs Err / ε 8.07875e+07
Max Err Loc (0, 345, 41)
ORT @ max err 11.556744
ISS @ max err 12.158657
/encoder/layers.0/final_layer_norm/Add_1:0 (PASS, Node, custom[qfp.28]32)
Bitwise Mismatches 92160 / 92160 (100.0000%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0.038141
PSNR 48.88 dB
Max Abs Err 0.350066
Max Abs Err / ε 9.397e+07
Max Err Loc (0, 216, 248)
ORT @ max err 0.37861335
ISS @ max err 0.028547648
/encoder/layers.0/final_layer_norm/Add_1_output_0 (PASS, Output, custom[qfp.28]32)
Bitwise Mismatches 92160 / 92160 (100.0000%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0.038141
PSNR 48.88 dB
Max Abs Err 0.350066
Max Abs Err / ε 9.397e+07
Max Err Loc (0, 216, 248)
ORT @ max err 0.37861335
ISS @ max err 0.028547648
13 entries
Differences detected between ort and iss
[{'S': 'PASS', 'Type': 'Node', 'Name': 'pos_embed_QuantizeLinear:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 2, 'Mismatch %': 0.002170138888888889, 'RMSE': 0.004658474953124562, 'PSNR (dB)': 94.76592831283065, 'Max Abs Err': 1.0, 'Max Abs Err Loc': (0, 315, 122), 'ORT[loc]': 60, 'ISS[loc]': 59, 'Max Abs Err / ε': 1.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Node', 'Name': 'src_QuantizeLinear:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 1, 'Mismatch %': 0.0010850694444444445, 'RMSE': 0.0032940392293420617, 'PSNR (dB)': 97.77622826947047, 'Max Abs Err': 1.0, 'Max Abs Err Loc': (0, 249, 161), 'ORT[loc]': 18, 'ISS[loc]': 19, 'Max Abs Err / ε': 1.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.0/self_attn/Add_quant:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 2, 'Mismatch %': 0.002170138888888889, 'RMSE': 0.004658474953124562, 'PSNR (dB)': 94.76592831283065, 'Max Abs Err': 1.0, 'Max Abs Err Loc': (0, 315, 122), 'ORT[loc]': 68, 'ISS[loc]': 67, 'Max Abs Err / ε': 1.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.0/self_attn/MatMul_1_quant:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 36878, 'Mismatch %': 40.01519097222222, 'RMSE': 86.95827350111742, 'PSNR (dB)': 9.344585437983971, 'Max Abs Err': 255.0, 'Max Abs Err Loc': (3, 22, 16), 'ORT[loc]': 127, 'ISS[loc]': -128, 'Max Abs Err / ε': 255.0, 'Mismatches Above Tol': 32909, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.0/Add_output_0_DequantizeLinear:0', 'dtype': 'custom[qfp.30]32', 'total': 92160, 'Bitwise Mismatches': 3512, 'Mismatch %': 3.810763888888889, 'RMSE': 0.005424570658722566, 'PSNR (dB)': 50.44049812707741, 'Max Abs Err': 0.12360543012619019, 'Max Abs Err Loc': (0, 3, 248), 'ORT[loc]': 0.7292718, 'ISS[loc]': 0.6056664, 'Max Abs Err / ε': 132720320.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.0/self_attn_layer_norm/Add_1:0', 'dtype': 'custom[qfp.28]32', 'total': 92160, 'Bitwise Mismatches': 92157, 'Mismatch %': 99.99674479166667, 'RMSE': 0.01710575855413618, 'PSNR (dB)': 54.846871288778075, 'Max Abs Err': 0.5257439613342285, 'Max Abs Err Loc': (0, 120, 0), 'ORT[loc]': 3.9197855, 'ISS[loc]': 3.3940415, 'Max Abs Err / ε': 141128320.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.0/self_attn_layer_norm/Add_1_output_0_QuantizeLinear:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 3695, 'Mismatch %': 4.009331597222222, 'RMSE': 0.31047957975436075, 'PSNR (dB)': 58.29014277070611, 'Max Abs Err': 8.0, 'Max Abs Err Loc': (0, 120, 0), 'ORT[loc]': 62, 'ISS[loc]': 54, 'Max Abs Err / ε': 8.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.0/fc1/MatMul_quant:0', 'dtype': 'int8', 'total': 737280, 'Bitwise Mismatches': 153741, 'Mismatch %': 20.852457682291668, 'RMSE': 0.46608181590208314, 'PSNR (dB)': 54.761560421833295, 'Max Abs Err': 3.0, 'Max Abs Err Loc': (0, 14, 387), 'ORT[loc]': -7, 'ISS[loc]': -4, 'Max Abs Err / ε': 3.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'SKIPPED', 'Type': 'Node', 'Name': '/encoder/layers.0/activation_fn/Relu:0', 'dtype': 'int8', 'Mismatches Above Tol': None, 'total': 0, 'rtol': 0.1, 'atol': -1, 'Comment': 'Skipping validation for /encoder/layers.0/activation_fn/Relu:0 dtype mismatch between ISS int8 and ORT float32'}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.0/fc2/MatMul_quant:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 22731, 'Mismatch %': 24.664713541666668, 'RMSE': 0.5186098371554906, 'PSNR (dB)': 53.833988601746796, 'Max Abs Err': 6.0, 'Max Abs Err Loc': (0, 345, 41), 'ORT[loc]': 115, 'ISS[loc]': 121, 'Max Abs Err / ε': 6.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.0/Add_1_output_0_DequantizeLinear:0', 'dtype': 'custom[qfp.27]32', 'total': 92160, 'Bitwise Mismatches': 18776, 'Mismatch %': 20.37326388888889, 'RMSE': 0.05879580693534268, 'PSNR (dB)': 54.321032032865844, 'Max Abs Err': 0.6019134521484375, 'Max Abs Err Loc': (0, 345, 41), 'ORT[loc]': 11.556744, 'ISS[loc]': 12.158657, 'Max Abs Err / ε': 80787456.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.0/final_layer_norm/Add_1:0', 'dtype': 'custom[qfp.28]32', 'total': 92160, 'Bitwise Mismatches': 92160, 'Mismatch %': 100.0, 'RMSE': 0.03814095206110301, 'PSNR (dB)': 48.88199623112094, 'Max Abs Err': 0.3500657044351101, 'Max Abs Err Loc': (0, 216, 248), 'ORT[loc]': 0.37861335, 'ISS[loc]': 0.028547648, 'Max Abs Err / ε': 93970047.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Output', 'Name': '/encoder/layers.0/final_layer_norm/Add_1_output_0', 'dtype': 'custom[qfp.28]32', 'total': 92160, 'Bitwise Mismatches': 92160, 'Mismatch %': 100.0, 'RMSE': 0.03814095206110301, 'PSNR (dB)': 48.88199623112094, 'Max Abs Err': 0.3500657044351101, 'Max Abs Err Loc': (0, 216, 248), 'ORT[loc]': 0.37861335, 'ISS[loc]': 0.028547648, 'Max Abs Err / ε': 93970047.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}]
Summary
| Model | DETR Transformer Encoder (6 layers, d_model=256, 8 heads, seq_len=360) |
| Target | QC-U, 8 MB OCM, 4 kB LRM, 16 MACs/PE, 1.56 GHz |
| Quantization | Symmetric INT8 (QOperator), Softmax and LayerNorm in float |
| Custom Ops | None — all ops compile natively |
Key takeaways
- The full 6-layer DETR encoder compiles natively on CGC with ~97.9% element match vs ORT.
- CGC maps multi-head attention to
nn::multiheadAttentionHead, tiling 360 tokens on an 18×20 PE grid. - Individual layers can be extracted and validated in isolation using the SDK's
cut_onnxhelper.
