NOTE: The Jupyter Notebook below is included in the Chimera SDK and can be run interactively by running the following CLI command:
$ quadric sdk notebook
From the Jupyter Notebook window in your browser, select the notebook named /quadric/sdk-cli/examples/models/detr/encoder/detr_encoder.ipynb.
DETR Transformer Encoder on Chimera GPNPU
This notebook compiles and validates the DETR transformer encoder on Quadric hardware. The encoder (6 identical layers of self-attention + FFN) compiles fully natively using CGC — no custom ops required.
Why does this compile natively?
CGC recognises the multi-head attention pattern and maps it to the SDK’s built-in nn::multiheadAttentionHead kernel, which tiles the 360-token sequence across an 18×20 PE grid. LayerNorm and the FFN (256→2048→256) are handled by standard nn::layer_norm and matrixMulMatrix kernels.
Model: DETR Transformer Encoder (ResNet-50 backbone variant, 6 layers, d_model=256, 8 heads, FFN dim=2048, seq_len=360)
1. Setup
import onnx
from tvm.contrib.epu.chimera_job.chimera_job import ChimeraJob
from tvm.contrib.epu.chimera_job.hw_config import HWConfig
from tvm.contrib.epu.chimera_job.quantize import quadric_quantize
from tvm.contrib.epu.onnx_util import cut_onnx
ENCODER_ONNX = "onnx/detr_3_transformer_encoder.onnx"
OUTPUT_DIR = "onnx"
2. Quantize
Quantize the float32 encoder to symmetric INT8 using the QOperator format. Softmax and LayerNorm are excluded from quantization (kept in float) since they require higher precision for numerical stability.
Note: This step uses synthetic (random) calibration data. The purpose of this notebook is to demonstrate that CGC can compile and run the DETR encoder natively and numerically match ORT — not to produce a production-quality quantized model. For deployment, replace
synthetic_input=Truewith a representative calibration dataset.
result = quadric_quantize(
ENCODER_ONNX,
num_images=1,
synthetic_input=True,
output_folder=OUTPUT_DIR,
)
quantized_model = result.qmodel_path
tranges_file = result.tranges_path
print(f"Quantized model: {quantized_model}")
print(f"Tensor ranges: {tranges_file}")
2026-07-18 12:09 - INFO - epu - quantize - Generating synthetic data
2026-07-18 12:09 - INFO - epu - quantize - Optimized model to opset
2026-07-18 12:09 - INFO - epu - quantize - Saved optimized model to detr_3_transformer_encoder_float32_opt.onnx
2026-07-18 12:09 - INFO - epu - quantize - Input shapes: [1, 360, 256]. Input names: src
2026-07-18 12:09 - INFO - epu - quantize - Input shapes: [1, 360, 256]. Input names: pos_embed
2026-07-18 12:09 - INFO - epu - quantize - Output shapes: [[1, 360, 256]]. Output names: ['encoder_memory']
2026-07-18 12:09 - DEBUG - epu - quantize - Full exclusion set for quantization: ['Softmax', 'Sigmoid', 'QuadricCustomOp']
2026-07-18 12:09 - DEBUG - epu - quantize - excl_nodes ['/encoder/layers.0/self_attn/Softmax', '/encoder/layers.1/self_attn/Softmax', '/encoder/layers.2/self_attn/Softmax', '/encoder/layers.3/self_attn/Softmax', '/encoder/layers.4/self_attn/Softmax', '/encoder/layers.5/self_attn/Softmax', '/encoder/layers.0/self_attn_layer_norm/ReduceMean', '/encoder/layers.0/self_attn_layer_norm/Sub', '/encoder/layers.0/self_attn_layer_norm/Constant_output_0', '/encoder/layers.0/self_attn_layer_norm/Pow', '/encoder/layers.0/self_attn_layer_norm/ReduceMean_1', '/encoder/layers.0/self_attn_layer_norm/Constant_1_output_0', '/encoder/layers.0/self_attn_layer_norm/Add', '/encoder/layers.0/self_attn_layer_norm/Sqrt', '/encoder/layers.0/self_attn_layer_norm/Div', 'encoder.layers.0.self_attn_layer_norm.weight', '/encoder/layers.0/self_attn_layer_norm/Mul', 'encoder.layers.0.self_attn_layer_norm.bias', '/encoder/layers.0/self_attn_layer_norm/Add_1', '/encoder/layers.0/final_layer_norm/ReduceMean', '/encoder/layers.0/final_layer_norm/Sub', '/encoder/layers.0/self_attn_layer_norm/Constant_output_0', '/encoder/layers.0/final_layer_norm/Pow', '/encoder/layers.0/final_layer_norm/ReduceMean_1', '/encoder/layers.0/self_attn_layer_norm/Constant_1_output_0', '/encoder/layers.0/final_layer_norm/Add', '/encoder/layers.0/final_layer_norm/Sqrt', '/encoder/layers.0/final_layer_norm/Div', 'encoder.layers.0.final_layer_norm.weight', '/encoder/layers.0/final_layer_norm/Mul', 'encoder.layers.0.final_layer_norm.bias', '/encoder/layers.0/final_layer_norm/Add_1', '/encoder/layers.1/self_attn_layer_norm/ReduceMean', '/encoder/layers.1/self_attn_layer_norm/Sub', '/encoder/layers.0/self_attn_layer_norm/Constant_output_0', '/encoder/layers.1/self_attn_layer_norm/Pow', '/encoder/layers.1/self_attn_layer_norm/ReduceMean_1', '/encoder/layers.0/self_attn_layer_norm/Constant_1_output_0', '/encoder/layers.1/self_attn_layer_norm/Add', '/encoder/layers.1/self_attn_layer_norm/Sqrt', '/encoder/layers.1/self_attn_layer_norm/Div', 'encoder.layers.1.self_attn_layer_norm.weight', '/encoder/layers.1/self_attn_layer_norm/Mul', 'encoder.layers.1.self_attn_layer_norm.bias', '/encoder/layers.1/self_attn_layer_norm/Add_1', '/encoder/layers.1/final_layer_norm/ReduceMean', '/encoder/layers.1/final_layer_norm/Sub', '/encoder/layers.0/self_attn_layer_norm/Constant_output_0', '/encoder/layers.1/final_layer_norm/Pow', '/encoder/layers.1/final_layer_norm/ReduceMean_1', '/encoder/layers.0/self_attn_layer_norm/Constant_1_output_0', '/encoder/layers.1/final_layer_norm/Add', '/encoder/layers.1/final_layer_norm/Sqrt', '/encoder/layers.1/final_layer_norm/Div', 'encoder.layers.1.final_layer_norm.weight', '/encoder/layers.1/final_layer_norm/Mul', 'encoder.layers.1.final_layer_norm.bias', '/encoder/layers.1/final_layer_norm/Add_1', '/encoder/layers.2/self_attn_layer_norm/ReduceMean', '/encoder/layers.2/self_attn_layer_norm/Sub', '/encoder/layers.0/self_attn_layer_norm/Constant_output_0', '/encoder/layers.2/self_attn_layer_norm/Pow', '/encoder/layers.2/self_attn_layer_norm/ReduceMean_1', '/encoder/layers.0/self_attn_layer_norm/Constant_1_output_0', '/encoder/layers.2/self_attn_layer_norm/Add', '/encoder/layers.2/self_attn_layer_norm/Sqrt', '/encoder/layers.2/self_attn_layer_norm/Div', 'encoder.layers.2.self_attn_layer_norm.weight', '/encoder/layers.2/self_attn_layer_norm/Mul', 'encoder.layers.2.self_attn_layer_norm.bias', '/encoder/layers.2/self_attn_layer_norm/Add_1', '/encoder/layers.2/final_layer_norm/ReduceMean', '/encoder/layers.2/final_layer_norm/Sub', '/encoder/layers.0/self_attn_layer_norm/Constant_output_0', '/encoder/layers.2/final_layer_norm/Pow', '/encoder/layers.2/final_layer_norm/ReduceMean_1', '/encoder/layers.0/self_attn_layer_norm/Constant_1_output_0', '/encoder/layers.2/final_layer_norm/Add', '/encoder/layers.2/final_layer_norm/Sqrt', '/encoder/layers.2/final_layer_norm/Div', 'encoder.layers.2.final_layer_norm.weight', '/encoder/layers.2/final_layer_norm/Mul', 'encoder.layers.2.final_layer_norm.bias', '/encoder/layers.2/final_layer_norm/Add_1', '/encoder/layers.3/self_attn_layer_norm/ReduceMean', '/encoder/layers.3/self_attn_layer_norm/Sub', '/encoder/layers.0/self_attn_layer_norm/Constant_output_0', '/encoder/layers.3/self_attn_layer_norm/Pow', '/encoder/layers.3/self_attn_layer_norm/ReduceMean_1', '/encoder/layers.0/self_attn_layer_norm/Constant_1_output_0', '/encoder/layers.3/self_attn_layer_norm/Add', '/encoder/layers.3/self_attn_layer_norm/Sqrt', '/encoder/layers.3/self_attn_layer_norm/Div', 'encoder.layers.3.self_attn_layer_norm.weight', '/encoder/layers.3/self_attn_layer_norm/Mul', 'encoder.layers.3.self_attn_layer_norm.bias', '/encoder/layers.3/self_attn_layer_norm/Add_1', '/encoder/layers.3/final_layer_norm/ReduceMean', '/encoder/layers.3/final_layer_norm/Sub', '/encoder/layers.0/self_attn_layer_norm/Constant_output_0', '/encoder/layers.3/final_layer_norm/Pow', '/encoder/layers.3/final_layer_norm/ReduceMean_1', '/encoder/layers.0/self_attn_layer_norm/Constant_1_output_0', '/encoder/layers.3/final_layer_norm/Add', '/encoder/layers.3/final_layer_norm/Sqrt', '/encoder/layers.3/final_layer_norm/Div', 'encoder.layers.3.final_layer_norm.weight', '/encoder/layers.3/final_layer_norm/Mul', 'encoder.layers.3.final_layer_norm.bias', '/encoder/layers.3/final_layer_norm/Add_1', '/encoder/layers.4/self_attn_layer_norm/ReduceMean', '/encoder/layers.4/self_attn_layer_norm/Sub', '/encoder/layers.0/self_attn_layer_norm/Constant_output_0', '/encoder/layers.4/self_attn_layer_norm/Pow', '/encoder/layers.4/self_attn_layer_norm/ReduceMean_1', '/encoder/layers.0/self_attn_layer_norm/Constant_1_output_0', '/encoder/layers.4/self_attn_layer_norm/Add', '/encoder/layers.4/self_attn_layer_norm/Sqrt', '/encoder/layers.4/self_attn_layer_norm/Div', 'encoder.layers.4.self_attn_layer_norm.weight', '/encoder/layers.4/self_attn_layer_norm/Mul', 'encoder.layers.4.self_attn_layer_norm.bias', '/encoder/layers.4/self_attn_layer_norm/Add_1', '/encoder/layers.4/final_layer_norm/ReduceMean', '/encoder/layers.4/final_layer_norm/Sub', '/encoder/layers.0/self_attn_layer_norm/Constant_output_0', '/encoder/layers.4/final_layer_norm/Pow', '/encoder/layers.4/final_layer_norm/ReduceMean_1', '/encoder/layers.0/self_attn_layer_norm/Constant_1_output_0', '/encoder/layers.4/final_layer_norm/Add', '/encoder/layers.4/final_layer_norm/Sqrt', '/encoder/layers.4/final_layer_norm/Div', 'encoder.layers.4.final_layer_norm.weight', '/encoder/layers.4/final_layer_norm/Mul', 'encoder.layers.4.final_layer_norm.bias', '/encoder/layers.4/final_layer_norm/Add_1', '/encoder/layers.5/self_attn_layer_norm/ReduceMean', '/encoder/layers.5/self_attn_layer_norm/Sub', '/encoder/layers.0/self_attn_layer_norm/Constant_output_0', '/encoder/layers.5/self_attn_layer_norm/Pow', '/encoder/layers.5/self_attn_layer_norm/ReduceMean_1', '/encoder/layers.0/self_attn_layer_norm/Constant_1_output_0', '/encoder/layers.5/self_attn_layer_norm/Add', '/encoder/layers.5/self_attn_layer_norm/Sqrt', '/encoder/layers.5/self_attn_layer_norm/Div', 'encoder.layers.5.self_attn_layer_norm.weight', '/encoder/layers.5/self_attn_layer_norm/Mul', 'encoder.layers.5.self_attn_layer_norm.bias', '/encoder/layers.5/self_attn_layer_norm/Add_1', '/encoder/layers.5/final_layer_norm/ReduceMean', '/encoder/layers.5/final_layer_norm/Sub', '/encoder/layers.0/self_attn_layer_norm/Constant_output_0', '/encoder/layers.5/final_layer_norm/Pow', '/encoder/layers.5/final_layer_norm/ReduceMean_1', '/encoder/layers.0/self_attn_layer_norm/Constant_1_output_0', '/encoder/layers.5/final_layer_norm/Add', '/encoder/layers.5/final_layer_norm/Sqrt', '/encoder/layers.5/final_layer_norm/Div', 'encoder.layers.5.final_layer_norm.weight', '/encoder/layers.5/final_layer_norm/Mul', 'encoder.layers.5.final_layer_norm.bias', '/encoder/layers.5/final_layer_norm/Add_1', '/encoder/layers.0/self_attn_layer_norm/ReduceMean', '/encoder/layers.0/self_attn_layer_norm/Sub', '/encoder/layers.0/self_attn_layer_norm/Constant_output_0', '/encoder/layers.0/self_attn_layer_norm/Pow', '/encoder/layers.0/self_attn_layer_norm/ReduceMean_1', '/encoder/layers.0/self_attn_layer_norm/Constant_1_output_0', '/encoder/layers.0/self_attn_layer_norm/Add', '/encoder/layers.0/self_attn_layer_norm/Sqrt', '/encoder/layers.0/self_attn_layer_norm/Div', '/encoder/layers.0/final_layer_norm/ReduceMean', '/encoder/layers.0/final_layer_norm/Sub', '/encoder/layers.0/self_attn_layer_norm/Constant_output_0', '/encoder/layers.0/final_layer_norm/Pow', '/encoder/layers.0/final_layer_norm/ReduceMean_1', '/encoder/layers.0/self_attn_layer_norm/Constant_1_output_0', '/encoder/layers.0/final_layer_norm/Add', '/encoder/layers.0/final_layer_norm/Sqrt', '/encoder/layers.0/final_layer_norm/Div', '/encoder/layers.1/self_attn_layer_norm/ReduceMean', '/encoder/layers.1/self_attn_layer_norm/Sub', '/encoder/layers.0/self_attn_layer_norm/Constant_output_0', '/encoder/layers.1/self_attn_layer_norm/Pow', '/encoder/layers.1/self_attn_layer_norm/ReduceMean_1', '/encoder/layers.0/self_attn_layer_norm/Constant_1_output_0', '/encoder/layers.1/self_attn_layer_norm/Add', '/encoder/layers.1/self_attn_layer_norm/Sqrt', '/encoder/layers.1/self_attn_layer_norm/Div', '/encoder/layers.1/final_layer_norm/ReduceMean', '/encoder/layers.1/final_layer_norm/Sub', '/encoder/layers.0/self_attn_layer_norm/Constant_output_0', '/encoder/layers.1/final_layer_norm/Pow', '/encoder/layers.1/final_layer_norm/ReduceMean_1', '/encoder/layers.0/self_attn_layer_norm/Constant_1_output_0', '/encoder/layers.1/final_layer_norm/Add', '/encoder/layers.1/final_layer_norm/Sqrt', '/encoder/layers.1/final_layer_norm/Div', '/encoder/layers.2/self_attn_layer_norm/ReduceMean', '/encoder/layers.2/self_attn_layer_norm/Sub', '/encoder/layers.0/self_attn_layer_norm/Constant_output_0', '/encoder/layers.2/self_attn_layer_norm/Pow', '/encoder/layers.2/self_attn_layer_norm/ReduceMean_1', '/encoder/layers.0/self_attn_layer_norm/Constant_1_output_0', '/encoder/layers.2/self_attn_layer_norm/Add', '/encoder/layers.2/self_attn_layer_norm/Sqrt', '/encoder/layers.2/self_attn_layer_norm/Div', '/encoder/layers.2/final_layer_norm/ReduceMean', '/encoder/layers.2/final_layer_norm/Sub', '/encoder/layers.0/self_attn_layer_norm/Constant_output_0', '/encoder/layers.2/final_layer_norm/Pow', '/encoder/layers.2/final_layer_norm/ReduceMean_1', '/encoder/layers.0/self_attn_layer_norm/Constant_1_output_0', '/encoder/layers.2/final_layer_norm/Add', '/encoder/layers.2/final_layer_norm/Sqrt', '/encoder/layers.2/final_layer_norm/Div', '/encoder/layers.3/self_attn_layer_norm/ReduceMean', '/encoder/layers.3/self_attn_layer_norm/Sub', '/encoder/layers.0/self_attn_layer_norm/Constant_output_0', '/encoder/layers.3/self_attn_layer_norm/Pow', '/encoder/layers.3/self_attn_layer_norm/ReduceMean_1', '/encoder/layers.0/self_attn_layer_norm/Constant_1_output_0', '/encoder/layers.3/self_attn_layer_norm/Add', '/encoder/layers.3/self_attn_layer_norm/Sqrt', '/encoder/layers.3/self_attn_layer_norm/Div', '/encoder/layers.3/final_layer_norm/ReduceMean', '/encoder/layers.3/final_layer_norm/Sub', '/encoder/layers.0/self_attn_layer_norm/Constant_output_0', '/encoder/layers.3/final_layer_norm/Pow', '/encoder/layers.3/final_layer_norm/ReduceMean_1', '/encoder/layers.0/self_attn_layer_norm/Constant_1_output_0', '/encoder/layers.3/final_layer_norm/Add', '/encoder/layers.3/final_layer_norm/Sqrt', '/encoder/layers.3/final_layer_norm/Div', '/encoder/layers.4/self_attn_layer_norm/ReduceMean', '/encoder/layers.4/self_attn_layer_norm/Sub', '/encoder/layers.0/self_attn_layer_norm/Constant_output_0', '/encoder/layers.4/self_attn_layer_norm/Pow', '/encoder/layers.4/self_attn_layer_norm/ReduceMean_1', '/encoder/layers.0/self_attn_layer_norm/Constant_1_output_0', '/encoder/layers.4/self_attn_layer_norm/Add', '/encoder/layers.4/self_attn_layer_norm/Sqrt', '/encoder/layers.4/self_attn_layer_norm/Div', '/encoder/layers.4/final_layer_norm/ReduceMean', '/encoder/layers.4/final_layer_norm/Sub', '/encoder/layers.0/self_attn_layer_norm/Constant_output_0', '/encoder/layers.4/final_layer_norm/Pow', '/encoder/layers.4/final_layer_norm/ReduceMean_1', '/encoder/layers.0/self_attn_layer_norm/Constant_1_output_0', '/encoder/layers.4/final_layer_norm/Add', '/encoder/layers.4/final_layer_norm/Sqrt', '/encoder/layers.4/final_layer_norm/Div', '/encoder/layers.5/self_attn_layer_norm/ReduceMean', '/encoder/layers.5/self_attn_layer_norm/Sub', '/encoder/layers.0/self_attn_layer_norm/Constant_output_0', '/encoder/layers.5/self_attn_layer_norm/Pow', '/encoder/layers.5/self_attn_layer_norm/ReduceMean_1', '/encoder/layers.0/self_attn_layer_norm/Constant_1_output_0', '/encoder/layers.5/self_attn_layer_norm/Add', '/encoder/layers.5/self_attn_layer_norm/Sqrt', '/encoder/layers.5/self_attn_layer_norm/Div', '/encoder/layers.5/final_layer_norm/ReduceMean', '/encoder/layers.5/final_layer_norm/Sub', '/encoder/layers.0/self_attn_layer_norm/Constant_output_0', '/encoder/layers.5/final_layer_norm/Pow', '/encoder/layers.5/final_layer_norm/ReduceMean_1', '/encoder/layers.0/self_attn_layer_norm/Constant_1_output_0', '/encoder/layers.5/final_layer_norm/Add', '/encoder/layers.5/final_layer_norm/Sqrt', '/encoder/layers.5/final_layer_norm/Div']
2026-07-18 12:09 - INFO - epu - quantize - Quantization started...
WARNING:root:Please use QuantFormat.QDQ for activation type QInt8 and weight type QInt8. Or it will lead to bad performance on x64.
2026-07-18 12:09 - INFO - epu - quantize - Quantization done succesfully!
2026-07-18 12:09 - INFO - epu - quantize - ONNX full precision model size: 30.18 MB
2026-07-18 12:09 - INFO - epu - quantize - ONNX quantized model size: 7.71 MB
2026-07-18 12:09 - INFO - epu - quantize - Saved quantized model to onnx/detr_3_transformer_encoder_opt_sym_int8_q.onnx
2026-07-18 12:09 - INFO - epu - quantize - Saved shape inferenced model to onnx/detr_3_transformer_encoder_opt_sym_int8_q.onnx
2026-07-18 12:09 - INFO - epu - quantize - Checking for remaining FLOAT/FLOAT16 types.
2026-07-18 12:09 - INFO - epu - quantize - Model still has FLOAT/FLOAT16 types. Creating ranges for floating point tensors using calibration data
2026-07-18 12:09 - INFO - epu - quantize - Saved tensor ranges to onnx/detr_3_transformer_encoder_opt_sym_int8_q.onnx.tranges
Quantized model: onnx/detr_3_transformer_encoder_opt_sym_int8_q.onnx
Tensor ranges: onnx/detr_3_transformer_encoder_opt_sym_int8_q.onnx.tranges
3. Compile with CGC (Chimera Graph Compiler)
Compile the quantized encoder to ASM targeting QC-U with ISS validation enabled. CGC handles all 6 encoder layers natively:
| Block | CGC kernel | Notes |
|---|---|---|
| Self-attention | nn::multiheadAttentionHead | 8 heads, 360 tokens mapped to 18×20 PE grid |
| LayerNorm | nn::layer_norm | Float precision |
| FFN | matrixMulMatrix + conv2d | 256→2048→256 |
hw_config = HWConfig(
product="QC-U",
ocm_size="8MB",
lrm_size="4kB",
macs_per_pe=16,
clock_freq_ghz=1.56,
ext_rd_bw="64GBps",
ext_wr_bw="64GBps",
num_cores=1,
)
cgc_job = ChimeraJob(
quantized_model,
hw_config=hw_config,
trange_file=tranges_file,
target_lang="ASM",
validate_iss=True,
)
print(f"Compiling {quantized_model} ...")
cgc_job.compile()
print("Compilation successful!")
print(cgc_job)
Compiling onnx/detr_3_transformer_encoder_opt_sym_int8_q.onnx ...
2026-07-18 12:09 - INFO - epu - chimera_job - START==================================onnx_ingest
2026-07-18 12:09 - INFO - epu - chimera_job - Numerical ranges provided
2026-07-18 12:09 - INFO - epu - codegen - START===============================optimize_relay
2026-07-18 12:09 - INFO - epu - codegen - START====================quantize_to_cpu_runnable_fx
2026-07-18 12:09 - INFO - epu - fx -
Source name Op Output 0 Range Output 0 Frac Bits
------------------------------------------------------------ ---------------------- --------------------------- --------------------
/encoder/layers.0/self_attn/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-0.970993f, 1.03682f] 30
/encoder/layers.0/self_attn/Softmax nn.softmax [0.000992848f, 0.00766353f] 30
/encoder/layers.0/Add_output_0_DequantizeLinear contrib.epu.dequantize [-1.42094f, 1.40939f] 30
/encoder/layers.0/self_attn_layer_norm/Add_1 nn.layer_norm [-7.49175f, 7.88788f] 27
/encoder/layers.0/Add_1_output_0_DequantizeLinear contrib.epu.dequantize [-12.9783f, 13.9681f] 27
/encoder/layers.0/final_layer_norm/Add_1 nn.layer_norm [-6.43891f, 5.13585f] 28
/encoder/layers.1/self_attn/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-13.1801f, 12.7682f] 27
/encoder/layers.1/self_attn/Softmax nn.softmax [3.25776e-10f, 0.944214f] 27
/encoder/layers.1/Add_output_0_DequantizeLinear contrib.epu.dequantize [-6.7147f, 5.61393f] 28
/encoder/layers.1/self_attn_layer_norm/Add_1 nn.layer_norm [-10.706f, 10.1506f] 27
/encoder/layers.1/Add_1_output_0_DequantizeLinear contrib.epu.dequantize [-34.2882f, 18.899f] 25
/encoder/layers.1/final_layer_norm/Add_1 nn.layer_norm [-6.67823f, 4.74336f] 28
/encoder/layers.2/self_attn/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-9.0613f, 8.99051f] 27
/encoder/layers.2/self_attn/Softmax nn.softmax [6.24903e-08f, 0.63077f] 27
/encoder/layers.2/Add_output_0_DequantizeLinear contrib.epu.dequantize [-7.14621f, 4.85719f] 28
/encoder/layers.2/self_attn_layer_norm/Add_1 nn.layer_norm [-12.9216f, 9.82898f] 27
/encoder/layers.2/Add_1_output_0_DequantizeLinear contrib.epu.dequantize [-54.407f, 28.6874f] 25
/encoder/layers.2/final_layer_norm/Add_1 nn.layer_norm [-6.26368f, 5.26473f] 28
/encoder/layers.3/self_attn/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-7.7752f, 7.71446f] 28
/encoder/layers.3/self_attn/Softmax nn.softmax [5.52401e-08f, 0.752377f] 28
/encoder/layers.3/Add_output_0_DequantizeLinear contrib.epu.dequantize [-4.77979f, 4.63042f] 28
/encoder/layers.3/self_attn_layer_norm/Add_1 nn.layer_norm [-13.1f, 10.4364f] 27
/encoder/layers.3/Add_1_output_0_DequantizeLinear contrib.epu.dequantize [-76.3629f, 75.7663f] 24
/encoder/layers.3/final_layer_norm/Add_1 nn.layer_norm [-5.74997f, 6.16712f] 28
/encoder/layers.4/self_attn/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-10.8155f, 11.0771f] 27
/encoder/layers.4/self_attn/Softmax nn.softmax [3.90885e-09f, 0.644964f] 27
/encoder/layers.4/Add_output_0_DequantizeLinear contrib.epu.dequantize [-5.60247f, 5.5587f] 28
/encoder/layers.4/self_attn_layer_norm/Add_1 nn.layer_norm [-16.9726f, 16.3017f] 26
/encoder/layers.4/Add_1_output_0_DequantizeLinear contrib.epu.dequantize [-29.5776f, 29.3465f] 26
/encoder/layers.4/final_layer_norm/Add_1 nn.layer_norm [-5.92646f, 5.47685f] 28
/encoder/layers.5/self_attn/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-7.57581f, 8.43972f] 27
/encoder/layers.5/self_attn/Softmax nn.softmax [4.34293e-07f, 0.522208f] 27
/encoder/layers.5/Add_output_0_DequantizeLinear contrib.epu.dequantize [-5.49868f, 5.28389f] 28
/encoder/layers.5/self_attn_layer_norm/Add_1 nn.layer_norm [-13.9091f, 10.8703f] 27
/encoder/layers.5/Add_1_output_0_DequantizeLinear contrib.epu.dequantize [-15.2386f, 12.9384f] 26
/encoder/layers.5/final_layer_norm/Add_1 nn.layer_norm [-4.1048f, 4.63989f] -
2026-07-18 12:09 - INFO - epu - codegen - START====================build_cpu_runnable_fx_relay
2026-07-18 12:09 - INFO - epu - codegen - START=======================quantize_to_chimera_fx
2026-07-18 12:09 - INFO - epu - codegen - START=================================relay_to_tir
2026-07-18 12:09 - INFO - epu - codegen - START===========================relay_to_epu_relay
2026-07-18 12:09 - INFO - epu - codegen - START==============================adapt_and_order
2026-07-18 12:09 - INFO - epu - mac_counter -
2026-07-18 12:09 - INFO - epu - mac_counter - ============================================================
2026-07-18 12:09 - INFO - epu - mac_counter - MAC Operation Count Summary
2026-07-18 12:09 - INFO - epu - mac_counter - ============================================================
2026-07-18 12:09 - INFO - epu - mac_counter - conv2d: 47,185,920 ops (23,592,960 MACs) - /encoder/layers.0/self_attn/out_proj/MatMul_quant
2026-07-18 12:09 - INFO - epu - mac_counter - conv2d: 47,185,920 ops (23,592,960 MACs) - /encoder/layers.1/self_attn/out_proj/MatMul_quant
2026-07-18 12:09 - INFO - epu - mac_counter - conv2d: 47,185,920 ops (23,592,960 MACs) - /encoder/layers.2/self_attn/out_proj/MatMul_quant
2026-07-18 12:09 - INFO - epu - mac_counter - conv2d: 47,185,920 ops (23,592,960 MACs) - /encoder/layers.3/self_attn/out_proj/MatMul_quant
2026-07-18 12:09 - INFO - epu - mac_counter - conv2d: 47,185,920 ops (23,592,960 MACs) - /encoder/layers.4/self_attn/out_proj/MatMul_quant
2026-07-18 12:09 - INFO - epu - mac_counter - conv2d: 47,185,920 ops (23,592,960 MACs) - /encoder/layers.5/self_attn/out_proj/MatMul_quant
2026-07-18 12:09 - INFO - epu - mac_counter - ------------------------------------------------------------
2026-07-18 12:09 - INFO - epu - mac_counter - Total: 283,115,520 ops (141,557,760 MACs)
2026-07-18 12:09 - INFO - epu - mac_counter - ============================================================
2026-07-18 12:09 - INFO - epu - mac_counter -
2026-07-18 12:09 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-07-18 12:09 - INFO - epu - codegen - START=============================plan_lrm_virtual
2026-07-18 12:10 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-07-18 12:10 - INFO - epu - codegen - START===============================lrm_alloc_loop
2026-07-18 12:10 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-07-18 12:10 - INFO - epu - codegen - START================================lrm_splitting
2026-07-18 12:11 - INFO - epu - codegen - START==============================ext_split_relay
2026-07-18 12:12 - INFO - epu - codegen - START====================================build_tir
2026-07-18 12:12 - INFO - epu - chimera_job - Compilation of detr_3_transformer_encoder_opt_sym_int8_q_QC_U_1d56_8MB_4kB_64GBps_64GBps_16_OFF_x1_x1 successful
Compilation successful!
╒═════════════════════╤════════════════════════════════════════════════════════════════════════════════════════╕
│ Module Name │ detr_3_transformer_encoder_opt_sym_int8_q_QC_U_1d56_8MB_4kB_64GBps_64GBps_16_OFF_x1_x1 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ ONNX File │ onnx/detr_3_transformer_encoder_opt_sym_int8_q.onnx │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ Product Target │ QC-U │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ Number of Cores │ 1 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ ISS Clock Frequency │ 1.560 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ L2M Size │ 8MB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ LRM Size │ 4kB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ External Read BW │ 64GBps │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ External Write BW │ 64GBps │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ MACS per PE │ 16 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ Max L2M │ 6.475MB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ Max LRM │ 0.250kB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ Max Temp Ext Bytes │ 0.000MB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ Network GMACs │ 3.229 │
╘═════════════════════╧════════════════════════════════════════════════════════════════════════════════════════╛
╒════╤════════╤════════════════╤═══════════════╤══════════════════════════╤═══════╕
│ │ Type │ Name │ shape │ type │ mse │
╞════╪════════╪════════════════╪═══════════════╪══════════════════════════╪═══════╡
│ 0 │ Input │ src │ [1, 360, 256] │ tensor[FixedPoint32<29>] │ n/a │
├────┼────────┼────────────────┼───────────────┼──────────────────────────┼───────┤
│ 1 │ Input │ pos_embed │ [1, 360, 256] │ tensor[FixedPoint32<30>] │ n/a │
├────┼────────┼────────────────┼───────────────┼──────────────────────────┼───────┤
│ 2 │ Output │ encoder_memory │ [1, 360, 256] │ tensor[FixedPoint32<28>] │ n/a │
╘════╧════════╧════════════════╧═══════════════╧══════════════════════════╧═══════╛
4. Validate: ISS vs ORT
Compare the ISS (hardware simulator) output against the ONNX Runtime reference to verify numerical correctness of the compiled model.
validation_result = cgc_job.validate_ort_iss()
print(cgc_job)
print(validation_result)
2026-07-18 12:12 - INFO - epu - iss_testing - Found tranges for input: <tvm.contrib.epu.interval.Interval object at 0x7486d420f190>
2026-07-18 12:12 - INFO - epu - iss_testing - Found tranges for input: <tvm.contrib.epu.interval.Interval object at 0x74857c5ff6a0>
2026-07-18 12:12 - INFO - epu - iss_testing - Found tranges for input: <tvm.contrib.epu.interval.Interval object at 0x7486d420f130>
2026-07-18 12:12 - INFO - epu - iss_testing - Found tranges for input: <tvm.contrib.epu.interval.Interval object at 0x7486d420f130>
2026-07-18 12:12 - INFO - epu - iss_testing - Started Executing Onnxruntime...
2026-07-18 12:12 - INFO - epu - iss_testing - Done 0:00:00.268258
2026-07-18 12:12 - INFO - epu - iss_testing - Found tranges for input: <tvm.contrib.epu.interval.Interval object at 0x74857c5ff6a0>
2026-07-18 12:12 - INFO - epu - iss_testing - Found tranges for input: <tvm.contrib.epu.interval.Interval object at 0x74857c5ff6a0>
FILM 79/79: 100%|███████████████████████████████████████████████████| 79/79 [01:34<00:00, 1.20s/it]
2026-07-18 12:14 - WARNING - epu - iss_testing - Node was skipped due to being multi-output: /encoder/layers.1/self_attn/MatMul_1_quant
2026-07-18 12:14 - WARNING - epu - iss_testing - Node was skipped due to being multi-output: /encoder/layers.2/self_attn/MatMul_1_quant
2026-07-18 12:14 - WARNING - epu - iss_testing - Node was skipped due to being multi-output: /encoder/layers.0/self_attn/MatMul_1_quant
2026-07-18 12:14 - WARNING - epu - iss_testing - Node was skipped due to being multi-output: /encoder/layers.3/self_attn/MatMul_1_quant
2026-07-18 12:14 - WARNING - epu - iss_testing - Node was skipped due to being multi-output: /encoder/layers.4/self_attn/MatMul_1_quant
2026-07-18 12:14 - WARNING - epu - iss_testing - Node was skipped due to being multi-output: /encoder/layers.5/self_attn/MatMul_1_quant
2026-07-18 12:14 - INFO - epu - iss_testing -
======================================================================
ISS validation results (quantized model, rtol=0.1, atol=-1)
======================================================================
pos_embed_QuantizeLinear:0 (PASS, Node, int8)
Bitwise Mismatches 0 / 92160 (0.0000%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0
PSNR inf dB
Max Abs Err 0
Max Abs Err / ε 0
Max Err Loc (0, 0, 0)
ORT @ max err 41
ISS @ max err 41
src_QuantizeLinear:0 (PASS, Node, int8)
Bitwise Mismatches 0 / 92160 (0.0000%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0
PSNR inf dB
Max Abs Err 0
Max Abs Err / ε 0
Max Err Loc (0, 0, 0)
ORT @ max err 39
ISS @ max err 39
/encoder/layers.0/self_attn/Add_quant:0 (PASS, Node, int8)
Bitwise Mismatches 0 / 92160 (0.0000%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0
PSNR inf dB
Max Abs Err 0
Max Abs Err / ε 0
Max Err Loc (0, 0, 0)
ORT @ max err 52
ISS @ max err 52
/encoder/layers.0/self_attn/MatMul_1_quant:0 (FAIL, Node, int8)
Bitwise Mismatches 34661 / 92160 (37.6096%)
Mismatches > Tol 31544 / 92160 (34.2274%)
Thresholds rtol=0.1, atol=-1
RMSE 85.6707
PSNR 9.47 dB
Max Abs Err 255
Max Abs Err / ε 255
Max Err Loc (3, 22, 16)
ORT @ max err 127
ISS @ max err -128
/encoder/layers.0/Add_output_0_DequantizeLinear:0 (PASS, Node, custom[qfp.30]32)
Bitwise Mismatches 3696 / 92160 (4.0104%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0.00458071
PSNR 51.67 dB
Max Abs Err 0.115523
Max Abs Err / ε 1.24042e+08
Max Err Loc (0, 192, 248)
ORT @ max err 0.046209384
ISS @ max err -0.06931408
/encoder/layers.0/self_attn_layer_norm/Add_1:0 (PASS, Node, custom[qfp.27]32)
Bitwise Mismatches 92077 / 92160 (99.9099%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0.0148193
PSNR 56.40 dB
Max Abs Err 0.434839
Max Abs Err / ε 5.83631e+07
Max Err Loc (0, 100, 0)
ORT @ max err 3.1678429
ISS @ max err 2.7330039
/encoder/layers.0/self_attn_layer_norm/Add_1_output_0_QuantizeLinear:0 (PASS, Node, int8)
Bitwise Mismatches 3469 / 92160 (3.7641%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0.278945
PSNR 59.22 dB
Max Abs Err 7
Max Abs Err / ε 7
Max Err Loc (0, 1, 0)
ORT @ max err 30
ISS @ max err 23
/encoder/layers.0/fc1/MatMul_quant:0 (PASS, Node, int8)
Bitwise Mismatches 139090 / 737280 (18.8653%)
Mismatches > Tol 0 / 737280 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0.4406
PSNR 55.25 dB
Max Abs Err 3
Max Abs Err / ε 3
Max Err Loc (0, 1, 206)
ORT @ max err 3
ISS @ max err 6
[ERROR] /encoder/layers.0/activation_fn/Relu:0
Error Skipping validation for /encoder/layers.0/activation_fn/Relu:0 dtype mismatch between ISS int8 and ORT float32
/encoder/layers.0/fc2/MatMul_quant:0 (PASS, Node, int8)
Bitwise Mismatches 25048 / 92160 (27.1788%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0.56167
PSNR 53.14 dB
Max Abs Err 7
Max Abs Err / ε 7
Max Err Loc (0, 283, 231)
ORT @ max err -114
ISS @ max err -121
/encoder/layers.0/Add_1_output_0_DequantizeLinear:0 (PASS, Node, custom[qfp.27]32)
Bitwise Mismatches 19462 / 92160 (21.1176%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0.0549628
PSNR 53.52 dB
Max Abs Err 0.549926
Max Abs Err / ε 7.38098e+07
Max Err Loc (0, 283, 231)
ORT @ max err -12.098379
ISS @ max err -12.648305
/encoder/layers.0/final_layer_norm/Add_1:0 (PASS, Node, custom[qfp.28]32)
Bitwise Mismatches 92041 / 92160 (99.8709%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0.0352108
PSNR 49.26 dB
Max Abs Err 0.322629
Max Abs Err / ε 8.6605e+07
Max Err Loc (0, 78, 248)
ORT @ max err 0.66234875
ISS @ max err 0.33972013
/encoder/layers.0/final_layer_norm/Add_1_output_0_QuantizeLinear:0 (PASS, Node, int8)
Bitwise Mismatches 23057 / 92160 (25.0184%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0.700539
PSNR 51.22 dB
Max Abs Err 6
Max Abs Err / ε 6
Max Err Loc (0, 78, 248)
ORT @ max err 12
ISS @ max err 6
/encoder/layers.1/self_attn/Add_quant:0 (PASS, Node, int8)
Bitwise Mismatches 22998 / 92160 (24.9544%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0.698577
PSNR 51.25 dB
Max Abs Err 6
Max Abs Err / ε 6
Max Err Loc (0, 78, 248)
ORT @ max err 25
ISS @ max err 19
/encoder/layers.1/self_attn/MatMul_1_quant:0 (FAIL, Node, int8)
Bitwise Mismatches 90191 / 92160 (97.8635%)
Mismatches > Tol 51007 / 92160 (55.3461%)
Thresholds rtol=0.1, atol=-1
RMSE 19.0385
PSNR 22.54 dB
Max Abs Err 123
Max Abs Err / ε 123
Max Err Loc (4, 64, 4)
ORT @ max err -106
ISS @ max err 17
/encoder/layers.1/Add_output_0_DequantizeLinear:0 (PASS, Node, custom[qfp.28]32)
Bitwise Mismatches 45248 / 92160 (49.0972%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0.0515703
PSNR 46.50 dB
Max Abs Err 0.330231
Max Abs Err / ε 8.86457e+07
Max Err Loc (0, 78, 248)
ORT @ max err 0.6054237
ISS @ max err 0.2751926
/encoder/layers.1/self_attn_layer_norm/Add_1:0 (PASS, Node, custom[qfp.27]32)
Bitwise Mismatches 92160 / 92160 (100.0000%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0.049441
PSNR 51.01 dB
Max Abs Err 0.376432
Max Abs Err / ε 5.05239e+07
Max Err Loc (0, 78, 248)
ORT @ max err 1.1311849
ISS @ max err 0.75475246
/encoder/layers.1/self_attn_layer_norm/Add_1_output_0_QuantizeLinear:0 (PASS, Node, int8)
Bitwise Mismatches 31389 / 92160 (34.0592%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0.654694
PSNR 51.81 dB
Max Abs Err 5
Max Abs Err / ε 5
Max Err Loc (0, 78, 248)
ORT @ max err 13
ISS @ max err 8
/encoder/layers.1/fc1/MatMul_quant:0 (PASS, Node, int8)
Bitwise Mismatches 469112 / 737280 (63.6274%)
Mismatches > Tol 0 / 737280 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 1.11857
PSNR 47.16 dB
Max Abs Err 6
Max Abs Err / ε 6
Max Err Loc (0, 64, 1395)
ORT @ max err -63
ISS @ max err -57
[ERROR] /encoder/layers.1/activation_fn/Relu:0
Error Skipping validation for /encoder/layers.1/activation_fn/Relu:0 dtype mismatch between ISS int8 and ORT float32
/encoder/layers.1/fc2/MatMul_quant:0 (PASS, Node, int8)
Bitwise Mismatches 9854 / 92160 (10.6923%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0.328166
PSNR 57.81 dB
Max Abs Err 4
Max Abs Err / ε 4
Max Err Loc (0, 272, 142)
ORT @ max err -23
ISS @ max err -19
/encoder/layers.1/Add_1_output_0_DequantizeLinear:0 (PASS, Node, custom[qfp.25]32)
Bitwise Mismatches 19100 / 92160 (20.7248%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0.128239
PSNR 46.97 dB
Max Abs Err 1.07995
Max Abs Err / ε 3.62369e+07
Max Err Loc (0, 272, 142)
ORT @ max err -7.559614
ISS @ max err -6.479669
/encoder/layers.1/final_layer_norm/Add_1:0 (PASS, Node, custom[qfp.28]32)
Bitwise Mismatches 92160 / 92160 (100.0000%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0.0781414
PSNR 42.00 dB
Max Abs Err 0.486731
Max Abs Err / ε 1.30656e+08
Max Err Loc (0, 116, 18)
ORT @ max err -0.10809779
ISS @ max err -0.5948286
/encoder/layers.1/final_layer_norm/Add_1_output_0_QuantizeLinear:0 (PASS, Node, int8)
Bitwise Mismatches 26204 / 92160 (28.4332%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 1.46396
PSNR 44.82 dB
Max Abs Err 9
Max Abs Err / ε 9
Max Err Loc (0, 54, 166)
ORT @ max err 4
ISS @ max err 13
/encoder/layers.2/self_attn/Add_quant:0 (PASS, Node, int8)
Bitwise Mismatches 26175 / 92160 (28.4017%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 1.45877
PSNR 44.85 dB
Max Abs Err 9
Max Abs Err / ε 9
Max Err Loc (0, 54, 166)
ORT @ max err 6
ISS @ max err 15
/encoder/layers.2/self_attn/MatMul_1_quant:0 (FAIL, Node, int8)
Bitwise Mismatches 89752 / 92160 (97.3872%)
Mismatches > Tol 60994 / 92160 (66.1827%)
Thresholds rtol=0.1, atol=-1
RMSE 18.1067
PSNR 22.97 dB
Max Abs Err 115
Max Abs Err / ε 115
Max Err Loc (1, 267, 17)
ORT @ max err -60
ISS @ max err 55
/encoder/layers.2/Add_output_0_DequantizeLinear:0 (PASS, Node, custom[qfp.28]32)
Bitwise Mismatches 46413 / 92160 (50.3613%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0.0865961
PSNR 41.29 dB
Max Abs Err 0.558298
Max Abs Err / ε 1.49867e+08
Max Err Loc (0, 266, 30)
ORT @ max err -0.39080852
ISS @ max err 0.16748936
/encoder/layers.2/self_attn_layer_norm/Add_1:0 (PASS, Node, custom[qfp.27]32)
Bitwise Mismatches 92160 / 92160 (100.0000%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0.112135
PSNR 44.68 dB
Max Abs Err 0.87938
Max Abs Err / ε 1.18028e+08
Max Err Loc (0, 54, 166)
ORT @ max err 0.74862397
ISS @ max err 1.628004
/encoder/layers.2/self_attn_layer_norm/Add_1_output_0_QuantizeLinear:0 (PASS, Node, int8)
Bitwise Mismatches 41957 / 92160 (45.5263%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 1.28273
PSNR 45.97 dB
Max Abs Err 10
Max Abs Err / ε 10
Max Err Loc (0, 54, 166)
ORT @ max err 8
ISS @ max err 18
/encoder/layers.2/fc1/MatMul_quant:0 (PASS, Node, int8)
Bitwise Mismatches 584244 / 737280 (79.2432%)
Mismatches > Tol 0 / 737280 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 1.97053
PSNR 42.24 dB
Max Abs Err 11
Max Abs Err / ε 11
Max Err Loc (0, 296, 1322)
ORT @ max err -22
ISS @ max err -33
[ERROR] /encoder/layers.2/activation_fn/Relu:0
Error Skipping validation for /encoder/layers.2/activation_fn/Relu:0 dtype mismatch between ISS int8 and ORT float32
/encoder/layers.2/fc2/MatMul_quant:0 (PASS, Node, int8)
Bitwise Mismatches 19154 / 92160 (20.7834%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0.471554
PSNR 54.66 dB
Max Abs Err 8
Max Abs Err / ε 8
Max Err Loc (0, 190, 142)
ORT @ max err -67
ISS @ max err -59
/encoder/layers.2/Add_1_output_0_DequantizeLinear:0 (PASS, Node, custom[qfp.25]32)
Bitwise Mismatches 26406 / 92160 (28.6523%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0.284744
PSNR 45.14 dB
Max Abs Err 3.46227
Max Abs Err / ε 1.16174e+08
Max Err Loc (0, 190, 142)
ORT @ max err -33.633446
ISS @ max err -30.17118
/encoder/layers.2/final_layer_norm/Add_1:0 (FAIL, Node, custom[qfp.28]32)
Bitwise Mismatches 92160 / 92160 (100.0000%)
Mismatches > Tol 204 / 92160 (0.2214%)
Thresholds rtol=0.1, atol=-1
RMSE 0.128548
PSNR 36.78 dB
Max Abs Err 0.864373
Max Abs Err / ε 2.32028e+08
Max Err Loc (0, 354, 166)
ORT @ max err -0.81512535
ISS @ max err -1.6794984
/encoder/layers.2/final_layer_norm/Add_1_output_0_QuantizeLinear:0 (FAIL, Node, int8)
Bitwise Mismatches 45189 / 92160 (49.0332%)
Mismatches > Tol 305 / 92160 (0.3309%)
Thresholds rtol=0.1, atol=-1
RMSE 3.58029
PSNR 37.05 dB
Max Abs Err 24
Max Abs Err / ε 24
Max Err Loc (0, 251, 42)
ORT @ max err -39
ISS @ max err -15
/encoder/layers.3/self_attn/Add_quant:0 (FAIL, Node, int8)
Bitwise Mismatches 43195 / 92160 (46.8696%)
Mismatches > Tol 258 / 92160 (0.2799%)
Thresholds rtol=0.1, atol=-1
RMSE 3.21432
PSNR 37.99 dB
Max Abs Err 22
Max Abs Err / ε 22
Max Err Loc (0, 354, 166)
ORT @ max err -15
ISS @ max err -37
/encoder/layers.3/self_attn/MatMul_1_quant:0 (FAIL, Node, int8)
Bitwise Mismatches 71844 / 92160 (77.9557%)
Mismatches > Tol 50606 / 92160 (54.9110%)
Thresholds rtol=0.1, atol=-1
RMSE 14.2651
PSNR 25.05 dB
Max Abs Err 76
Max Abs Err / ε 76
Max Err Loc (6, 252, 29)
ORT @ max err -30
ISS @ max err 46
/encoder/layers.3/Add_output_0_DequantizeLinear:0 (FAIL, Node, custom[qfp.28]32)
Bitwise Mismatches 68850 / 92160 (74.7070%)
Mismatches > Tol 270 / 92160 (0.2930%)
Thresholds rtol=0.1, atol=-1
RMSE 0.137466
PSNR 35.91 dB
Max Abs Err 0.933552
Max Abs Err / ε 2.50599e+08
Max Err Loc (0, 119, 59)
ORT @ max err 0.0
ISS @ max err 0.9335523
/encoder/layers.3/self_attn_layer_norm/Add_1:0 (FAIL, Node, custom[qfp.27]32)
Bitwise Mismatches 92160 / 92160 (100.0000%)
Mismatches > Tol 9 / 92160 (0.0098%)
Thresholds rtol=0.1, atol=-1
RMSE 0.185661
PSNR 41.12 dB
Max Abs Err 1.59523
Max Abs Err / ε 2.14108e+08
Max Err Loc (0, 251, 42)
ORT @ max err -1.9033345
ISS @ max err -0.30810267
/encoder/layers.3/self_attn_layer_norm/Add_1_output_0_QuantizeLinear:0 (FAIL, Node, int8)
Bitwise Mismatches 61395 / 92160 (66.6178%)
Mismatches > Tol 100 / 92160 (0.1085%)
Thresholds rtol=0.1, atol=-1
RMSE 2.93314
PSNR 38.78 dB
Max Abs Err 25
Max Abs Err / ε 25
Max Err Loc (0, 251, 42)
ORT @ max err -30
ISS @ max err -5
/encoder/layers.3/fc1/MatMul_quant:0 (FAIL, Node, int8)
Bitwise Mismatches 639166 / 737280 (86.6924%)
Mismatches > Tol 177 / 737280 (0.0240%)
Thresholds rtol=0.1, atol=-1
RMSE 3.25979
PSNR 37.87 dB
Max Abs Err 22
Max Abs Err / ε 22
Max Err Loc (0, 199, 749)
ORT @ max err -4
ISS @ max err 18
[ERROR] /encoder/layers.3/activation_fn/Relu:0
Error Skipping validation for /encoder/layers.3/activation_fn/Relu:0 dtype mismatch between ISS int8 and ORT float32
/encoder/layers.3/fc2/MatMul_quant:0 (FAIL, Node, int8)
Bitwise Mismatches 23760 / 92160 (25.7812%)
Mismatches > Tol 2 / 92160 (0.0022%)
Thresholds rtol=0.1, atol=-1
RMSE 0.673146
PSNR 51.57 dB
Max Abs Err 19
Max Abs Err / ε 19
Max Err Loc (0, 75, 93)
ORT @ max err -80
ISS @ max err -61
/encoder/layers.3/Add_1_output_0_DequantizeLinear:0 (FAIL, Node, custom[qfp.24]32)
Bitwise Mismatches 33512 / 92160 (36.3628%)
Mismatches > Tol 1 / 92160 (0.0011%)
Thresholds rtol=0.1, atol=-1
RMSE 0.460409
PSNR 50.35 dB
Max Abs Err 10.7385
Max Abs Err / ε 1.80163e+08
Max Err Loc (0, 75, 93)
ORT @ max err -51.902935
ISS @ max err -41.164394
/encoder/layers.3/final_layer_norm/Add_1:0 (FAIL, Node, custom[qfp.28]32)
Bitwise Mismatches 92139 / 92160 (99.9772%)
Mismatches > Tol 496 / 92160 (0.5382%)
Thresholds rtol=0.1, atol=-1
RMSE 0.165704
PSNR 36.23 dB
Max Abs Err 1.36697
Max Abs Err / ε 3.66942e+08
Max Err Loc (0, 119, 59)
ORT @ max err -0.24591827
ISS @ max err 1.1210474
/encoder/layers.3/final_layer_norm/Add_1_output_0_QuantizeLinear:0 (FAIL, Node, int8)
Bitwise Mismatches 51098 / 92160 (55.4449%)
Mismatches > Tol 585 / 92160 (0.6348%)
Thresholds rtol=0.1, atol=-1
RMSE 3.92833
PSNR 36.25 dB
Max Abs Err 32
Max Abs Err / ε 32
Max Err Loc (0, 119, 59)
ORT @ max err -6
ISS @ max err 26
/encoder/layers.4/self_attn/Add_quant:0 (FAIL, Node, int8)
Bitwise Mismatches 49877 / 92160 (54.1200%)
Mismatches > Tol 437 / 92160 (0.4742%)
Thresholds rtol=0.1, atol=-1
RMSE 3.68536
PSNR 36.80 dB
Max Abs Err 30
Max Abs Err / ε 30
Max Err Loc (0, 119, 59)
ORT @ max err 5
ISS @ max err 35
/encoder/layers.4/self_attn/MatMul_1_quant:0 (FAIL, Node, int8)
Bitwise Mismatches 89065 / 92160 (96.6417%)
Mismatches > Tol 38656 / 92160 (41.9444%)
Thresholds rtol=0.1, atol=-1
RMSE 14.1123
PSNR 25.14 dB
Max Abs Err 102
Max Abs Err / ε 102
Max Err Loc (6, 314, 4)
ORT @ max err -7
ISS @ max err 95
/encoder/layers.4/Add_output_0_DequantizeLinear:0 (FAIL, Node, custom[qfp.28]32)
Bitwise Mismatches 71956 / 92160 (78.0773%)
Mismatches > Tol 608 / 92160 (0.6597%)
Thresholds rtol=0.1, atol=-1
RMSE 0.175435
PSNR 36.07 dB
Max Abs Err 1.35685
Max Abs Err / ε 3.64226e+08
Max Err Loc (0, 119, 59)
ORT @ max err -0.26261586
ISS @ max err 1.0942327
/encoder/layers.4/self_attn_layer_norm/Add_1:0 (FAIL, Node, custom[qfp.26]32)
Bitwise Mismatches 92159 / 92160 (99.9989%)
Mismatches > Tol 3 / 92160 (0.0033%)
Thresholds rtol=0.1, atol=-1
RMSE 0.220842
PSNR 43.32 dB
Max Abs Err 2.38832
Max Abs Err / ε 1.60277e+08
Max Err Loc (0, 123, 93)
ORT @ max err -12.805854
ISS @ max err -10.417539
/encoder/layers.4/self_attn_layer_norm/Add_1_output_0_QuantizeLinear:0 (FAIL, Node, int8)
Bitwise Mismatches 54762 / 92160 (59.4206%)
Mismatches > Tol 12 / 92160 (0.0130%)
Thresholds rtol=0.1, atol=-1
RMSE 2.02519
PSNR 42.00 dB
Max Abs Err 21
Max Abs Err / ε 21
Max Err Loc (0, 123, 93)
ORT @ max err -115
ISS @ max err -94
/encoder/layers.4/fc1/MatMul_quant:0 (FAIL, Node, int8)
Bitwise Mismatches 631485 / 737280 (85.6506%)
Mismatches > Tol 791 / 737280 (0.1073%)
Thresholds rtol=0.1, atol=-1
RMSE 3.55054
PSNR 37.12 dB
Max Abs Err 58
Max Abs Err / ε 58
Max Err Loc (0, 186, 1442)
ORT @ max err -35
ISS @ max err -93
[ERROR] /encoder/layers.4/activation_fn/Relu:0
Error Skipping validation for /encoder/layers.4/activation_fn/Relu:0 dtype mismatch between ISS int8 and ORT float32
/encoder/layers.4/fc2/MatMul_quant:0 (FAIL, Node, int8)
Bitwise Mismatches 45512 / 92160 (49.3837%)
Mismatches > Tol 42 / 92160 (0.0456%)
Thresholds rtol=0.1, atol=-1
RMSE 1.21041
PSNR 46.47 dB
Max Abs Err 27
Max Abs Err / ε 27
Max Err Loc (0, 4, 93)
ORT @ max err -52
ISS @ max err -25
/encoder/layers.4/Add_1_output_0_DequantizeLinear:0 (FAIL, Node, custom[qfp.26]32)
Bitwise Mismatches 51648 / 92160 (56.0417%)
Mismatches > Tol 2 / 92160 (0.0022%)
Thresholds rtol=0.1, atol=-1
RMSE 0.291424
PSNR 45.66 dB
Max Abs Err 4.6215
Max Abs Err / ε 3.10143e+08
Max Err Loc (0, 4, 47)
ORT @ max err -17.792757
ISS @ max err -13.171262
/encoder/layers.4/final_layer_norm/Add_1:0 (FAIL, Node, custom[qfp.28]32)
Bitwise Mismatches 92160 / 92160 (100.0000%)
Mismatches > Tol 150 / 92160 (0.1628%)
Thresholds rtol=0.1, atol=-1
RMSE 0.133979
PSNR 37.90 dB
Max Abs Err 1.21452
Max Abs Err / ε 3.26019e+08
Max Err Loc (0, 57, 42)
ORT @ max err 1.2169237
ISS @ max err 0.0024067163
/encoder/layers.4/final_layer_norm/Add_1_output_0_QuantizeLinear:0 (FAIL, Node, int8)
Bitwise Mismatches 54909 / 92160 (59.5801%)
Mismatches > Tol 198 / 92160 (0.2148%)
Thresholds rtol=0.1, atol=-1
RMSE 3.1085
PSNR 38.28 dB
Max Abs Err 28
Max Abs Err / ε 28
Max Err Loc (0, 57, 42)
ORT @ max err 28
ISS @ max err 0
/encoder/layers.5/self_attn/Add_quant:0 (FAIL, Node, int8)
Bitwise Mismatches 53912 / 92160 (58.4983%)
Mismatches > Tol 157 / 92160 (0.1704%)
Thresholds rtol=0.1, atol=-1
RMSE 2.85675
PSNR 39.01 dB
Max Abs Err 26
Max Abs Err / ε 26
Max Err Loc (0, 57, 42)
ORT @ max err 34
ISS @ max err 8
/encoder/layers.5/self_attn/MatMul_1_quant:0 (FAIL, Node, int8)
Bitwise Mismatches 81010 / 92160 (87.9015%)
Mismatches > Tol 49702 / 92160 (53.9301%)
Thresholds rtol=0.1, atol=-1
RMSE 14.6604
PSNR 24.81 dB
Max Abs Err 81
Max Abs Err / ε 81
Max Err Loc (7, 210, 2)
ORT @ max err -26
ISS @ max err 55
/encoder/layers.5/Add_output_0_DequantizeLinear:0 (FAIL, Node, custom[qfp.28]32)
Bitwise Mismatches 67223 / 92160 (72.9416%)
Mismatches > Tol 189 / 92160 (0.2051%)
Thresholds rtol=0.1, atol=-1
RMSE 0.137373
PSNR 37.58 dB
Max Abs Err 1.15988
Max Abs Err / ε 3.11352e+08
Max Err Loc (0, 57, 42)
ORT @ max err 1.2028368
ISS @ max err 0.042958457
/encoder/layers.5/self_attn_layer_norm/Add_1:0 (FAIL, Node, custom[qfp.27]32)
Bitwise Mismatches 92160 / 92160 (100.0000%)
Mismatches > Tol 39 / 92160 (0.0423%)
Thresholds rtol=0.1, atol=-1
RMSE 0.203503
PSNR 41.61 dB
Max Abs Err 2.36252
Max Abs Err / ε 3.17092e+08
Max Err Loc (0, 123, 93)
ORT @ max err -12.057574
ISS @ max err -9.695057
/encoder/layers.5/self_attn_layer_norm/Add_1_output_0_QuantizeLinear:0 (FAIL, Node, int8)
Bitwise Mismatches 58061 / 92160 (63.0002%)
Mismatches > Tol 61 / 92160 (0.0662%)
Thresholds rtol=0.1, atol=-1
RMSE 2.10156
PSNR 41.68 dB
Max Abs Err 24
Max Abs Err / ε 24
Max Err Loc (0, 123, 93)
ORT @ max err -122
ISS @ max err -98
/encoder/layers.5/fc1/MatMul_quant:0 (FAIL, Node, int8)
Bitwise Mismatches 614098 / 737280 (83.2924%)
Mismatches > Tol 190 / 737280 (0.0258%)
Thresholds rtol=0.1, atol=-1
RMSE 2.9119
PSNR 38.85 dB
Max Abs Err 26
Max Abs Err / ε 26
Max Err Loc (0, 186, 220)
ORT @ max err -70
ISS @ max err -44
[ERROR] /encoder/layers.5/activation_fn/Relu:0
Error Skipping validation for /encoder/layers.5/activation_fn/Relu:0 dtype mismatch between ISS int8 and ORT float32
/encoder/layers.5/fc2/MatMul_quant:0 (FAIL, Node, int8)
Bitwise Mismatches 34357 / 92160 (37.2797%)
Mismatches > Tol 128 / 92160 (0.1389%)
Thresholds rtol=0.1, atol=-1
RMSE 0.791194
PSNR 50.17 dB
Max Abs Err 11
Max Abs Err / ε 11
Max Err Loc (0, 103, 47)
ORT @ max err -26
ISS @ max err -15
/encoder/layers.5/Add_1_output_0_DequantizeLinear:0 (FAIL, Node, custom[qfp.26]32)
Bitwise Mismatches 54222 / 92160 (58.8346%)
Mismatches > Tol 19 / 92160 (0.0206%)
Thresholds rtol=0.1, atol=-1
RMSE 0.221234
PSNR 41.88 dB
Max Abs Err 2.44392
Max Abs Err / ε 1.64009e+08
Max Err Loc (0, 123, 93)
ORT @ max err -12.219604
ISS @ max err -9.775682
/encoder/layers.5/final_layer_norm/Add_1:0 (FAIL, Node, custom[qfp.28]32)
Bitwise Mismatches 92160 / 92160 (100.0000%)
Mismatches > Tol 761 / 92160 (0.8257%)
Thresholds rtol=0.1, atol=-1
RMSE 0.119715
PSNR 35.68 dB
Max Abs Err 1.21964
Max Abs Err / ε 3.27394e+08
Max Err Loc (0, 19, 214)
ORT @ max err -2.7607398
ISS @ max err -1.541104
encoder_memory (FAIL, Output, custom[qfp.28]32)
Bitwise Mismatches 92160 / 92160 (100.0000%)
Mismatches > Tol 761 / 92160 (0.8257%)
Thresholds rtol=0.1, atol=-1
RMSE 0.119715
PSNR 35.68 dB
Max Abs Err 1.21964
Max Abs Err / ε 3.27394e+08
Max Err Loc (0, 19, 214)
ORT @ max err -2.7607398
ISS @ max err -1.541104
68 entries
Differences detected between ort and iss
╒═════════════════════╤════════════════════════════════════════════════════════════════════════════════════════╕
│ Module Name │ detr_3_transformer_encoder_opt_sym_int8_q_QC_U_1d56_8MB_4kB_64GBps_64GBps_16_OFF_x1_x1 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ ONNX File │ onnx/detr_3_transformer_encoder_opt_sym_int8_q.onnx │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ Product Target │ QC-U │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ Number of Cores │ 1 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ ISS Clock Frequency │ 1.560 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ L2M Size │ 8MB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ LRM Size │ 4kB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ External Read BW │ 64GBps │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ External Write BW │ 64GBps │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ MACS per PE │ 16 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ Max L2M │ 6.475MB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ Max LRM │ 0.250kB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ Max Temp Ext Bytes │ 0.000MB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ Network GMACs │ 3.229 │
╘═════════════════════╧════════════════════════════════════════════════════════════════════════════════════════╛
╒════╤════════╤════════════════╤═══════════════╤══════════════════════════╤═══════╕
│ │ Type │ Name │ shape │ type │ mse │
╞════╪════════╪════════════════╪═══════════════╪══════════════════════════╪═══════╡
│ 0 │ Input │ src │ [1, 360, 256] │ tensor[FixedPoint32<29>] │ n/a │
├────┼────────┼────────────────┼───────────────┼──────────────────────────┼───────┤
│ 1 │ Input │ pos_embed │ [1, 360, 256] │ tensor[FixedPoint32<30>] │ n/a │
├────┼────────┼────────────────┼───────────────┼──────────────────────────┼───────┤
│ 2 │ Output │ encoder_memory │ [1, 360, 256] │ tensor[FixedPoint32<28>] │ n/a │
╘════╧════════╧════════════════╧═══════════════╧══════════════════════════╧═══════╛
Post-ISS Report 1.56 GHz ***
Fully placed-and-routed gate simulation:
╒══════════════════════════════════╤═════════╕
│ Latency (ms) │ 5.56 │
├──────────────────────────────────┼─────────┤
│ FPS │ 179.87 │
├──────────────────────────────────┼─────────┤
│ Average Power @ 3nm SSGNP (mW) │ 1152.44 │
├──────────────────────────────────┼─────────┤
│ FPS per Watt @ 3nm SSGNP (FPS/W) │ 156.07 │
├──────────────────────────────────┼─────────┤
│ Ext Rd Bytes (MB) │ 13.07 │
├──────────────────────────────────┼─────────┤
│ Ext Wr Bytes (MB) │ 0.35 │
├──────────────────────────────────┼─────────┤
│ Avg Ext Rd BW (GBps) │ 2.30 │
├──────────────────────────────────┼─────────┤
│ Avg Ext Wr BW (GBps) │ 0.06 │
├──────────────────────────────────┼─────────┤
│ MAC Utilization │ 2.27% │
╘══════════════════════════════════╧═════════╛
*** Data generated using 7nm SSGNP gatesim and scaled to 3nm
[SDK-CLI] : TotalCycles: 8,673,111
[SDK-CLI] : Executions/second: 179.87
compute : ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 1.849M
data_array : ▇▇ 253.084K
mac : ▇▇▇▇ 489.686K
data_external: ▏ 24.939K
data_ocm : ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 6.007M
for more information check run directory: /quadric/sdk-cli/examples/models/detr/encoder/ccl_build/detr_3_transformer_encoder_opt_sym_int8_q_QC_U_1d56_8MB_4kB_64GBps_64GBps_16_OFF_x1_x1/build
[{'S': 'PASS', 'Type': 'Node', 'Name': 'pos_embed_QuantizeLinear:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 0, 'Mismatch %': 0.0, 'RMSE': 0.0, 'PSNR (dB)': inf, 'Max Abs Err': 0.0, 'Max Abs Err Loc': (0, 0, 0), 'ORT[loc]': 41, 'ISS[loc]': 41, 'Max Abs Err / ε': 0.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Node', 'Name': 'src_QuantizeLinear:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 0, 'Mismatch %': 0.0, 'RMSE': 0.0, 'PSNR (dB)': inf, 'Max Abs Err': 0.0, 'Max Abs Err Loc': (0, 0, 0), 'ORT[loc]': 39, 'ISS[loc]': 39, 'Max Abs Err / ε': 0.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.0/self_attn/Add_quant:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 0, 'Mismatch %': 0.0, 'RMSE': 0.0, 'PSNR (dB)': inf, 'Max Abs Err': 0.0, 'Max Abs Err Loc': (0, 0, 0), 'ORT[loc]': 52, 'ISS[loc]': 52, 'Max Abs Err / ε': 0.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.0/self_attn/MatMul_1_quant:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 34661, 'Mismatch %': 37.609592013888886, 'RMSE': 85.67073760694521, 'PSNR (dB)': 9.474153486666118, 'Max Abs Err': 255.0, 'Max Abs Err Loc': (3, 22, 16), 'ORT[loc]': 127, 'ISS[loc]': -128, 'Max Abs Err / ε': 255.0, 'Mismatches Above Tol': 31544, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.0/Add_output_0_DequantizeLinear:0', 'dtype': 'custom[qfp.30]32', 'total': 92160, 'Bitwise Mismatches': 3696, 'Mismatch %': 4.010416666666667, 'RMSE': 0.004580713726255634, 'PSNR (dB)': 51.67161233475455, 'Max Abs Err': 0.11552346125245094, 'Max Abs Err Loc': (0, 192, 248), 'ORT[loc]': 0.046209384, 'ISS[loc]': -0.06931408, 'Max Abs Err / ε': 124042372.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.0/self_attn_layer_norm/Add_1:0', 'dtype': 'custom[qfp.27]32', 'total': 92160, 'Bitwise Mismatches': 92077, 'Mismatch %': 99.90993923611111, 'RMSE': 0.014819250183709077, 'PSNR (dB)': 56.398497127849296, 'Max Abs Err': 0.43483901023864746, 'Max Abs Err Loc': (0, 100, 0), 'ORT[loc]': 3.1678429, 'ISS[loc]': 2.7330039, 'Max Abs Err / ε': 58363104.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.0/self_attn_layer_norm/Add_1_output_0_QuantizeLinear:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 3469, 'Mismatch %': 3.7641059027777777, 'RMSE': 0.2789450301781896, 'PSNR (dB)': 59.22043104445329, 'Max Abs Err': 7.0, 'Max Abs Err Loc': (0, 1, 0), 'ORT[loc]': 30, 'ISS[loc]': 23, 'Max Abs Err / ε': 7.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.0/fc1/MatMul_quant:0', 'dtype': 'int8', 'total': 737280, 'Bitwise Mismatches': 139090, 'Mismatch %': 18.86528862847222, 'RMSE': 0.44060006578386934, 'PSNR (dB)': 55.24991245553578, 'Max Abs Err': 3.0, 'Max Abs Err Loc': (0, 1, 206), 'ORT[loc]': 3, 'ISS[loc]': 6, 'Max Abs Err / ε': 3.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'SKIPPED', 'Type': 'Node', 'Name': '/encoder/layers.0/activation_fn/Relu:0', 'dtype': 'int8', 'Mismatches Above Tol': None, 'total': 0, 'rtol': 0.1, 'atol': -1, 'Comment': 'Skipping validation for /encoder/layers.0/activation_fn/Relu:0 dtype mismatch between ISS int8 and ORT float32'}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.0/fc2/MatMul_quant:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 25048, 'Mismatch %': 27.178819444444443, 'RMSE': 0.5616699122062511, 'PSNR (dB)': 53.141180408449486, 'Max Abs Err': 7.0, 'Max Abs Err Loc': (0, 283, 231), 'ORT[loc]': -114, 'ISS[loc]': -121, 'Max Abs Err / ε': 7.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.0/Add_1_output_0_DequantizeLinear:0', 'dtype': 'custom[qfp.27]32', 'total': 92160, 'Bitwise Mismatches': 19462, 'Mismatch %': 21.11762152777778, 'RMSE': 0.0549627875338843, 'PSNR (dB)': 53.52028186362741, 'Max Abs Err': 0.5499258041381836, 'Max Abs Err Loc': (0, 283, 231), 'ORT[loc]': -12.098379, 'ISS[loc]': -12.648305, 'Max Abs Err / ε': 73809792.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.0/final_layer_norm/Add_1:0', 'dtype': 'custom[qfp.28]32', 'total': 92160, 'Bitwise Mismatches': 92041, 'Mismatch %': 99.87087673611111, 'RMSE': 0.03521079304794415, 'PSNR (dB)': 49.2585786006678, 'Max Abs Err': 0.32262861728668213, 'Max Abs Err Loc': (0, 78, 248), 'ORT[loc]': 0.66234875, 'ISS[loc]': 0.33972013, 'Max Abs Err / ε': 86604960.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.0/final_layer_norm/Add_1_output_0_QuantizeLinear:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 23057, 'Mismatch %': 25.018446180555557, 'RMSE': 0.7005392268341105, 'PSNR (dB)': 51.22215443449817, 'Max Abs Err': 6.0, 'Max Abs Err Loc': (0, 78, 248), 'ORT[loc]': 12, 'ISS[loc]': 6, 'Max Abs Err / ε': 6.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.1/self_attn/Add_quant:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 22998, 'Mismatch %': 24.954427083333332, 'RMSE': 0.6985771128793792, 'PSNR (dB)': 51.24651654929458, 'Max Abs Err': 6.0, 'Max Abs Err Loc': (0, 78, 248), 'ORT[loc]': 25, 'ISS[loc]': 19, 'Max Abs Err / ε': 6.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.1/self_attn/MatMul_1_quant:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 90191, 'Mismatch %': 97.86349826388889, 'RMSE': 19.03847814752038, 'PSNR (dB)': 22.538159011900618, 'Max Abs Err': 123.0, 'Max Abs Err Loc': (4, 64, 4), 'ORT[loc]': -106, 'ISS[loc]': 17, 'Max Abs Err / ε': 123.0, 'Mismatches Above Tol': 51007, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.1/Add_output_0_DequantizeLinear:0', 'dtype': 'custom[qfp.28]32', 'total': 92160, 'Bitwise Mismatches': 45248, 'Mismatch %': 49.09722222222222, 'RMSE': 0.05157025665542468, 'PSNR (dB)': 46.49865282454412, 'Max Abs Err': 0.33023110032081604, 'Max Abs Err Loc': (0, 78, 248), 'ORT[loc]': 0.6054237, 'ISS[loc]': 0.2751926, 'Max Abs Err / ε': 88645736.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.1/self_attn_layer_norm/Add_1:0', 'dtype': 'custom[qfp.27]32', 'total': 92160, 'Bitwise Mismatches': 92160, 'Mismatch %': 100.0, 'RMSE': 0.04944099377147563, 'PSNR (dB)': 51.01199336322048, 'Max Abs Err': 0.37643247842788696, 'Max Abs Err Loc': (0, 78, 248), 'ORT[loc]': 1.1311849, 'ISS[loc]': 0.75475246, 'Max Abs Err / ε': 50523912.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.1/self_attn_layer_norm/Add_1_output_0_QuantizeLinear:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 31389, 'Mismatch %': 34.059244791666664, 'RMSE': 0.6546939223365713, 'PSNR (dB)': 51.81003742283246, 'Max Abs Err': 5.0, 'Max Abs Err Loc': (0, 78, 248), 'ORT[loc]': 13, 'ISS[loc]': 8, 'Max Abs Err / ε': 5.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.1/fc1/MatMul_quant:0', 'dtype': 'int8', 'total': 737280, 'Bitwise Mismatches': 469112, 'Mismatch %': 63.62738715277778, 'RMSE': 1.118573101509302, 'PSNR (dB)': 47.15751617717956, 'Max Abs Err': 6.0, 'Max Abs Err Loc': (0, 64, 1395), 'ORT[loc]': -63, 'ISS[loc]': -57, 'Max Abs Err / ε': 6.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'SKIPPED', 'Type': 'Node', 'Name': '/encoder/layers.1/activation_fn/Relu:0', 'dtype': 'int8', 'Mismatches Above Tol': None, 'total': 0, 'rtol': 0.1, 'atol': -1, 'Comment': 'Skipping validation for /encoder/layers.1/activation_fn/Relu:0 dtype mismatch between ISS int8 and ORT float32'}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.1/fc2/MatMul_quant:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 9854, 'Mismatch %': 10.692274305555555, 'RMSE': 0.32816633337548673, 'PSNR (dB)': 57.808923115118944, 'Max Abs Err': 4.0, 'Max Abs Err Loc': (0, 272, 142), 'ORT[loc]': -23, 'ISS[loc]': -19, 'Max Abs Err / ε': 4.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.1/Add_1_output_0_DequantizeLinear:0', 'dtype': 'custom[qfp.25]32', 'total': 92160, 'Bitwise Mismatches': 19100, 'Mismatch %': 20.72482638888889, 'RMSE': 0.12823851614062154, 'PSNR (dB)': 46.97257974156906, 'Max Abs Err': 1.0799450874328613, 'Max Abs Err Loc': (0, 272, 142), 'ORT[loc]': -7.559614, 'ISS[loc]': -6.479669, 'Max Abs Err / ε': 36236944.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.1/final_layer_norm/Add_1:0', 'dtype': 'custom[qfp.28]32', 'total': 92160, 'Bitwise Mismatches': 92160, 'Mismatch %': 100.0, 'RMSE': 0.07814142008540732, 'PSNR (dB)': 42.00204082115604, 'Max Abs Err': 0.48673081398010254, 'Max Abs Err Loc': (0, 116, 18), 'ORT[loc]': -0.10809779, 'ISS[loc]': -0.5948286, 'Max Abs Err / ε': 130655808.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.1/final_layer_norm/Add_1_output_0_QuantizeLinear:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 26204, 'Mismatch %': 28.43315972222222, 'RMSE': 1.4639586446325745, 'PSNR (dB)': 44.820227438460456, 'Max Abs Err': 9.0, 'Max Abs Err Loc': (0, 54, 166), 'ORT[loc]': 4, 'ISS[loc]': 13, 'Max Abs Err / ε': 9.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.2/self_attn/Add_quant:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 26175, 'Mismatch %': 28.401692708333332, 'RMSE': 1.4587685362527911, 'PSNR (dB)': 44.85107585720912, 'Max Abs Err': 9.0, 'Max Abs Err Loc': (0, 54, 166), 'ORT[loc]': 6, 'ISS[loc]': 15, 'Max Abs Err / ε': 9.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.2/self_attn/MatMul_1_quant:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 89752, 'Mismatch %': 97.38715277777777, 'RMSE': 18.106693341137692, 'PSNR (dB)': 22.974020681709927, 'Max Abs Err': 115.0, 'Max Abs Err Loc': (1, 267, 17), 'ORT[loc]': -60, 'ISS[loc]': 55, 'Max Abs Err / ε': 115.0, 'Mismatches Above Tol': 60994, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.2/Add_output_0_DequantizeLinear:0', 'dtype': 'custom[qfp.28]32', 'total': 92160, 'Bitwise Mismatches': 46413, 'Mismatch %': 50.361328125, 'RMSE': 0.08659606748385876, 'PSNR (dB)': 41.29280656859659, 'Max Abs Err': 0.5582978874444962, 'Max Abs Err Loc': (0, 266, 30), 'ORT[loc]': -0.39080852, 'ISS[loc]': 0.16748936, 'Max Abs Err / ε': 149866948.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.2/self_attn_layer_norm/Add_1:0', 'dtype': 'custom[qfp.27]32', 'total': 92160, 'Bitwise Mismatches': 92160, 'Mismatch %': 100.0, 'RMSE': 0.11213530029100563, 'PSNR (dB)': 44.6761871268567, 'Max Abs Err': 0.8793799877166748, 'Max Abs Err Loc': (0, 54, 166), 'ORT[loc]': 0.74862397, 'ISS[loc]': 1.628004, 'Max Abs Err / ε': 118028384.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.2/self_attn_layer_norm/Add_1_output_0_QuantizeLinear:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 41957, 'Mismatch %': 45.52625868055556, 'RMSE': 1.2827311899051788, 'PSNR (dB)': 45.968090511926505, 'Max Abs Err': 10.0, 'Max Abs Err Loc': (0, 54, 166), 'ORT[loc]': 8, 'ISS[loc]': 18, 'Max Abs Err / ε': 10.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.2/fc1/MatMul_quant:0', 'dtype': 'int8', 'total': 737280, 'Bitwise Mismatches': 584244, 'Mismatch %': 79.2431640625, 'RMSE': 1.9705256129789965, 'PSNR (dB)': 42.23916192433557, 'Max Abs Err': 11.0, 'Max Abs Err Loc': (0, 296, 1322), 'ORT[loc]': -22, 'ISS[loc]': -33, 'Max Abs Err / ε': 11.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'SKIPPED', 'Type': 'Node', 'Name': '/encoder/layers.2/activation_fn/Relu:0', 'dtype': 'int8', 'Mismatches Above Tol': None, 'total': 0, 'rtol': 0.1, 'atol': -1, 'Comment': 'Skipping validation for /encoder/layers.2/activation_fn/Relu:0 dtype mismatch between ISS int8 and ORT float32'}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.2/fc2/MatMul_quant:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 19154, 'Mismatch %': 20.78342013888889, 'RMSE': 0.4715541127484734, 'PSNR (dB)': 54.6601728689258, 'Max Abs Err': 8.0, 'Max Abs Err Loc': (0, 190, 142), 'ORT[loc]': -67, 'ISS[loc]': -59, 'Max Abs Err / ε': 8.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.2/Add_1_output_0_DequantizeLinear:0', 'dtype': 'custom[qfp.25]32', 'total': 92160, 'Bitwise Mismatches': 26406, 'Mismatch %': 28.65234375, 'RMSE': 0.28474373720663104, 'PSNR (dB)': 45.136832289638306, 'Max Abs Err': 3.4622650146484375, 'Max Abs Err Loc': (0, 190, 142), 'ORT[loc]': -33.633446, 'ISS[loc]': -30.17118, 'Max Abs Err / ε': 116174336.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.2/final_layer_norm/Add_1:0', 'dtype': 'custom[qfp.28]32', 'total': 92160, 'Bitwise Mismatches': 92160, 'Mismatch %': 100.0, 'RMSE': 0.12854774592429494, 'PSNR (dB)': 36.78419777323971, 'Max Abs Err': 0.8643730878829956, 'Max Abs Err Loc': (0, 354, 166), 'ORT[loc]': -0.81512535, 'ISS[loc]': -1.6794984, 'Max Abs Err / ε': 232028384.0, 'Mismatches Above Tol': 204, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.2/final_layer_norm/Add_1_output_0_QuantizeLinear:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 45189, 'Mismatch %': 49.033203125, 'RMSE': 3.5802872833405424, 'PSNR (dB)': 37.05244608941171, 'Max Abs Err': 24.0, 'Max Abs Err Loc': (0, 251, 42), 'ORT[loc]': -39, 'ISS[loc]': -15, 'Max Abs Err / ε': 24.0, 'Mismatches Above Tol': 305, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.3/self_attn/Add_quant:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 43195, 'Mismatch %': 46.86957465277778, 'RMSE': 3.214320711890599, 'PSNR (dB)': 37.989019474036944, 'Max Abs Err': 22.0, 'Max Abs Err Loc': (0, 354, 166), 'ORT[loc]': -15, 'ISS[loc]': -37, 'Max Abs Err / ε': 22.0, 'Mismatches Above Tol': 258, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.3/self_attn/MatMul_1_quant:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 71844, 'Mismatch %': 77.95572916666667, 'RMSE': 14.265052052812301, 'PSNR (dB)': 25.045336394168892, 'Max Abs Err': 76.0, 'Max Abs Err Loc': (6, 252, 29), 'ORT[loc]': -30, 'ISS[loc]': 46, 'Max Abs Err / ε': 76.0, 'Mismatches Above Tol': 50606, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.3/Add_output_0_DequantizeLinear:0', 'dtype': 'custom[qfp.28]32', 'total': 92160, 'Bitwise Mismatches': 68850, 'Mismatch %': 74.70703125, 'RMSE': 0.13746588064821816, 'PSNR (dB)': 35.914631783260475, 'Max Abs Err': 0.9335523247718811, 'Max Abs Err Loc': (0, 119, 59), 'ORT[loc]': 0.0, 'ISS[loc]': 0.9335523, 'Max Abs Err / ε': 250598544.0, 'Mismatches Above Tol': 270, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.3/self_attn_layer_norm/Add_1:0', 'dtype': 'custom[qfp.27]32', 'total': 92160, 'Bitwise Mismatches': 92160, 'Mismatch %': 100.0, 'RMSE': 0.18566086731666337, 'PSNR (dB)': 41.119249913544024, 'Max Abs Err': 1.595231831073761, 'Max Abs Err Loc': (0, 251, 42), 'ORT[loc]': -1.9033345, 'ISS[loc]': -0.30810267, 'Max Abs Err / ε': 214108392.0, 'Mismatches Above Tol': 9, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.3/self_attn_layer_norm/Add_1_output_0_QuantizeLinear:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 61395, 'Mismatch %': 66.61783854166667, 'RMSE': 2.9331380143305754, 'PSNR (dB)': 38.78415363817816, 'Max Abs Err': 25.0, 'Max Abs Err Loc': (0, 251, 42), 'ORT[loc]': -30, 'ISS[loc]': -5, 'Max Abs Err / ε': 25.0, 'Mismatches Above Tol': 100, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.3/fc1/MatMul_quant:0', 'dtype': 'int8', 'total': 737280, 'Bitwise Mismatches': 639166, 'Mismatch %': 86.69243706597223, 'RMSE': 3.2597934375061763, 'PSNR (dB)': 37.86700198642129, 'Max Abs Err': 22.0, 'Max Abs Err Loc': (0, 199, 749), 'ORT[loc]': -4, 'ISS[loc]': 18, 'Max Abs Err / ε': 22.0, 'Mismatches Above Tol': 177, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'SKIPPED', 'Type': 'Node', 'Name': '/encoder/layers.3/activation_fn/Relu:0', 'dtype': 'int8', 'Mismatches Above Tol': None, 'total': 0, 'rtol': 0.1, 'atol': -1, 'Comment': 'Skipping validation for /encoder/layers.3/activation_fn/Relu:0 dtype mismatch between ISS int8 and ORT float32'}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.3/fc2/MatMul_quant:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 23760, 'Mismatch %': 25.78125, 'RMSE': 0.673145600891813, 'PSNR (dB)': 51.56862336952841, 'Max Abs Err': 19.0, 'Max Abs Err Loc': (0, 75, 93), 'ORT[loc]': -80, 'ISS[loc]': -61, 'Max Abs Err / ε': 19.0, 'Mismatches Above Tol': 2, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.3/Add_1_output_0_DequantizeLinear:0', 'dtype': 'custom[qfp.24]32', 'total': 92160, 'Bitwise Mismatches': 33512, 'Mismatch %': 36.36284722222222, 'RMSE': 0.4604092962025834, 'PSNR (dB)': 50.3472459649721, 'Max Abs Err': 10.738540649414062, 'Max Abs Err Loc': (0, 75, 93), 'ORT[loc]': -51.902935, 'ISS[loc]': -41.164394, 'Max Abs Err / ε': 180162816.0, 'Mismatches Above Tol': 1, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.3/final_layer_norm/Add_1:0', 'dtype': 'custom[qfp.28]32', 'total': 92160, 'Bitwise Mismatches': 92139, 'Mismatch %': 99.97721354166667, 'RMSE': 0.16570351073707806, 'PSNR (dB)': 36.22732437771462, 'Max Abs Err': 1.366965651512146, 'Max Abs Err Loc': (0, 119, 59), 'ORT[loc]': -0.24591827, 'ISS[loc]': 1.1210474, 'Max Abs Err / ε': 366942048.0, 'Mismatches Above Tol': 496, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.3/final_layer_norm/Add_1_output_0_QuantizeLinear:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 51098, 'Mismatch %': 55.44487847222222, 'RMSE': 3.9283334106738783, 'PSNR (dB)': 36.24663679485877, 'Max Abs Err': 32.0, 'Max Abs Err Loc': (0, 119, 59), 'ORT[loc]': -6, 'ISS[loc]': 26, 'Max Abs Err / ε': 32.0, 'Mismatches Above Tol': 585, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.4/self_attn/Add_quant:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 49877, 'Mismatch %': 54.12000868055556, 'RMSE': 3.6853586649854493, 'PSNR (dB)': 36.80120839900251, 'Max Abs Err': 30.0, 'Max Abs Err Loc': (0, 119, 59), 'ORT[loc]': 5, 'ISS[loc]': 35, 'Max Abs Err / ε': 30.0, 'Mismatches Above Tol': 437, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.4/self_attn/MatMul_1_quant:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 89065, 'Mismatch %': 96.64171006944444, 'RMSE': 14.11230339827135, 'PSNR (dB)': 25.138845514292317, 'Max Abs Err': 102.0, 'Max Abs Err Loc': (6, 314, 4), 'ORT[loc]': -7, 'ISS[loc]': 95, 'Max Abs Err / ε': 102.0, 'Mismatches Above Tol': 38656, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.4/Add_output_0_DequantizeLinear:0', 'dtype': 'custom[qfp.28]32', 'total': 92160, 'Bitwise Mismatches': 71956, 'Mismatch %': 78.07725694444444, 'RMSE': 0.1754354345385888, 'PSNR (dB)': 36.071850725656326, 'Max Abs Err': 1.3568485379219055, 'Max Abs Err Loc': (0, 119, 59), 'ORT[loc]': -0.26261586, 'ISS[loc]': 1.0942327, 'Max Abs Err / ε': 364226256.0, 'Mismatches Above Tol': 608, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.4/self_attn_layer_norm/Add_1:0', 'dtype': 'custom[qfp.26]32', 'total': 92160, 'Bitwise Mismatches': 92159, 'Mismatch %': 99.99891493055556, 'RMSE': 0.22084197195107358, 'PSNR (dB)': 43.3215591558755, 'Max Abs Err': 2.388315200805664, 'Max Abs Err Loc': (0, 123, 93), 'ORT[loc]': -12.805854, 'ISS[loc]': -10.417539, 'Max Abs Err / ε': 160277120.0, 'Mismatches Above Tol': 3, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.4/self_attn_layer_norm/Add_1_output_0_QuantizeLinear:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 54762, 'Mismatch %': 59.420572916666664, 'RMSE': 2.025193963618739, 'PSNR (dB)': 42.0014711238851, 'Max Abs Err': 21.0, 'Max Abs Err Loc': (0, 123, 93), 'ORT[loc]': -115, 'ISS[loc]': -94, 'Max Abs Err / ε': 21.0, 'Mismatches Above Tol': 12, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.4/fc1/MatMul_quant:0', 'dtype': 'int8', 'total': 737280, 'Bitwise Mismatches': 631485, 'Mismatch %': 85.650634765625, 'RMSE': 3.5505367249412094, 'PSNR (dB)': 37.12492342610632, 'Max Abs Err': 58.0, 'Max Abs Err Loc': (0, 186, 1442), 'ORT[loc]': -35, 'ISS[loc]': -93, 'Max Abs Err / ε': 58.0, 'Mismatches Above Tol': 791, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'SKIPPED', 'Type': 'Node', 'Name': '/encoder/layers.4/activation_fn/Relu:0', 'dtype': 'int8', 'Mismatches Above Tol': None, 'total': 0, 'rtol': 0.1, 'atol': -1, 'Comment': 'Skipping validation for /encoder/layers.4/activation_fn/Relu:0 dtype mismatch between ISS int8 and ORT float32'}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.4/fc2/MatMul_quant:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 45512, 'Mismatch %': 49.38368055555556, 'RMSE': 1.210410391549999, 'PSNR (dB)': 46.472150738425235, 'Max Abs Err': 27.0, 'Max Abs Err Loc': (0, 4, 93), 'ORT[loc]': -52, 'ISS[loc]': -25, 'Max Abs Err / ε': 27.0, 'Mismatches Above Tol': 42, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.4/Add_1_output_0_DequantizeLinear:0', 'dtype': 'custom[qfp.26]32', 'total': 92160, 'Bitwise Mismatches': 51648, 'Mismatch %': 56.041666666666664, 'RMSE': 0.2914244949079477, 'PSNR (dB)': 45.66083673484225, 'Max Abs Err': 4.621495246887207, 'Max Abs Err Loc': (0, 4, 47), 'ORT[loc]': -17.792757, 'ISS[loc]': -13.171262, 'Max Abs Err / ε': 310143296.0, 'Mismatches Above Tol': 2, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.4/final_layer_norm/Add_1:0', 'dtype': 'custom[qfp.28]32', 'total': 92160, 'Bitwise Mismatches': 92160, 'Mismatch %': 100.0, 'RMSE': 0.1339793880076684, 'PSNR (dB)': 37.904714899515646, 'Max Abs Err': 1.2145169973373413, 'Max Abs Err Loc': (0, 57, 42), 'ORT[loc]': 1.2169237, 'ISS[loc]': 0.0024067163, 'Max Abs Err / ε': 326019424.0, 'Mismatches Above Tol': 150, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.4/final_layer_norm/Add_1_output_0_QuantizeLinear:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 54909, 'Mismatch %': 59.580078125, 'RMSE': 3.108499841943234, 'PSNR (dB)': 38.27978661624388, 'Max Abs Err': 28.0, 'Max Abs Err Loc': (0, 57, 42), 'ORT[loc]': 28, 'ISS[loc]': 0, 'Max Abs Err / ε': 28.0, 'Mismatches Above Tol': 198, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.5/self_attn/Add_quant:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 53912, 'Mismatch %': 58.498263888888886, 'RMSE': 2.856750655124727, 'PSNR (dB)': 39.013356895733146, 'Max Abs Err': 26.0, 'Max Abs Err Loc': (0, 57, 42), 'ORT[loc]': 34, 'ISS[loc]': 8, 'Max Abs Err / ε': 26.0, 'Mismatches Above Tol': 157, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.5/self_attn/MatMul_1_quant:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 81010, 'Mismatch %': 87.90147569444444, 'RMSE': 14.660412373880286, 'PSNR (dB)': 24.807879878999998, 'Max Abs Err': 81.0, 'Max Abs Err Loc': (7, 210, 2), 'ORT[loc]': -26, 'ISS[loc]': 55, 'Max Abs Err / ε': 81.0, 'Mismatches Above Tol': 49702, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.5/Add_output_0_DequantizeLinear:0', 'dtype': 'custom[qfp.28]32', 'total': 92160, 'Bitwise Mismatches': 67223, 'Mismatch %': 72.94162326388889, 'RMSE': 0.13737290142341269, 'PSNR (dB)': 37.579259299609845, 'Max Abs Err': 1.1598782949149609, 'Max Abs Err Loc': (0, 57, 42), 'ORT[loc]': 1.2028368, 'ISS[loc]': 0.042958457, 'Max Abs Err / ε': 311352459.0, 'Mismatches Above Tol': 189, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.5/self_attn_layer_norm/Add_1:0', 'dtype': 'custom[qfp.27]32', 'total': 92160, 'Bitwise Mismatches': 92160, 'Mismatch %': 100.0, 'RMSE': 0.2035032855799216, 'PSNR (dB)': 41.610463565643926, 'Max Abs Err': 2.3625173568725586, 'Max Abs Err Loc': (0, 123, 93), 'ORT[loc]': -12.057574, 'ISS[loc]': -9.695057, 'Max Abs Err / ε': 317091712.0, 'Mismatches Above Tol': 39, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.5/self_attn_layer_norm/Add_1_output_0_QuantizeLinear:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 58061, 'Mismatch %': 63.000217013888886, 'RMSE': 2.101555723347191, 'PSNR (dB)': 41.67998540996461, 'Max Abs Err': 24.0, 'Max Abs Err Loc': (0, 123, 93), 'ORT[loc]': -122, 'ISS[loc]': -98, 'Max Abs Err / ε': 24.0, 'Mismatches Above Tol': 61, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.5/fc1/MatMul_quant:0', 'dtype': 'int8', 'total': 737280, 'Bitwise Mismatches': 614098, 'Mismatch %': 83.29237196180556, 'RMSE': 2.9118950458844717, 'PSNR (dB)': 38.847289257772985, 'Max Abs Err': 26.0, 'Max Abs Err Loc': (0, 186, 220), 'ORT[loc]': -70, 'ISS[loc]': -44, 'Max Abs Err / ε': 26.0, 'Mismatches Above Tol': 190, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'SKIPPED', 'Type': 'Node', 'Name': '/encoder/layers.5/activation_fn/Relu:0', 'dtype': 'int8', 'Mismatches Above Tol': None, 'total': 0, 'rtol': 0.1, 'atol': -1, 'Comment': 'Skipping validation for /encoder/layers.5/activation_fn/Relu:0 dtype mismatch between ISS int8 and ORT float32'}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.5/fc2/MatMul_quant:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 34357, 'Mismatch %': 37.27973090277778, 'RMSE': 0.7911936635201551, 'PSNR (dB)': 50.16514759982008, 'Max Abs Err': 11.0, 'Max Abs Err Loc': (0, 103, 47), 'ORT[loc]': -26, 'ISS[loc]': -15, 'Max Abs Err / ε': 11.0, 'Mismatches Above Tol': 128, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.5/Add_1_output_0_DequantizeLinear:0', 'dtype': 'custom[qfp.26]32', 'total': 92160, 'Bitwise Mismatches': 54222, 'Mismatch %': 58.834635416666664, 'RMSE': 0.2212340857041729, 'PSNR (dB)': 41.87639006798035, 'Max Abs Err': 2.4439210891723633, 'Max Abs Err Loc': (0, 123, 93), 'ORT[loc]': -12.219604, 'ISS[loc]': -9.775682, 'Max Abs Err / ε': 164008768.0, 'Mismatches Above Tol': 19, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.5/final_layer_norm/Add_1:0', 'dtype': 'custom[qfp.28]32', 'total': 92160, 'Bitwise Mismatches': 92160, 'Mismatch %': 100.0, 'RMSE': 0.11971544492652443, 'PSNR (dB)': 35.675701121261085, 'Max Abs Err': 1.2196358442306519, 'Max Abs Err Loc': (0, 19, 214), 'ORT[loc]': -2.7607398, 'ISS[loc]': -1.541104, 'Max Abs Err / ε': 327393504.0, 'Mismatches Above Tol': 761, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Output', 'Name': 'encoder_memory', 'dtype': 'custom[qfp.28]32', 'total': 92160, 'Bitwise Mismatches': 92160, 'Mismatch %': 100.0, 'RMSE': 0.11971544492652443, 'PSNR (dB)': 35.675701121261085, 'Max Abs Err': 1.2196358442306519, 'Max Abs Err Loc': (0, 19, 214), 'ORT[loc]': -2.7607398, 'ISS[loc]': -1.541104, 'Max Abs Err / ε': 327393504.0, 'Mismatches Above Tol': 761, 'rtol': 0.1, 'atol': -1, 'Comment': ''}]
5. Single-Layer Extraction (optional)
Extract, quantize, compile, and validate a single encoder layer in isolation. Change LAYER to target any of the 6 layers (0–5). This reuses the hw_config and imports from the cells above — run sections 1–3 first.
| Layer | Input edges | Output edge |
|---|---|---|
| 0 | src, pos_embed | /encoder/layers.0/final_layer_norm/Add_1_output_0 |
| 1–4 | prev layer output, pos_embed | /encoder/layers.{N}/final_layer_norm/Add_1_output_0 |
| 5 | prev layer output, pos_embed | encoder_memory (graph output) |
LAYER = 0
if LAYER == 0:
in_edges = ["src", "pos_embed"]
else:
in_edges = [f"/encoder/layers.{LAYER - 1}/final_layer_norm/Add_1_output_0", "pos_embed"]
out_edges = (
["encoder_memory"]
if LAYER == 5
else [f"/encoder/layers.{LAYER}/final_layer_norm/Add_1_output_0"]
)
layer_onnx = f"{OUTPUT_DIR}/encoder_layer_{LAYER}.onnx"
onnx.save(
cut_onnx(onnx.load(ENCODER_ONNX), cut_before=in_edges, cut_after=out_edges),
layer_onnx,
)
layer_q = quadric_quantize(layer_onnx, num_images=1, synthetic_input=True, output_folder=OUTPUT_DIR)
layer_job = ChimeraJob(
layer_q.qmodel_path,
hw_config=hw_config,
trange_file=layer_q.tranges_path,
target_lang="ASM",
validate_iss=True,
)
layer_job.compile()
print(f"Layer {LAYER} compiled!")
print(layer_job)
2026-07-18 12:14 - INFO - epu - quantize - Generating synthetic data
2026-07-18 12:14 - INFO - epu - quantize - Optimized model to opset
2026-07-18 12:14 - INFO - epu - quantize - Saved optimized model to encoder_layer_0_float32_opt.onnx
2026-07-18 12:14 - INFO - epu - quantize - Input shapes: [1, 360, 256]. Input names: src
2026-07-18 12:14 - INFO - epu - quantize - Input shapes: [1, 360, 256]. Input names: pos_embed
2026-07-18 12:14 - INFO - epu - quantize - Output shapes: [[1, 360, 256]]. Output names: ['/encoder/layers.0/final_layer_norm/Add_1_output_0']
2026-07-18 12:14 - INFO - epu - quantize - Quantization started...
WARNING:root:Please use QuantFormat.QDQ for activation type QInt8 and weight type QInt8. Or it will lead to bad performance on x64.
2026-07-18 12:14 - INFO - epu - quantize - Quantization done succesfully!
2026-07-18 12:14 - INFO - epu - quantize - ONNX full precision model size: 5.03 MB
2026-07-18 12:14 - INFO - epu - quantize - ONNX quantized model size: 1.29 MB
2026-07-18 12:14 - INFO - epu - quantize - Saved quantized model to onnx/encoder_layer_0_opt_sym_int8_q.onnx
2026-07-18 12:14 - INFO - epu - quantize - Saved shape inferenced model to onnx/encoder_layer_0_opt_sym_int8_q.onnx
2026-07-18 12:14 - INFO - epu - quantize - Checking for remaining FLOAT/FLOAT16 types.
2026-07-18 12:14 - INFO - epu - quantize - Model still has FLOAT/FLOAT16 types. Creating ranges for floating point tensors using calibration data
2026-07-18 12:14 - INFO - epu - quantize - Saved tensor ranges to onnx/encoder_layer_0_opt_sym_int8_q.onnx.tranges
2026-07-18 12:14 - INFO - epu - chimera_job - START==================================onnx_ingest
2026-07-18 12:14 - INFO - epu - chimera_job - Numerical ranges provided
2026-07-18 12:14 - INFO - epu - codegen - START===============================optimize_relay
2026-07-18 12:14 - INFO - epu - codegen - START====================quantize_to_cpu_runnable_fx
2026-07-18 12:14 - INFO - epu - fx -
Source name Op Output 0 Range Output 0 Frac Bits
------------------------------------------------------------ ---------------------- -------------------------- --------------------
/encoder/layers.0/self_attn/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-1.02516f, 0.961083f] 30
/encoder/layers.0/self_attn/Softmax nn.softmax [0.000955444f, 0.0070394f] 30
/encoder/layers.0/Add_output_0_DequantizeLinear contrib.epu.dequantize [-1.42084f, 1.38377f] 30
/encoder/layers.0/self_attn_layer_norm/Add_1 nn.layer_norm [-8.43854f, 7.77878f] 27
/encoder/layers.0/Add_1_output_0_DequantizeLinear contrib.epu.dequantize [-14.2651f, 14.8497f] 27
/encoder/layers.0/final_layer_norm/Add_1 nn.layer_norm [-6.79641f, 4.98499f] -
2026-07-18 12:14 - INFO - epu - codegen - START====================build_cpu_runnable_fx_relay
2026-07-18 12:14 - INFO - epu - codegen - START=======================quantize_to_chimera_fx
2026-07-18 12:14 - INFO - epu - codegen - START=================================relay_to_tir
2026-07-18 12:14 - INFO - epu - codegen - START===========================relay_to_epu_relay
2026-07-18 12:14 - INFO - epu - codegen - START==============================adapt_and_order
2026-07-18 12:14 - INFO - epu - mac_counter -
2026-07-18 12:14 - INFO - epu - mac_counter - ============================================================
2026-07-18 12:14 - INFO - epu - mac_counter - MAC Operation Count Summary
2026-07-18 12:14 - INFO - epu - mac_counter - ============================================================
2026-07-18 12:14 - INFO - epu - mac_counter - conv2d: 47,185,920 ops (23,592,960 MACs) - /encoder/layers.0/self_attn/out_proj/MatMul_quant
2026-07-18 12:14 - INFO - epu - mac_counter - ------------------------------------------------------------
2026-07-18 12:14 - INFO - epu - mac_counter - Total: 47,185,920 ops (23,592,960 MACs)
2026-07-18 12:14 - INFO - epu - mac_counter - ============================================================
2026-07-18 12:14 - INFO - epu - mac_counter -
2026-07-18 12:14 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-07-18 12:14 - INFO - epu - codegen - START=============================plan_lrm_virtual
2026-07-18 12:14 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-07-18 12:14 - INFO - epu - codegen - START===============================lrm_alloc_loop
2026-07-18 12:14 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-07-18 12:14 - INFO - epu - codegen - START================================lrm_splitting
2026-07-18 12:14 - INFO - epu - codegen - START==============================ext_split_relay
2026-07-18 12:15 - INFO - epu - codegen - START====================================build_tir
2026-07-18 12:15 - INFO - epu - chimera_job - Compilation of encoder_layer_0_opt_sym_int8_q_QC_U_1d56_8MB_4kB_64GBps_64GBps_16_OFF_x1_x1 successful
Layer 0 compiled!
╒═════════════════════╤═════════════════════════════════════════════════════════════════════════════╕
│ Module Name │ encoder_layer_0_opt_sym_int8_q_QC_U_1d56_8MB_4kB_64GBps_64GBps_16_OFF_x1_x1 │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────────┤
│ ONNX File │ onnx/encoder_layer_0_opt_sym_int8_q.onnx │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────────┤
│ Product Target │ QC-U │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────────┤
│ Number of Cores │ 1 │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────────┤
│ ISS Clock Frequency │ 1.560 │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────────┤
│ L2M Size │ 8MB │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────────┤
│ LRM Size │ 4kB │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────────┤
│ External Read BW │ 64GBps │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────────┤
│ External Write BW │ 64GBps │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────────┤
│ MACS per PE │ 16 │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────────┤
│ Max L2M │ 6.188MB │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────────┤
│ Max LRM │ 0.250kB │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────────┤
│ Max Temp Ext Bytes │ 0.000MB │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────────┤
│ Network GMACs │ 0.538 │
╘═════════════════════╧═════════════════════════════════════════════════════════════════════════════╛
╒════╤════════╤═══════════════════════════════════════════════════╤═══════════════╤══════════════════════════╤═══════╕
│ │ Type │ Name │ shape │ type │ mse │
╞════╪════════╪═══════════════════════════════════════════════════╪═══════════════╪══════════════════════════╪═══════╡
│ 0 │ Input │ src │ [1, 360, 256] │ tensor[FixedPoint32<29>] │ n/a │
├────┼────────┼───────────────────────────────────────────────────┼───────────────┼──────────────────────────┼───────┤
│ 1 │ Input │ pos_embed │ [1, 360, 256] │ tensor[FixedPoint32<30>] │ n/a │
├────┼────────┼───────────────────────────────────────────────────┼───────────────┼──────────────────────────┼───────┤
│ 2 │ Output │ /encoder/layers.0/final_layer_norm/Add_1_output_0 │ [1, 360, 256] │ tensor[FixedPoint32<28>] │ n/a │
╘════╧════════╧═══════════════════════════════════════════════════╧═══════════════╧══════════════════════════╧═══════╛
Validate the extracted layer against ORT:
layer_validation = layer_job.validate_ort_iss()
print(layer_validation)
2026-07-18 12:15 - INFO - epu - iss_testing - Found tranges for input: <tvm.contrib.epu.interval.Interval object at 0x74857bfc0460>
2026-07-18 12:15 - INFO - epu - iss_testing - Found tranges for input: <tvm.contrib.epu.interval.Interval object at 0x74857bfc0460>
2026-07-18 12:15 - INFO - epu - iss_testing - Found tranges for input: <tvm.contrib.epu.interval.Interval object at 0x74857a67c070>
2026-07-18 12:15 - INFO - epu - iss_testing - Found tranges for input: <tvm.contrib.epu.interval.Interval object at 0x74857a67c070>
2026-07-18 12:15 - INFO - epu - iss_testing - Started Executing Onnxruntime...
2026-07-18 12:15 - INFO - epu - iss_testing - Done 0:00:00.059077
2026-07-18 12:15 - INFO - epu - iss_testing - Found tranges for input: <tvm.contrib.epu.interval.Interval object at 0x74857bfc0460>
2026-07-18 12:15 - INFO - epu - iss_testing - Found tranges for input: <tvm.contrib.epu.interval.Interval object at 0x74857bfc0460>
FILM 14/14: 100%|███████████████████████████████████████████████████| 14/14 [00:14<00:00, 1.01s/it]
2026-07-18 12:15 - WARNING - epu - iss_testing - Node was skipped due to being multi-output: /encoder/layers.0/self_attn/MatMul_1_quant
2026-07-18 12:15 - INFO - epu - iss_testing -
======================================================================
ISS validation results (quantized model, rtol=0.1, atol=-1)
======================================================================
pos_embed_QuantizeLinear:0 (PASS, Node, int8)
Bitwise Mismatches 0 / 92160 (0.0000%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0
PSNR inf dB
Max Abs Err 0
Max Abs Err / ε 0
Max Err Loc (0, 0, 0)
ORT @ max err 53
ISS @ max err 53
src_QuantizeLinear:0 (PASS, Node, int8)
Bitwise Mismatches 0 / 92160 (0.0000%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0
PSNR inf dB
Max Abs Err 0
Max Abs Err / ε 0
Max Err Loc (0, 0, 0)
ORT @ max err 48
ISS @ max err 48
/encoder/layers.0/self_attn/Add_quant:0 (PASS, Node, int8)
Bitwise Mismatches 0 / 92160 (0.0000%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0
PSNR inf dB
Max Abs Err 0
Max Abs Err / ε 0
Max Err Loc (0, 0, 0)
ORT @ max err 66
ISS @ max err 66
/encoder/layers.0/self_attn/MatMul_1_quant:0 (FAIL, Node, int8)
Bitwise Mismatches 34657 / 92160 (37.6053%)
Mismatches > Tol 31269 / 92160 (33.9290%)
Thresholds rtol=0.1, atol=-1
RMSE 85.8803
PSNR 9.45 dB
Max Abs Err 255
Max Abs Err / ε 255
Max Err Loc (3, 24, 5)
ORT @ max err 127
ISS @ max err -128
/encoder/layers.0/Add_output_0_DequantizeLinear:0 (PASS, Node, custom[qfp.30]32)
Bitwise Mismatches 3720 / 92160 (4.0365%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0.00500025
PSNR 51.08 dB
Max Abs Err 0.123551
Max Abs Err / ε 1.32662e+08
Max Err Loc (0, 348, 248)
ORT @ max err 0.13590625
ISS @ max err 0.012355113
/encoder/layers.0/self_attn_layer_norm/Add_1:0 (PASS, Node, custom[qfp.27]32)
Bitwise Mismatches 92094 / 92160 (99.9284%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0.0164353
PSNR 55.47 dB
Max Abs Err 0.465487
Max Abs Err / ε 6.24766e+07
Max Err Loc (0, 304, 0)
ORT @ max err 1.5103265
ISS @ max err 1.0448397
/encoder/layers.0/self_attn_layer_norm/Add_1_output_0_QuantizeLinear:0 (PASS, Node, int8)
Bitwise Mismatches 3490 / 92160 (3.7869%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0.281269
PSNR 59.15 dB
Max Abs Err 7
Max Abs Err / ε 7
Max Err Loc (0, 90, 0)
ORT @ max err 41
ISS @ max err 34
/encoder/layers.0/fc1/MatMul_quant:0 (PASS, Node, int8)
Bitwise Mismatches 151914 / 737280 (20.6047%)
Mismatches > Tol 0 / 737280 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0.464587
PSNR 54.79 dB
Max Abs Err 3
Max Abs Err / ε 3
Max Err Loc (0, 12, 985)
ORT @ max err -59
ISS @ max err -56
[ERROR] /encoder/layers.0/activation_fn/Relu:0
Error Skipping validation for /encoder/layers.0/activation_fn/Relu:0 dtype mismatch between ISS int8 and ORT float32
/encoder/layers.0/fc2/MatMul_quant:0 (PASS, Node, int8)
Bitwise Mismatches 22061 / 92160 (23.9377%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0.511521
PSNR 53.95 dB
Max Abs Err 5
Max Abs Err / ε 5
Max Err Loc (0, 185, 41)
ORT @ max err 74
ISS @ max err 79
/encoder/layers.0/Add_1_output_0_DequantizeLinear:0 (PASS, Node, custom[qfp.27]32)
Bitwise Mismatches 19725 / 92160 (21.4030%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0.0585281
PSNR 52.82 dB
Max Abs Err 0.584634
Max Abs Err / ε 7.84682e+07
Max Err Loc (0, 241, 231)
ORT @ max err -10.8742
ISS @ max err -11.458834
/encoder/layers.0/final_layer_norm/Add_1:0 (PASS, Node, custom[qfp.28]32)
Bitwise Mismatches 92113 / 92160 (99.9490%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0.0373785
PSNR 49.01 dB
Max Abs Err 0.335832
Max Abs Err / ε 9.01493e+07
Max Err Loc (0, 293, 248)
ORT @ max err 0.3440259
ISS @ max err 0.008193437
/encoder/layers.0/final_layer_norm/Add_1_output_0 (PASS, Output, custom[qfp.28]32)
Bitwise Mismatches 92113 / 92160 (99.9490%)
Mismatches > Tol 0 / 92160 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0.0373785
PSNR 49.01 dB
Max Abs Err 0.335832
Max Abs Err / ε 9.01493e+07
Max Err Loc (0, 293, 248)
ORT @ max err 0.3440259
ISS @ max err 0.008193437
13 entries
Differences detected between ort and iss
[{'S': 'PASS', 'Type': 'Node', 'Name': 'pos_embed_QuantizeLinear:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 0, 'Mismatch %': 0.0, 'RMSE': 0.0, 'PSNR (dB)': inf, 'Max Abs Err': 0.0, 'Max Abs Err Loc': (0, 0, 0), 'ORT[loc]': 53, 'ISS[loc]': 53, 'Max Abs Err / ε': 0.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Node', 'Name': 'src_QuantizeLinear:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 0, 'Mismatch %': 0.0, 'RMSE': 0.0, 'PSNR (dB)': inf, 'Max Abs Err': 0.0, 'Max Abs Err Loc': (0, 0, 0), 'ORT[loc]': 48, 'ISS[loc]': 48, 'Max Abs Err / ε': 0.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.0/self_attn/Add_quant:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 0, 'Mismatch %': 0.0, 'RMSE': 0.0, 'PSNR (dB)': inf, 'Max Abs Err': 0.0, 'Max Abs Err Loc': (0, 0, 0), 'ORT[loc]': 66, 'ISS[loc]': 66, 'Max Abs Err / ε': 0.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'FAIL', 'Type': 'Node', 'Name': '/encoder/layers.0/self_attn/MatMul_1_quant:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 34657, 'Mismatch %': 37.605251736111114, 'RMSE': 85.88026402069272, 'PSNR (dB)': 9.45293619042476, 'Max Abs Err': 255.0, 'Max Abs Err Loc': (3, 24, 5), 'ORT[loc]': 127, 'ISS[loc]': -128, 'Max Abs Err / ε': 255.0, 'Mismatches Above Tol': 31269, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.0/Add_output_0_DequantizeLinear:0', 'dtype': 'custom[qfp.30]32', 'total': 92160, 'Bitwise Mismatches': 3720, 'Mismatch %': 4.036458333333333, 'RMSE': 0.005000253709657544, 'PSNR (dB)': 51.084453831930425, 'Max Abs Err': 0.12355113588273525, 'Max Abs Err Loc': (0, 348, 248), 'ORT[loc]': 0.13590625, 'ISS[loc]': 0.012355113, 'Max Abs Err / ε': 132662022.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.0/self_attn_layer_norm/Add_1:0', 'dtype': 'custom[qfp.27]32', 'total': 92160, 'Bitwise Mismatches': 92094, 'Mismatch %': 99.92838541666667, 'RMSE': 0.016435331942454865, 'PSNR (dB)': 55.46898362089366, 'Max Abs Err': 0.4654867649078369, 'Max Abs Err Loc': (0, 304, 0), 'ORT[loc]': 1.5103265, 'ISS[loc]': 1.0448397, 'Max Abs Err / ε': 62476576.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.0/self_attn_layer_norm/Add_1_output_0_QuantizeLinear:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 3490, 'Mismatch %': 3.786892361111111, 'RMSE': 0.28126928946197527, 'PSNR (dB)': 59.14835728711702, 'Max Abs Err': 7.0, 'Max Abs Err Loc': (0, 90, 0), 'ORT[loc]': 41, 'ISS[loc]': 34, 'Max Abs Err / ε': 7.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.0/fc1/MatMul_quant:0', 'dtype': 'int8', 'total': 737280, 'Bitwise Mismatches': 151914, 'Mismatch %': 20.604654947916668, 'RMSE': 0.464586544738527, 'PSNR (dB)': 54.78947105472965, 'Max Abs Err': 3.0, 'Max Abs Err Loc': (0, 12, 985), 'ORT[loc]': -59, 'ISS[loc]': -56, 'Max Abs Err / ε': 3.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'SKIPPED', 'Type': 'Node', 'Name': '/encoder/layers.0/activation_fn/Relu:0', 'dtype': 'int8', 'Mismatches Above Tol': None, 'total': 0, 'rtol': 0.1, 'atol': -1, 'Comment': 'Skipping validation for /encoder/layers.0/activation_fn/Relu:0 dtype mismatch between ISS int8 and ORT float32'}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.0/fc2/MatMul_quant:0', 'dtype': 'int8', 'total': 92160, 'Bitwise Mismatches': 22061, 'Mismatch %': 23.93771701388889, 'RMSE': 0.5115209143655158, 'PSNR (dB)': 53.95353570371369, 'Max Abs Err': 5.0, 'Max Abs Err Loc': (0, 185, 41), 'ORT[loc]': 74, 'ISS[loc]': 79, 'Max Abs Err / ε': 5.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.0/Add_1_output_0_DequantizeLinear:0', 'dtype': 'custom[qfp.27]32', 'total': 92160, 'Bitwise Mismatches': 19725, 'Mismatch %': 21.402994791666668, 'RMSE': 0.058528109546443996, 'PSNR (dB)': 52.819879363050354, 'Max Abs Err': 0.5846338272094727, 'Max Abs Err Loc': (0, 241, 231), 'ORT[loc]': -10.8742, 'ISS[loc]': -11.458834, 'Max Abs Err / ε': 78468224.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Node', 'Name': '/encoder/layers.0/final_layer_norm/Add_1:0', 'dtype': 'custom[qfp.28]32', 'total': 92160, 'Bitwise Mismatches': 92113, 'Mismatch %': 99.94900173611111, 'RMSE': 0.037378470729770574, 'PSNR (dB)': 49.00896618985715, 'Max Abs Err': 0.33583247289061546, 'Max Abs Err Loc': (0, 293, 248), 'ORT[loc]': 0.3440259, 'ISS[loc]': 0.008193437, 'Max Abs Err / ε': 90149343.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}, {'S': 'PASS', 'Type': 'Output', 'Name': '/encoder/layers.0/final_layer_norm/Add_1_output_0', 'dtype': 'custom[qfp.28]32', 'total': 92160, 'Bitwise Mismatches': 92113, 'Mismatch %': 99.94900173611111, 'RMSE': 0.037378470729770574, 'PSNR (dB)': 49.00896618985715, 'Max Abs Err': 0.33583247289061546, 'Max Abs Err Loc': (0, 293, 248), 'ORT[loc]': 0.3440259, 'ISS[loc]': 0.008193437, 'Max Abs Err / ε': 90149343.0, 'Mismatches Above Tol': 0, 'rtol': 0.1, 'atol': -1, 'Comment': ''}]
Summary
| Model | DETR Transformer Encoder (6 layers, d_model=256, 8 heads, seq_len=360) |
| Target | QC-U, 8 MB OCM, 4 kB LRM, 16 MACs/PE, 1.56 GHz |
| Quantization | Symmetric INT8 (QOperator), Softmax and LayerNorm in float |
| Custom Ops | None — all ops compile natively |
Key takeaways
- The full 6-layer DETR encoder compiles natively on CGC with ~97.9% element match vs ORT.
- CGC maps multi-head attention to
nn::multiheadAttentionHead, tiling 360 tokens on an 18×20 PE grid. - Individual layers can be extracted and validated in isolation using the SDK's
cut_onnxhelper.
'%3e%3crect%20width='512'%20height='512'%20rx='60'%3e%3c/rect%3e%3cpath%20fill='%23fff'%20d='M355.904%20100H408.832L293.2%20232.16L429.232%20412H322.72L239.296%20302.928L143.84%20412H90.8805L214.56%20270.64L84.0645%20100H193.28L268.688%20199.696L355.904%20100ZM337.328%20380.32H366.656L177.344%20130.016H145.872L337.328%20380.32Z'%3e%3c/path%3e%3c/g%3e%3cdefs%3e%3cclipPath%20id='clip0_84_15698'%3e%3crect%20width='512'%20height='512'%20fill='%23fff'%3e%3c/rect%3e%3c/clipPath%3e%3c/defs%3e%3c/svg%3e)
