NOTE: The Jupyter Notebook below is included in the Chimera SDK and can be run interactively by running the following CLI command:
$ quadric sdk notebook
From the Jupyter Notebook window in your browser, select the notebook named /quadric/sdk-cli/examples/quantization/ClassifierTop1Top5Validation.ipynb.
Evaluating Classifier Performance: Top-1 and Top-5 Accuracy
This notebook focuses on Top-1 and Top-5 accuracy metrics for classifier validation in machine learning. Top-1 accuracy measures the percentage of predictions where the classifier's top choice is correct, crucial for scenarios demanding high precision. Top-5 accuracy evaluates whether the correct answer is within the classifier's top five predictions, useful in situations with numerous classes or when approximate correctness is sufficient.
Both metrics provide insights into a classifier's performance: Top-1 for precision and Top-5 for broader relevance. We'll apply these metrics to a dataset of NUM_IMAGES to assess our classifier's effectiveness, offering a balanced view of its capabilities in precise and general prediction scenarios.
In this tutorial, we will show how to compute top1 and top5 for a classifier using the Quadric SDK both as a library and using the command line tools.
Preparation
First let's export a classifier from torchvision and quantize it. This network will be used for the remainder of the tutorial.
from pathlib import Path
import torch
import torchvision
from torch.utils.data import Subset
from torchvision.transforms import CenterCrop, Compose, Normalize, Resize, ToTensor
from tvm.contrib.epu.chimera_job.constants import DEFAULT_ONNX_OPSET
from sdk_cli.lib.quantize import (
QuantizedONNXModel,
quantize_onnx_model,
)
from sdk_cli.utils.datasets import ImageNet_Mini_Quadric
from sdk_cli.utils.datasets.ImageNet import IMAGENET_1K_NORMALIZATION_PARAMETERS
## NOTE: Full list of available torchvision models can be found by running the following:
## print(torchvision.models.list_models())
pytorch_model = torchvision.models.resnet18(weights="DEFAULT")
## Convert PyTorch model into ONNX Format
onnx_model_path = Path(f"{pytorch_model.__class__.__name__}_float32.onnx")
## Create an example input tensor for the model
input_tensor_shape = (1, 3, 224, 224)
example_input = torch.randn(*input_tensor_shape, requires_grad=True)
## Export the model to ONNX format
torch.onnx.export(
pytorch_model,
example_input,
onnx_model_path,
export_params=True, # Store the trained parameter weights inside the model file
do_constant_folding=True, # Execute constant folding for optimization
opset_version=DEFAULT_ONNX_OPSET,
input_names=["input"],
output_names=["output"],
)
## NOTE: Change normalization parameters below if using dataset other than ImageNet
transforms = Compose(
[
Resize(224),
CenterCrop(224),
ToTensor(),
Normalize(
IMAGENET_1K_NORMALIZATION_PARAMETERS.channel_means,
IMAGENET_1K_NORMALIZATION_PARAMETERS.channel_standard_deviations,
),
]
)
## Path to directory containing data to use for for numerical range calibration during quantization
## Data is used also used to compare accuracy of fp32 and int8 models
dataset = ImageNet_Mini_Quadric.Dataset(transform=transforms)
subset_of_dataset = Subset(dataset, range(100))
quantized_onnx_model: QuantizedONNXModel = quantize_onnx_model(
onnx_model_path, subset_of_dataset, asymmetric_activation=True
)
2026-06-19 03:53 - INFO - sdk - quantize - ONNX model shapes inferred.
2026-06-19 03:53 - DEBUG - sdk - quantize - Forcing node types: []
2026-06-19 03:53 - DEBUG - sdk - quantize - ONNX Node types excluded from quantization: ['Softmax', 'Sigmoid', 'QuadricCustomOp']
2026-06-19 03:53 - DEBUG - sdk - quantize - ONNX Node names excluded from quantization: []
2026-06-19 03:53 - INFO - sdk - quantize - Starting quantization...
WARNING:root:Please use QuantFormat.QDQ for activation type QInt8 and weight type QInt8. Or it will lead to bad performance on x64.
2026-06-19 03:53 - INFO - sdk - quantize - Quantization completed! Quantized model saved to /quadric/sdk-cli/examples/quantization/ResNet_OpSet16_optimized_asym_int8_q.onnx
2026-06-19 03:53 - INFO - sdk - quantize - ONNX full precision model size: 44.58MB
2026-06-19 03:53 - INFO - sdk - quantize - ONNX quantized model size: 11.18MB
2026-06-19 03:53 - INFO - sdk - quantize - ONNX model shapes inferred.
2026-06-19 03:53 - INFO - sdk - quantize - ONNX Model with well-defined shapes has been saved at `/quadric/sdk-cli/examples/quantization/ResNet_OpSet16_optimized_asym_int8_q_shaped.onnx`.
2026-06-19 03:53 - DEBUG - sdk - quantize - Checking for FLOAT/FLOAT16 types...
2026-06-19 03:53 - INFO - sdk - quantize - Checking for remaining FLOAT/FLOAT16 types.
2026-06-19 03:53 - INFO - sdk - quantize - Model still has FLOAT/FLOAT16 types after quantization. Creating ranges for floating point tensors using calibration data...
2026-06-19 03:54 - INFO - sdk - quantize - Saved computed tensor ranges to /quadric/sdk-cli/examples/quantization/ResNet_OpSet16_optimized_asym_int8_q_shaped.tranges.
2026-06-19 03:54 - INFO - sdk - quantize -
╒═════════════════════════════════════════════════════════════════════════════════════════╤════════════════════════════════════════════════════════════════════════════════════════════╕
│ Quantized ONNX Model │ Tensor Ranges File │
╞═════════════════════════════════════════════════════════════════════════════════════════╪════════════════════════════════════════════════════════════════════════════════════════════╡
│ /quadric/sdk-cli/examples/quantization/ResNet_OpSet16_optimized_asym_int8_q_shaped.onnx │ /quadric/sdk-cli/examples/quantization/ResNet_OpSet16_optimized_asym_int8_q_shaped.tranges │
╘═════════════════════════════════════════════════════════════════════════════════════════╧════════════════════════════════════════════════════════════════════════════════════════════╛
Using sdk_cli.lib.validation in a python script
First let's look at using the sdk_cli.lib.validation.classification_validation() function to easily compute top1 and top5 for ort and iss given a quantized network, a path containing images, and supporting parameters. This can easily be adapted to work with any dataset similar to Imagenet mini.
from sdk_cli.lib import validation
THREADS = 3
NUM_IMAGES = 24
validation_scores = validation.classification_validation(
onnx_model_p=str(quantized_onnx_model.model_path),
validation_p="../common/validation/imagenet-mini-quadric/",
threads=THREADS,
num_images=NUM_IMAGES,
normalization_mean=IMAGENET_1K_NORMALIZATION_PARAMETERS.channel_means,
normalization_std=IMAGENET_1K_NORMALIZATION_PARAMETERS.channel_standard_deviations,
map_cls_p="../common/validation/map_clsloc.txt",
dict_label_p="../common/validation/imagenet_classes.txt",
)
print(validation_scores)
2026-06-19 03:54 - INFO - epu - chimera_job - START==================================onnx_ingest
2026-06-19 03:54 - INFO - sdk - validation - Extracted singular input: input
2026-06-19 03:54 - INFO - sdk - validation - Extracted singular output: output
2026-06-19 03:54 - INFO - sdk - validation - Extracted input shape: [1, 3, 224, 224]
2026-06-19 03:54 - INFO - sdk - validation - Implied image dimensions (W, H): (224, 224)
2026-06-19 03:54 - INFO - epu - chimera_job - START==================================onnx_ingest
2026-06-19 03:54 - INFO - epu - codegen - START===============================optimize_relay
2026-06-19 03:54 - INFO - epu - codegen - START====================quantize_to_cpu_runnable_fx
2026-06-19 03:54 - INFO - epu - fx -
Source name Op Output 0 Range Output 0 Frac Bits
----------------------- ---------------------- --------------------------------------- --------------------
output_DequantizeLinear contrib.epu.dequantize (-9.761742234230042, 38.10833987593651) 25
2026-06-19 03:54 - INFO - epu - codegen - START====================build_cpu_runnable_fx_relay
2026-06-19 03:54 - INFO - epu - codegen - START=======================quantize_to_chimera_fx
2026-06-19 03:54 - INFO - epu - codegen - START=================================relay_to_tir
2026-06-19 03:54 - INFO - epu - codegen - START===========================relay_to_epu_relay
2026-06-19 03:54 - INFO - epu - codegen - START==============================adapt_and_order
2026-06-19 03:54 - INFO - epu - mac_counter -
2026-06-19 03:54 - INFO - epu - mac_counter - ============================================================
2026-06-19 03:54 - INFO - epu - mac_counter - MAC Operation Count Summary
2026-06-19 03:54 - INFO - epu - mac_counter - ============================================================
2026-06-19 03:54 - INFO - epu - mac_counter - conv2d: 236,027,904 ops (118,013,952 MACs) - /conv1/Conv_quant
2026-06-19 03:54 - INFO - epu - mac_counter - conv2d: 231,211,008 ops (115,605,504 MACs) - /layer1/layer1.0/conv1/Conv_quant
2026-06-19 03:54 - INFO - epu - mac_counter - conv2d: 231,211,008 ops (115,605,504 MACs) - /layer1/layer1.0/conv2/Conv_quant
2026-06-19 03:54 - INFO - epu - mac_counter - conv2d: 231,211,008 ops (115,605,504 MACs) - /layer1/layer1.1/conv1/Conv_quant
2026-06-19 03:54 - INFO - epu - mac_counter - conv2d: 231,211,008 ops (115,605,504 MACs) - /layer1/layer1.1/conv2/Conv_quant
2026-06-19 03:54 - INFO - epu - mac_counter - conv2d: 115,605,504 ops (57,802,752 MACs) - /layer2/layer2.0/conv1/Conv_quant
2026-06-19 03:54 - INFO - epu - mac_counter - conv2d: 231,211,008 ops (115,605,504 MACs) - /layer2/layer2.0/conv2/Conv_quant
2026-06-19 03:54 - INFO - epu - mac_counter - conv2d: 12,845,056 ops (6,422,528 MACs) - /layer2/layer2.0/downsample/downsample.0/Conv_quant
2026-06-19 03:54 - INFO - epu - mac_counter - conv2d: 231,211,008 ops (115,605,504 MACs) - /layer2/layer2.1/conv1/Conv_quant
2026-06-19 03:54 - INFO - epu - mac_counter - conv2d: 231,211,008 ops (115,605,504 MACs) - /layer2/layer2.1/conv2/Conv_quant
2026-06-19 03:54 - INFO - epu - mac_counter - conv2d: 115,605,504 ops (57,802,752 MACs) - /layer3/layer3.0/conv1/Conv_quant
2026-06-19 03:54 - INFO - epu - mac_counter - conv2d: 231,211,008 ops (115,605,504 MACs) - /layer3/layer3.0/conv2/Conv_quant
2026-06-19 03:54 - INFO - epu - mac_counter - conv2d: 12,845,056 ops (6,422,528 MACs) - /layer3/layer3.0/downsample/downsample.0/Conv_quant
2026-06-19 03:54 - INFO - epu - mac_counter - conv2d: 231,211,008 ops (115,605,504 MACs) - /layer3/layer3.1/conv1/Conv_quant
2026-06-19 03:54 - INFO - epu - mac_counter - conv2d: 231,211,008 ops (115,605,504 MACs) - /layer3/layer3.1/conv2/Conv_quant
2026-06-19 03:54 - INFO - epu - mac_counter - conv2d: 115,605,504 ops (57,802,752 MACs) - /layer4/layer4.0/conv1/Conv_quant
2026-06-19 03:54 - INFO - epu - mac_counter - conv2d: 231,211,008 ops (115,605,504 MACs) - /layer4/layer4.0/conv2/Conv_quant
2026-06-19 03:54 - INFO - epu - mac_counter - conv2d: 12,845,056 ops (6,422,528 MACs) - /layer4/layer4.0/downsample/downsample.0/Conv_quant
2026-06-19 03:54 - INFO - epu - mac_counter - conv2d: 231,211,008 ops (115,605,504 MACs) - /layer4/layer4.1/conv1/Conv_quant
2026-06-19 03:54 - INFO - epu - mac_counter - conv2d: 231,211,008 ops (115,605,504 MACs) - /layer4/layer4.1/conv2/Conv_quant
2026-06-19 03:54 - INFO - epu - mac_counter - ------------------------------------------------------------
2026-06-19 03:54 - INFO - epu - mac_counter - Total: 3,627,122,688 ops (1,813,561,344 MACs)
2026-06-19 03:54 - INFO - epu - mac_counter - ============================================================
2026-06-19 03:54 - INFO - epu - mac_counter -
2026-06-19 03:54 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-06-19 03:54 - INFO - epu - codegen - START=============================plan_lrm_virtual
2026-06-19 03:54 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-06-19 03:54 - INFO - epu - codegen - START===============================lrm_alloc_loop
2026-06-19 03:54 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-06-19 03:54 - INFO - epu - codegen - START================================lrm_splitting
2026-06-19 03:55 - INFO - epu - codegen - START==============================ext_split_relay
2026-06-19 03:55 - INFO - epu - codegen - START====================================build_tir
2026-06-19 03:55 - INFO - epu - chimera_job - Compilation of ResNet_OpSet16_optimized_asym_int8_q_shaped_QC_U_1d7_16MB_4kB_128GBps_128GBps_16_OFF_x1_x1 successful
Processing: 100%|██████████████████████████████| 24/24 [02:07<00:00, 5.33s/it]
100%|██████████████████████████████████████████| 24/24 [00:05<00:00, 4.70it/s]
ISS mismatch not in ORT Top1:
[]
ISS mismatch not in ORT Top5:
[]
ORT mismatch not in ISS Top1:
[]
ORT mismatch not in ISS Top5:
[]
+------+--------+--------+
| | ORT | ISS |
+======+========+========+
| TOP1 | 62.50% | 62.50% |
+------+--------+--------+
| TOP5 | 87.50% | 87.50% |
+------+--------+--------+
Validation with the graph validate CLI tool
We also provide a convenient command line interface to perform the same task on the command line. However classifier validation is limited to only the dataset and class mappings contained within the docker container. For users who require the use of their own datasets, we recommend using the library call directly.
!sdk graph validate --help
Usage: sdk graph validate [OPTIONS] QUANTIZED_MODEL
Validate a quantized onnx against a set of validation images.
Options:
--mean [<float>, <float>, <float>]
Mean of the input normalization that the
model was trained with. [default: [0, 0,
0]]
--std [<float>, <float>, <float>]
Standard Deviation of the input
normalization that the model was trained
with. [default: [1, 1, 1]]
--num-images INTEGER Total maximum number of images to use
--num-threads INTEGER Number of threads to use. (Cannot exceed
machine limit)
--trange-file TEXT Optional tranges file
--help Show this message and exit.
!sdk graph validate {quant_result.qmodel_path} \
--mean '[0.485, 0.456, 0.406]' --std '[0.229, 0.224, 0.224]' \
--num-images {NUM_IMAGES} --num-threads {THREADS}
Usage: sdk graph validate [OPTIONS] QUANTIZED_MODEL
Try 'sdk graph validate --help' for help.
Error: Invalid value for '--num-images': '{NUM_IMAGES}' is not a valid integer.
The graph validate command writes a validation_report.json file to disk with the run summary.
!cat validation_report.json
cat: validation_report.json: No such file or directory
FILM Region Validation
In addition to top1 / top5 validation. We can use the built in FILM region validation to try to localize sources of error within the compilation process and correlated differences to regions of the ONNX graph.
from tvm.contrib.epu.chimera_job.chimera_job import ChimeraJob
## In addition to top1 / top5 validtion. We can use the built in FILM region validation to try to localize sources of error.
cgc_job = ChimeraJob(
model_p=str(quantized_onnx_model.model_path),
validate_iss=True,
)
cgc_job.compile()
cgc_job.validate_ort_iss()
2026-06-19 03:58 - INFO - epu - chimera_job - START==================================onnx_ingest
2026-06-19 03:58 - INFO - epu - codegen - START===============================optimize_relay
2026-06-19 03:58 - INFO - epu - codegen - START====================quantize_to_cpu_runnable_fx
2026-06-19 03:58 - INFO - epu - fx -
Source name Op Output 0 Range Output 0 Frac Bits
----------------------- ---------------------- --------------------------------------- --------------------
output_DequantizeLinear contrib.epu.dequantize (-9.761742234230042, 38.10833987593651) 25
2026-06-19 03:58 - INFO - epu - codegen - START====================build_cpu_runnable_fx_relay
2026-06-19 03:58 - INFO - epu - codegen - START=======================quantize_to_chimera_fx
2026-06-19 03:58 - INFO - epu - codegen - START=================================relay_to_tir
2026-06-19 03:58 - INFO - epu - codegen - START===========================relay_to_epu_relay
2026-06-19 03:58 - INFO - epu - codegen - START==============================adapt_and_order
2026-06-19 03:58 - INFO - epu - mac_counter -
2026-06-19 03:58 - INFO - epu - mac_counter - ============================================================
2026-06-19 03:58 - INFO - epu - mac_counter - MAC Operation Count Summary
2026-06-19 03:58 - INFO - epu - mac_counter - ============================================================
2026-06-19 03:58 - INFO - epu - mac_counter - conv2d: 236,027,904 ops (118,013,952 MACs) - /conv1/Conv_quant
2026-06-19 03:58 - INFO - epu - mac_counter - conv2d: 231,211,008 ops (115,605,504 MACs) - /layer1/layer1.0/conv1/Conv_quant
2026-06-19 03:58 - INFO - epu - mac_counter - conv2d: 231,211,008 ops (115,605,504 MACs) - /layer1/layer1.0/conv2/Conv_quant
2026-06-19 03:58 - INFO - epu - mac_counter - conv2d: 231,211,008 ops (115,605,504 MACs) - /layer1/layer1.1/conv1/Conv_quant
2026-06-19 03:58 - INFO - epu - mac_counter - conv2d: 231,211,008 ops (115,605,504 MACs) - /layer1/layer1.1/conv2/Conv_quant
2026-06-19 03:58 - INFO - epu - mac_counter - conv2d: 115,605,504 ops (57,802,752 MACs) - /layer2/layer2.0/conv1/Conv_quant
2026-06-19 03:58 - INFO - epu - mac_counter - conv2d: 231,211,008 ops (115,605,504 MACs) - /layer2/layer2.0/conv2/Conv_quant
2026-06-19 03:58 - INFO - epu - mac_counter - conv2d: 12,845,056 ops (6,422,528 MACs) - /layer2/layer2.0/downsample/downsample.0/Conv_quant
2026-06-19 03:58 - INFO - epu - mac_counter - conv2d: 231,211,008 ops (115,605,504 MACs) - /layer2/layer2.1/conv1/Conv_quant
2026-06-19 03:58 - INFO - epu - mac_counter - conv2d: 231,211,008 ops (115,605,504 MACs) - /layer2/layer2.1/conv2/Conv_quant
2026-06-19 03:58 - INFO - epu - mac_counter - conv2d: 115,605,504 ops (57,802,752 MACs) - /layer3/layer3.0/conv1/Conv_quant
2026-06-19 03:58 - INFO - epu - mac_counter - conv2d: 231,211,008 ops (115,605,504 MACs) - /layer3/layer3.0/conv2/Conv_quant
2026-06-19 03:58 - INFO - epu - mac_counter - conv2d: 12,845,056 ops (6,422,528 MACs) - /layer3/layer3.0/downsample/downsample.0/Conv_quant
2026-06-19 03:58 - INFO - epu - mac_counter - conv2d: 231,211,008 ops (115,605,504 MACs) - /layer3/layer3.1/conv1/Conv_quant
2026-06-19 03:58 - INFO - epu - mac_counter - conv2d: 231,211,008 ops (115,605,504 MACs) - /layer3/layer3.1/conv2/Conv_quant
2026-06-19 03:58 - INFO - epu - mac_counter - conv2d: 115,605,504 ops (57,802,752 MACs) - /layer4/layer4.0/conv1/Conv_quant
2026-06-19 03:58 - INFO - epu - mac_counter - conv2d: 231,211,008 ops (115,605,504 MACs) - /layer4/layer4.0/conv2/Conv_quant
2026-06-19 03:58 - INFO - epu - mac_counter - conv2d: 12,845,056 ops (6,422,528 MACs) - /layer4/layer4.0/downsample/downsample.0/Conv_quant
2026-06-19 03:58 - INFO - epu - mac_counter - conv2d: 231,211,008 ops (115,605,504 MACs) - /layer4/layer4.1/conv1/Conv_quant
2026-06-19 03:58 - INFO - epu - mac_counter - conv2d: 231,211,008 ops (115,605,504 MACs) - /layer4/layer4.1/conv2/Conv_quant
2026-06-19 03:58 - INFO - epu - mac_counter - ------------------------------------------------------------
2026-06-19 03:58 - INFO - epu - mac_counter - Total: 3,627,122,688 ops (1,813,561,344 MACs)
2026-06-19 03:58 - INFO - epu - mac_counter - ============================================================
2026-06-19 03:58 - INFO - epu - mac_counter -
2026-06-19 03:58 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-06-19 03:58 - INFO - epu - codegen - START=============================plan_lrm_virtual
2026-06-19 03:58 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-06-19 03:58 - INFO - epu - codegen - START===============================lrm_alloc_loop
2026-06-19 03:58 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-06-19 03:58 - INFO - epu - codegen - START================================lrm_splitting
2026-06-19 03:58 - INFO - epu - codegen - START==============================ext_split_relay
2026-06-19 03:59 - INFO - epu - codegen - START====================================build_tir
2026-06-19 03:59 - INFO - epu - chimera_job - Compilation of ResNet_OpSet16_optimized_asym_int8_q_shaped_QC_U_1d7_16MB_4kB_128GBps_128GBps_16_OFF_x1_x1 successful
2026-06-19 03:59 - INFO - epu - iss_testing - No tranges found for input, use default float range: <tvm.contrib.epu.interval.Interval object at 0x798f0abd9d50>
2026-06-19 03:59 - INFO - epu - iss_testing - No tranges found for input, use default float range: <tvm.contrib.epu.interval.Interval object at 0x798f0abd9d50>
2026-06-19 03:59 - INFO - epu - iss_testing - Started Executing Onnxruntime...
2026-06-19 03:59 - INFO - epu - iss_testing - Done 0:00:00.223592
2026-06-19 03:59 - INFO - epu - iss_testing - No tranges found for input, use default float range: <tvm.contrib.epu.interval.Interval object at 0x798ff27de230>
FILM 12/12: 100%|███████████████████████████████████████████████████| 12/12 [00:21<00:00, 1.81s/it]
2026-06-19 04:00 - WARNING - epu - iss_testing - /layer3/layer3.1/Add_quant::adapter_tb_strided_tb(tb(0)->tb(0))::/layer4/layer4.0/conv1/Conv_quant:0 not found in the graph that was run in ORT
2026-06-19 04:00 - INFO - epu - iss_testing -
======================================================================
ISS validation results (quantized model, rtol=0.1, atol=-1)
======================================================================
input_QuantizeLinear:0 (PASS, Node, int8)
Bitwise Mismatches 0 / 150528 (0.0000%)
Mismatches > Tol 0 / 150528 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0
PSNR inf dB
Max Abs Err 0
Max Abs Err / ε 0
Max Err Loc (0, 0, 0, 0)
ORT @ max err 18
ISS @ max err 18
/conv1/Conv_quant:0 (PASS, Node, int8x2)
Bitwise Mismatches 0 / 802816 (0.0000%)
Mismatches > Tol 0 / 802816 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0
PSNR inf dB
Max Abs Err 0
Max Abs Err / ε 0
Max Err Loc (0, 0, 0, 0)
ORT @ max err -128
ISS @ max err -128
/maxpool/MaxPool:0 (PASS, Node, int8x4)
Bitwise Mismatches 0 / 200704 (0.0000%)
Mismatches > Tol 0 / 200704 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0
PSNR inf dB
Max Abs Err 0
Max Abs Err / ε 0
Max Err Loc (0, 0, 0, 0)
ORT @ max err -127
ISS @ max err -127
/layer1/layer1.0/conv1/Conv_quant:0 (PASS, Node, int8x4)
Bitwise Mismatches 0 / 200704 (0.0000%)
Mismatches > Tol 0 / 200704 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0
PSNR inf dB
Max Abs Err 0
Max Abs Err / ε 0
Max Err Loc (0, 0, 0, 0)
ORT @ max err -128
ISS @ max err -128
/layer1/layer1.0/Add_quant:0 (PASS, Node, int8x4)
Bitwise Mismatches 0 / 200704 (0.0000%)
Mismatches > Tol 0 / 200704 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0
PSNR inf dB
Max Abs Err 0
Max Abs Err / ε 0
Max Err Loc (0, 0, 0, 0)
ORT @ max err -117
ISS @ max err -117
/layer1/layer1.1/conv1/Conv_quant:0 (PASS, Node, int8x4)
Bitwise Mismatches 0 / 200704 (0.0000%)
Mismatches > Tol 0 / 200704 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0
PSNR inf dB
Max Abs Err 0
Max Abs Err / ε 0
Max Err Loc (0, 0, 0, 0)
ORT @ max err -106
ISS @ max err -106
/layer1/layer1.1/Add_quant:0 (PASS, Node, int8x2)
Bitwise Mismatches 0 / 200704 (0.0000%)
Mismatches > Tol 0 / 200704 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0
PSNR inf dB
Max Abs Err 0
Max Abs Err / ε 0
Max Err Loc (0, 0, 0, 0)
ORT @ max err -118
ISS @ max err -118
/layer2/layer2.1/Add_quant:0 (PASS, Node, int8x2)
Bitwise Mismatches 0 / 100352 (0.0000%)
Mismatches > Tol 0 / 100352 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0
PSNR inf dB
Max Abs Err 0
Max Abs Err / ε 0
Max Err Loc (0, 0, 0, 0)
ORT @ max err -128
ISS @ max err -128
/avgpool/GlobalAveragePool_quant:0 (PASS, Node, int8)
Bitwise Mismatches 0 / 512 (0.0000%)
Mismatches > Tol 0 / 512 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0
PSNR inf dB
Max Abs Err 0
Max Abs Err / ε 0
Max Err Loc (0, 0, 0, 0)
ORT @ max err -128
ISS @ max err -128
output_DequantizeLinear:0 (PASS, Node, custom[qfp.25]32)
Bitwise Mismatches 0 / 1000 (0.0000%)
Mismatches > Tol 0 / 1000 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0
PSNR inf dB
Max Abs Err 0
Max Abs Err / ε 0
Max Err Loc (0, 0)
ORT @ max err -1.5018065
ISS @ max err -1.5018065
output (PASS, Output, custom[qfp.25]32)
Bitwise Mismatches 0 / 1000 (0.0000%)
Mismatches > Tol 0 / 1000 (0.0000%)
Thresholds rtol=0.1, atol=-1
RMSE 0
PSNR inf dB
Max Abs Err 0
Max Abs Err / ε 0
Max Err Loc (0, 0)
ORT @ max err -1.5018065
ISS @ max err -1.5018065
11 entries
[{'S': 'PASS',
'Type': 'Node',
'Name': 'input_QuantizeLinear:0',
'dtype': 'int8',
'total': 150528,
'Bitwise Mismatches': 0,
'Mismatch %': 0.0,
'RMSE': 0.0,
'PSNR (dB)': inf,
'Max Abs Err': 0.0,
'Max Abs Err Loc': (0, 0, 0, 0),
'ORT[loc]': 18,
'ISS[loc]': 18,
'Max Abs Err / ε': 0.0,
'Mismatches Above Tol': 0,
'rtol': 0.1,
'atol': -1,
'Comment': ''},
{'S': 'PASS',
'Type': 'Node',
'Name': '/conv1/Conv_quant:0',
'dtype': 'int8x2',
'total': 802816,
'Bitwise Mismatches': 0,
'Mismatch %': 0.0,
'RMSE': 0.0,
'PSNR (dB)': inf,
'Max Abs Err': 0.0,
'Max Abs Err Loc': (0, 0, 0, 0),
'ORT[loc]': -128,
'ISS[loc]': -128,
'Max Abs Err / ε': 0.0,
'Mismatches Above Tol': 0,
'rtol': 0.1,
'atol': -1,
'Comment': ''},
{'S': 'PASS',
'Type': 'Node',
'Name': '/maxpool/MaxPool:0',
'dtype': 'int8x4',
'total': 200704,
'Bitwise Mismatches': 0,
'Mismatch %': 0.0,
'RMSE': 0.0,
'PSNR (dB)': inf,
'Max Abs Err': 0.0,
'Max Abs Err Loc': (0, 0, 0, 0),
'ORT[loc]': -127,
'ISS[loc]': -127,
'Max Abs Err / ε': 0.0,
'Mismatches Above Tol': 0,
'rtol': 0.1,
'atol': -1,
'Comment': ''},
{'S': 'PASS',
'Type': 'Node',
'Name': '/layer1/layer1.0/conv1/Conv_quant:0',
'dtype': 'int8x4',
'total': 200704,
'Bitwise Mismatches': 0,
'Mismatch %': 0.0,
'RMSE': 0.0,
'PSNR (dB)': inf,
'Max Abs Err': 0.0,
'Max Abs Err Loc': (0, 0, 0, 0),
'ORT[loc]': -128,
'ISS[loc]': -128,
'Max Abs Err / ε': 0.0,
'Mismatches Above Tol': 0,
'rtol': 0.1,
'atol': -1,
'Comment': ''},
{'S': 'PASS',
'Type': 'Node',
'Name': '/layer1/layer1.0/Add_quant:0',
'dtype': 'int8x4',
'total': 200704,
'Bitwise Mismatches': 0,
'Mismatch %': 0.0,
'RMSE': 0.0,
'PSNR (dB)': inf,
'Max Abs Err': 0.0,
'Max Abs Err Loc': (0, 0, 0, 0),
'ORT[loc]': -117,
'ISS[loc]': -117,
'Max Abs Err / ε': 0.0,
'Mismatches Above Tol': 0,
'rtol': 0.1,
'atol': -1,
'Comment': ''},
{'S': 'PASS',
'Type': 'Node',
'Name': '/layer1/layer1.1/conv1/Conv_quant:0',
'dtype': 'int8x4',
'total': 200704,
'Bitwise Mismatches': 0,
'Mismatch %': 0.0,
'RMSE': 0.0,
'PSNR (dB)': inf,
'Max Abs Err': 0.0,
'Max Abs Err Loc': (0, 0, 0, 0),
'ORT[loc]': -106,
'ISS[loc]': -106,
'Max Abs Err / ε': 0.0,
'Mismatches Above Tol': 0,
'rtol': 0.1,
'atol': -1,
'Comment': ''},
{'S': 'PASS',
'Type': 'Node',
'Name': '/layer1/layer1.1/Add_quant:0',
'dtype': 'int8x2',
'total': 200704,
'Bitwise Mismatches': 0,
'Mismatch %': 0.0,
'RMSE': 0.0,
'PSNR (dB)': inf,
'Max Abs Err': 0.0,
'Max Abs Err Loc': (0, 0, 0, 0),
'ORT[loc]': -118,
'ISS[loc]': -118,
'Max Abs Err / ε': 0.0,
'Mismatches Above Tol': 0,
'rtol': 0.1,
'atol': -1,
'Comment': ''},
{'S': 'PASS',
'Type': 'Node',
'Name': '/layer2/layer2.1/Add_quant:0',
'dtype': 'int8x2',
'total': 100352,
'Bitwise Mismatches': 0,
'Mismatch %': 0.0,
'RMSE': 0.0,
'PSNR (dB)': inf,
'Max Abs Err': 0.0,
'Max Abs Err Loc': (0, 0, 0, 0),
'ORT[loc]': -128,
'ISS[loc]': -128,
'Max Abs Err / ε': 0.0,
'Mismatches Above Tol': 0,
'rtol': 0.1,
'atol': -1,
'Comment': ''},
{'S': 'PASS',
'Type': 'Node',
'Name': '/avgpool/GlobalAveragePool_quant:0',
'dtype': 'int8',
'total': 512,
'Bitwise Mismatches': 0,
'Mismatch %': 0.0,
'RMSE': 0.0,
'PSNR (dB)': inf,
'Max Abs Err': 0.0,
'Max Abs Err Loc': (0, 0, 0, 0),
'ORT[loc]': -128,
'ISS[loc]': -128,
'Max Abs Err / ε': 0.0,
'Mismatches Above Tol': 0,
'rtol': 0.1,
'atol': -1,
'Comment': ''},
{'S': 'PASS',
'Type': 'Node',
'Name': 'output_DequantizeLinear:0',
'dtype': 'custom[qfp.25]32',
'total': 1000,
'Bitwise Mismatches': 0,
'Mismatch %': 0.0,
'RMSE': 0.0,
'PSNR (dB)': inf,
'Max Abs Err': 0.0,
'Max Abs Err Loc': (0, 0),
'ORT[loc]': -1.5018065,
'ISS[loc]': -1.5018065,
'Max Abs Err / ε': 0.0,
'Mismatches Above Tol': 0,
'rtol': 0.1,
'atol': -1,
'Comment': ''},
{'S': 'PASS',
'Type': 'Output',
'Name': 'output',
'dtype': 'custom[qfp.25]32',
'total': 1000,
'Bitwise Mismatches': 0,
'Mismatch %': 0.0,
'RMSE': 0.0,
'PSNR (dB)': inf,
'Max Abs Err': 0.0,
'Max Abs Err Loc': (0, 0),
'ORT[loc]': -1.5018065,
'ISS[loc]': -1.5018065,
'Max Abs Err / ε': 0.0,
'Mismatches Above Tol': 0,
'rtol': 0.1,
'atol': -1,
'Comment': ''}]
