NOTE: The Jupyter Notebook below is included in the Chimera SDK and can be run interactively by running the following CLI command:
$ quadric sdk notebook
From the Jupyter Notebook window in your browser, select the notebook named /quadric/sdk-cli/examples/quantization/AsymmetricQuantization.ipynb.
from pathlib import Path
from typing import Dict
import numpy as np
import torch
from PIL import Image
from torch.utils.data import DataLoader, Subset
from torchvision.transforms import CenterCrop, Compose, Normalize, Resize, ToTensor
from tvm.contrib.epu.chimera_job.constants import DEFAULT_ONNX_OPSET
from sdk_cli.lib.chimera_iss_experiment import ChimeraISSExperiment
from sdk_cli.lib.quantize import (
QuantizedONNXModel,
quantize_onnx_model,
)
from sdk_cli.utils.dataloaders import CalibrationDataLoader
from sdk_cli.utils.datasets import ImageNet_Mini_Quadric, QuadricCalibration
from sdk_cli.utils.datasets.ImageNet import (
IMAGENET_1K_NORMALIZATION_PARAMETERS,
OPTIMIZED_IMAGENET_1K_LABELS,
)
from sdk_cli.utils.performance_trackers import ClassifierPerformanceTracker
from sdk_cli.visualizers.layouter import ClassifierLayouter
THREADS = 1
## NOTE: Change normalization parameters below if using dataset other than ImageNet
transforms = Compose(
[
Resize(224),
CenterCrop(224),
ToTensor(),
Normalize(
IMAGENET_1K_NORMALIZATION_PARAMETERS.channel_means,
IMAGENET_1K_NORMALIZATION_PARAMETERS.channel_standard_deviations,
),
]
)
## Path to directory containing data to use for for numerical range calibration during quantization
## Data is used also used to compare accuracy of fp32 and int8 models
dataset = ImageNet_Mini_Quadric.Dataset(transform=transforms)
subset_of_dataset = Subset(dataset, range(100))
calibration_dataloader = CalibrationDataLoader(
DataLoader(subset_of_dataset, batch_size=1, shuffle=True), ["input"]
)
import torchvision
MODELS: Dict[str, Dict[str, ChimeraISSExperiment]] = {
"mobilenet_v2": {"symmetrically_quantized": None, "asymmetrically_quantized": None},
# "resnet18": {"symmetrically_quantized": None, "asymmetrically_quantized": None},
# "resnet50": {"symmetrically_quantized": None, "asymmetrically_quantized": None},
# "vgg16": {"symmetrically_quantized": None, "asymmetrically_quantized": None},
}
for name, data in MODELS.items():
# Convert PyTorch model into ONNX Format
pytorch_model = eval(f"torchvision.models.{name}(weights='DEFAULT')")
onnx_model_path = Path(f"{name}_float32.onnx")
# Create an example input tensor for the model
input_tensor_shape = (1, 3, 224, 224)
example_input = torch.randn(*input_tensor_shape, requires_grad=True)
# Export the model to ONNX format
torch.onnx.export(
pytorch_model,
example_input,
str(onnx_model_path),
export_params=True, # Store the trained parameter weights inside the model file
do_constant_folding=True, # Execute constant folding for optimization
opset_version=DEFAULT_ONNX_OPSET,
input_names=["input"],
output_names=["output"],
)
# Quantize model using symmetric quantization
symmetrically_quantized_onnx_model: QuantizedONNXModel = quantize_onnx_model(
onnx_model_path, subset_of_dataset, asymmetric_activation=False
)
# Compile symmetrically quantized ONNX model for Chimera GPNPU
MODELS[name]["symmetrically_quantized"] = ChimeraISSExperiment(
onnx_model_path,
symmetrically_quantized_onnx_model,
performance_tracker=ClassifierPerformanceTracker(),
)
# Quantize model using asymmetric quantization
asymmetrically_quantized_onnx_model: QuantizedONNXModel = quantize_onnx_model(
onnx_model_path, subset_of_dataset, asymmetric_activation=True
)
# Compile asymmetrically quantized ONNX model for Chimera GPNPU
MODELS[name]["asymmetrically_quantized"] = ChimeraISSExperiment(
onnx_model_path,
asymmetrically_quantized_onnx_model,
performance_tracker=ClassifierPerformanceTracker(),
)
2026-06-19 03:53 - INFO - sdk - quantize - ONNX model shapes inferred.
2026-06-19 03:53 - DEBUG - sdk - quantize - Forcing node types: []
2026-06-19 03:53 - DEBUG - sdk - quantize - ONNX Node types excluded from quantization: ['Softmax', 'Sigmoid', 'QuadricCustomOp']
2026-06-19 03:53 - DEBUG - sdk - quantize - ONNX Node names excluded from quantization: []
2026-06-19 03:53 - INFO - sdk - quantize - Starting quantization...
WARNING:root:Please use QuantFormat.QDQ for activation type QInt8 and weight type QInt8. Or it will lead to bad performance on x64.
2026-06-19 03:53 - INFO - sdk - quantize - Quantization completed! Quantized model saved to /quadric/sdk-cli/examples/quantization/mobilenet_v2_OpSet16_optimized_sym_int8_q.onnx
2026-06-19 03:53 - INFO - sdk - quantize - ONNX full precision model size: 13.34MB
2026-06-19 03:53 - INFO - sdk - quantize - ONNX quantized model size: 3.48MB
2026-06-19 03:53 - INFO - sdk - quantize - ONNX model shapes inferred.
2026-06-19 03:53 - INFO - sdk - quantize - ONNX Model with well-defined shapes has been saved at `/quadric/sdk-cli/examples/quantization/mobilenet_v2_OpSet16_optimized_sym_int8_q_shaped.onnx`.
2026-06-19 03:53 - DEBUG - sdk - quantize - Checking for FLOAT/FLOAT16 types...
2026-06-19 03:53 - INFO - sdk - quantize - Checking for remaining FLOAT/FLOAT16 types.
2026-06-19 03:53 - INFO - sdk - quantize - Model still has FLOAT/FLOAT16 types after quantization. Creating ranges for floating point tensors using calibration data...
2026-06-19 03:54 - INFO - sdk - quantize - Saved computed tensor ranges to /quadric/sdk-cli/examples/quantization/mobilenet_v2_OpSet16_optimized_sym_int8_q_shaped.tranges.
2026-06-19 03:54 - INFO - sdk - quantize -
╒══════════════════════════════════════════════════════════════════════════════════════════════╤═════════════════════════════════════════════════════════════════════════════════════════════════╕
│ Quantized ONNX Model │ Tensor Ranges File │
╞══════════════════════════════════════════════════════════════════════════════════════════════╪═════════════════════════════════════════════════════════════════════════════════════════════════╡
│ /quadric/sdk-cli/examples/quantization/mobilenet_v2_OpSet16_optimized_sym_int8_q_shaped.onnx │ /quadric/sdk-cli/examples/quantization/mobilenet_v2_OpSet16_optimized_sym_int8_q_shaped.tranges │
╘══════════════════════════════════════════════════════════════════════════════════════════════╧═════════════════════════════════════════════════════════════════════════════════════════════════╛
2026-06-19 03:57 - INFO - sdk - quantize - ONNX model shapes inferred.
2026-06-19 03:57 - DEBUG - sdk - quantize - Forcing node types: []
2026-06-19 03:57 - DEBUG - sdk - quantize - ONNX Node types excluded from quantization: ['Softmax', 'Sigmoid', 'QuadricCustomOp']
2026-06-19 03:57 - DEBUG - sdk - quantize - ONNX Node names excluded from quantization: []
2026-06-19 03:57 - INFO - sdk - quantize - Starting quantization...
WARNING:root:Please use QuantFormat.QDQ for activation type QInt8 and weight type QInt8. Or it will lead to bad performance on x64.
2026-06-19 03:57 - INFO - sdk - quantize - Quantization completed! Quantized model saved to /quadric/sdk-cli/examples/quantization/mobilenet_v2_OpSet16_optimized_asym_int8_q.onnx
2026-06-19 03:57 - INFO - sdk - quantize - ONNX full precision model size: 13.34MB
2026-06-19 03:57 - INFO - sdk - quantize - ONNX quantized model size: 3.44MB
2026-06-19 03:57 - INFO - sdk - quantize - ONNX model shapes inferred.
2026-06-19 03:57 - INFO - sdk - quantize - ONNX Model with well-defined shapes has been saved at `/quadric/sdk-cli/examples/quantization/mobilenet_v2_OpSet16_optimized_asym_int8_q_shaped.onnx`.
2026-06-19 03:57 - DEBUG - sdk - quantize - Checking for FLOAT/FLOAT16 types...
2026-06-19 03:57 - INFO - sdk - quantize - Checking for remaining FLOAT/FLOAT16 types.
2026-06-19 03:57 - INFO - sdk - quantize - Model still has FLOAT/FLOAT16 types after quantization. Creating ranges for floating point tensors using calibration data...
2026-06-19 03:57 - INFO - sdk - quantize - Saved computed tensor ranges to /quadric/sdk-cli/examples/quantization/mobilenet_v2_OpSet16_optimized_asym_int8_q_shaped.tranges.
2026-06-19 03:57 - INFO - sdk - quantize -
╒═══════════════════════════════════════════════════════════════════════════════════════════════╤══════════════════════════════════════════════════════════════════════════════════════════════════╕
│ Quantized ONNX Model │ Tensor Ranges File │
╞═══════════════════════════════════════════════════════════════════════════════════════════════╪══════════════════════════════════════════════════════════════════════════════════════════════════╡
│ /quadric/sdk-cli/examples/quantization/mobilenet_v2_OpSet16_optimized_asym_int8_q_shaped.onnx │ /quadric/sdk-cli/examples/quantization/mobilenet_v2_OpSet16_optimized_asym_int8_q_shaped.tranges │
╘═══════════════════════════════════════════════════════════════════════════════════════════════╧══════════════════════════════════════════════════════════════════════════════════════════════════╛
def recover_original_image_from_onnxruntime_format(
image_formatted_for_onnxruntime: Dict[str, np.ndarray]
) -> Image:
"""Recover the original image loaded from disk as a PIL.Image from the onnxruntime formatted numpy array."""
original_image_as_numpy = np.moveaxis(
np.uint8(image_formatted_for_onnxruntime["input"][0] * 255.0), 0, -1
)
return Image.fromarray(original_image_as_numpy)
num_images = THREADS
## NOTE: `ToTensor` is needed at a minimum to convert from `PIL.Image` to `torch.Tensor`
dataset_without_transforms = QuadricCalibration.Dataset(transform=Compose([ToTensor()]))
subset_of_dataset = Subset(dataset_without_transforms, range(num_images))
for model_name in MODELS.keys():
calibration_dataloader = CalibrationDataLoader(
DataLoader(subset_of_dataset, batch_size=1, shuffle=True), ["input"]
)
for image in calibration_dataloader:
# Recover the original image loaded from disk as a PIL.Image from the onnxruntime formatted numpy array
# NOTE: We only do this because we wish to visualize the original image, before transformations
original_image = recover_original_image_from_onnxruntime_format(image)
# Perform image transforms outside of the dataloader so that we may visualize the original image
transformed_image = np.expand_dims(transforms(original_image).numpy(), axis=0)
transformed_image_formatted_for_onnxruntime = {"input": transformed_image}
ground_truth_label = None
# Run inference for symmetrically quantized model experiment
MODELS[model_name]["symmetrically_quantized"].run(
([], transformed_image_formatted_for_onnxruntime),
ground_truth_label=ground_truth_label,
ground_truth_class_map=OPTIMIZED_IMAGENET_1K_LABELS.class_map,
)
symmetrically_quantized_experiment_report = MODELS[model_name][
"symmetrically_quantized"
].report(
labels=list(OPTIMIZED_IMAGENET_1K_LABELS.class_map.keys()),
target_names=list(OPTIMIZED_IMAGENET_1K_LABELS.class_map.values()),
)
# Unpack report
# NOTE: Model accuracy is not reported because ground truth labels don't exist for calibration data
# original_model_accuracy = symmetrically_quantized_experiment_report[0]
# quantized_model_accuracy = symmetrically_quantized_experiment_report[1]
# chimera_model_accuracy = symmetrically_quantized_experiment_report[2]
# accuracy_deviation_due_to_quantization = symmetrically_quantized_experiment_report[3]
# accuracy_deviation_due_to_chimera_runtime = symmetrically_quantized_experiment_report[4]
# l2_norm_of_quantization = symmetrically_quantized_experiment_report[5]
# l2_norm_of_chimera_runtime = symmetrically_quantized_experiment_report[6]
# accuracy_report_of_quantization = symmetrically_quantized_experiment_report[7]
# accuracy_report_of_chimera_runtime = symmetrically_quantized_experiment_report[8]
top5_predictions_by_floating_point_model = symmetrically_quantized_experiment_report[9][-1]
top5_predictions_by_symmetrically_quantized_model = (
symmetrically_quantized_experiment_report[10][-1]
)
top5_predictions_by_symmetrically_quantized_chimera_model = (
symmetrically_quantized_experiment_report[11][-1]
)
# Run inference for symmetrically quantized model experiment
MODELS[model_name]["asymmetrically_quantized"].run(
([], transformed_image_formatted_for_onnxruntime),
ground_truth_label=ground_truth_label,
ground_truth_class_map=OPTIMIZED_IMAGENET_1K_LABELS.class_map,
threads=THREADS,
)
asymmetrically_quantized_experiment_report = MODELS[model_name][
"asymmetrically_quantized"
].report(
labels=list(OPTIMIZED_IMAGENET_1K_LABELS.class_map.keys()),
target_names=list(OPTIMIZED_IMAGENET_1K_LABELS.class_map.values()),
)
# Unpack report
# NOTE: Model accuracy is not reported because ground truth labels don't exist for calibration data
# original_model_accuracy = asymmetrically_quantized_experiment_report[0]
# quantized_model_accuracy = asymmetrically_quantized_experiment_report[1]
# chimera_model_accuracy = asymmetrically_quantized_experiment_report[2]
# accuracy_deviation_due_to_quantization = asymmetrically_quantized_experiment_report[3]
# accuracy_deviation_due_to_chimera_runtime = asymmetrically_quantized_experiment_report[4]
# l2_norm_of_quantization = asymmetrically_quantized_experiment_report[5]
# l2_norm_of_chimera_runtime = asymmetrically_quantized_experiment_report[6]
# accuracy_report_of_quantization = asymmetrically_quantized_experiment_report[7]
# accuracy_report_of_chimera_runtime = asymmetrically_quantized_experiment_report[8]
top5_predictions_by_floating_point_model = asymmetrically_quantized_experiment_report[9][-1]
top5_predictions_by_asymmetrically_quantized_model = (
asymmetrically_quantized_experiment_report[10][-1]
)
top5_predictions_by_asymmetrically_quantized_chimera_model = (
asymmetrically_quantized_experiment_report[11][-1]
)
# Display inference results.
classifier_layouter = ClassifierLayouter(original_image, f"{model_name}", "extraction")
classifier_layouter.add_data(top5_predictions_by_floating_point_model, "float top5 %")
classifier_layouter.add_data(
top5_predictions_by_symmetrically_quantized_model, "ort_sym top5 %"
)
classifier_layouter.add_data(
top5_predictions_by_symmetrically_quantized_chimera_model, "iss_sym top5 %"
)
classifier_layouter.add_data(top5_predictions_by_floating_point_model, "float top5 %")
classifier_layouter.add_data(
top5_predictions_by_asymmetrically_quantized_model, "ort_asym top5 %"
)
classifier_layouter.add_data(
top5_predictions_by_asymmetrically_quantized_chimera_model,
"iss_asym top5 %",
)
classifier_layouter.display()
2026-06-19 03:59 - WARNING - epu - chimera_job - Using max available 32 for batch processing
Processing: 100%|████████████████████████████████| 1/1 [00:11<00:00, 11.04s/it]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/7 [00:00<?, ?it/s]
FILM 1/7: 14%|███████▌ | 1/7 [00:00<00:00, 3718.35it/s]
FILM 1/7: 14%|███████▌ | 1/7 [00:00<00:00, 1162.18it/s]
FILM 1/7: 29%|███████████████▋ | 2/7 [00:02<00:06, 1.25s/it]
FILM 2/7: 29%|███████████████▋ | 2/7 [00:02<00:06, 1.25s/it]
FILM 2/7: 29%|███████████████▋ | 2/7 [00:02<00:06, 1.25s/it]
FILM 2/7: 43%|███████████████████████▌ | 3/7 [00:04<00:06, 1.57s/it]
FILM 3/7: 43%|███████████████████████▌ | 3/7 [00:04<00:06, 1.57s/it]
FILM 3/7: 43%|███████████████████████▌ | 3/7 [00:04<00:06, 1.57s/it]
FILM 3/7: 57%|███████████████████████████████▍ | 4/7 [00:05<00:03, 1.22s/it]
FILM 4/7: 57%|███████████████████████████████▍ | 4/7 [00:05<00:03, 1.22s/it]
FILM 4/7: 57%|███████████████████████████████▍ | 4/7 [00:05<00:03, 1.22s/it]
FILM 4/7: 71%|███████████████████████████████████████▎ | 5/7 [00:06<00:02, 1.19s/it]
FILM 5/7: 71%|███████████████████████████████████████▎ | 5/7 [00:06<00:02, 1.19s/it]
FILM 5/7: 71%|███████████████████████████████████████▎ | 5/7 [00:06<00:02, 1.19s/it]
FILM 5/7: 86%|███████████████████████████████████████████████▏ | 6/7 [00:07<00:01, 1.28s/it]
FILM 6/7: 86%|███████████████████████████████████████████████▏ | 6/7 [00:07<00:01, 1.28s/it]
FILM 6/7: 86%|███████████████████████████████████████████████▏ | 6/7 [00:07<00:01, 1.28s/it]
FILM 6/7: 100%|███████████████████████████████████████████████████████| 7/7 [00:07<00:00, 1.11it/s]
FILM 7/7: 100%|███████████████████████████████████████████████████████| 7/7 [00:07<00:00, 1.11it/s]
FILM 7/7: 100%|███████████████████████████████████████████████████████| 7/7 [00:07<00:00, 1.12s/it]
100%|████████████████████████████████████████████| 1/1 [00:08<00:00, 8.26s/it]
## Generate run statistics reports for each MODEL
for model_name in MODELS:
MODELS[model_name][
"symmetrically_quantized"
].chimera_compiled_model_inference_session.plot_run_statistics()
MODELS[model_name][
"asymmetrically_quantized"
].chimera_compiled_model_inference_session.plot_run_statistics()
2026-06-19 04:00 - INFO - epu - chimera_job - Combined plots generated and saved to:
/quadric/sdk-cli/examples/quantization/ccl_build/mobilenet_v2_OpSet16_optimized_sym_int8_q_shaped_QC_U_1d7_16MB_4kB_128GBps_128GBps_16_OFF_x1_x1/run/20260619_035945_c5d6e7/data/mobilenet_v2_OpSet16_optimized_sym_int8_q_shaped_QC_U_1d7_16MB_4kB_128GBps_128GBps_16_OFF_x1_x1.combined.png
2026-06-19 04:00 - INFO - epu - chimera_job - Combined plots generated and saved to:
/quadric/sdk-cli/examples/quantization/ccl_build/mobilenet_v2_OpSet16_optimized_asym_int8_q_shaped_QC_U_1d7_16MB_4kB_128GBps_128GBps_16_OFF_x1_x1/run/20260619_035957_4c4816/data/mobilenet_v2_OpSet16_optimized_asym_int8_q_shaped_QC_U_1d7_16MB_4kB_128GBps_128GBps_16_OFF_x1_x1.combined.png
