Tutorial: Quantizing and Validating a Neural Network

NOTE: The Jupyter Notebook below is included in the Chimera SDK and can be run interactively by running the following CLI command:

$ quadric sdk notebook

From the Jupyter Notebook window in your browser, select the notebook named /quadric/sdk-cli/examples/quantization/QuadricQuantization.ipynb.

In this tutorial we will perform post-training quantization (PTQ) on a pre-trained neural network.

A few basic assumptions:

• This notebook should be able to be used as-is for any classifier from the torchvision model hub that has been pre-trained on the imagenet dataset.
• Images, labels, and models used in this notebook conform to the ImageNet-1K standard, i.e. use 224x224 image resolution and have subjects represented in the 1000 ImageNet classes. Any different input size or class labels will require user-supplied datasets.

At a high level we will perform the following steps:

1. Download a classifier from the torchvision repository
2. Prepare and condition a validation set of image data
3. Export the neural network to ONNX with the correct OPSET
4. Quantize the network with supported quantization methods, using a calibration dataset
5. Validate and compare the results of quantizing the network

First let's set up a few high-level variables. These will be used throughout the tutorial.

NOTE: This tutorial describes the steps that one might take to optimize a model for accurate performance with lower precision (INT8) weights. There are many ways to perform quantization with varying degrees of effort and results. If you just want to do an analysis of the runtime performance (throughout and latency) of your model, we strongly recommend that you use the SDK CLI command for naive quantization.

import logging
import sys
import warnings
from pathlib import Path
from typing import Dict

get_ipython().run_line_magic("matplotlib", "inline")
import matplotlib.pyplot as plt
import numpy as np
import torch
import torchvision
from matplotlib.gridspec import GridSpec
from PIL import Image
from torch.utils.data import DataLoader, Subset
from torchvision.transforms import CenterCrop, Compose, Normalize, Resize, ToTensor

from tvm.contrib.epu.chimera_job.constants import DEFAULT_ONNX_OPSET
from sdk_cli.utils.dataloaders import CalibrationDataLoader
from sdk_cli.utils.datasets import ImageNet_Mini_Quadric, QuadricCalibration
from sdk_cli.utils.datasets.ImageNet import (
    IMAGENET_1K_NORMALIZATION_PARAMETERS,
    OPTIMIZED_IMAGENET_1K_LABELS,
)
from sdk_cli.utils.model_helpers import ClassifyResult
from sdk_cli.utils.performance_trackers import ClassifierPerformanceTracker
from sdk_cli.lib.quantize import (
    QuantizationExperiment,
    QuantizedONNXModel,
    quantize_onnx_model,
    run_quantization_experiment,
)
from sdk_cli.visualizers.layouter import ClassifierLayouter
from tvm.relay.backend.contrib.epu.util import logger as tvm_logger

2. Loading the model

For this tutorial, we will support classifiers from the torchvision library. The torchvision library is an open source library that contains many pre-trained neural network models.

By default we will use RESNET18 for classification tasks. All input tensors in this dataset are of dimension NCHW format [3, 224, 224] and need to be float-normalized with a mean, sigma of [0.485, 0.456, 0.406], [0.229, 0.224, 0.225]

For more documentation about the models supported please see: https://pytorch.org/vision/0.8/models.html

Currently Chimera Graph Compiler will support mobilenet and resnet style networks as well as backbones of similar network structure. However, in this notebook, one can explore the validation accuracies of all classifiers in the torchvision library.

By default, we will quantize and run the RESNET18 model from torchvision with pre-trained weights. The user is free to experiment with other models from the torchvision classifier set.

## NOTE: Full list of available torchvision models can be found by running the following:
## print(torchvision.models.list_models())

pytorch_model = torchvision.models.resnet18(weights="DEFAULT")

The next step is to convert our pre-trained torchvision model into the standard format used: ONNX. For this task, first we will need to define an input tensor with the same shape as tensors that will pass the model and an opset number. The input tensor needs to be defined in NCHW format as is the convention for most pre-trained networks in the torchvision library. Generally, CGC will only support the NCHW convention for input tensor arrangement.

## Convert PyTorch model into ONNX Format
onnx_model_path = Path(f"{pytorch_model.__class__.__name__}_float32.onnx")

## Create an example input tensor for the model
input_tensor_shape = (1, 3, 224, 224)
example_input = torch.randn(*input_tensor_shape, requires_grad=True)

## Export the model to ONNX format
torch.onnx.export(
    pytorch_model,
    example_input,
    str(onnx_model_path),
    export_params=True,  # Store the trained parameter weights inside the model file
    do_constant_folding=True,  # Execute constant folding for optimization
    opset_version=DEFAULT_ONNX_OPSET,
    input_names=["input"],
    output_names=["output"],
)

## NOTE: Change normalization parameters below if using dataset other than ImageNet
transforms = Compose(
    [
        Resize(224),
        CenterCrop(224),
        ToTensor(),
        Normalize(
            IMAGENET_1K_NORMALIZATION_PARAMETERS.channel_means,
            IMAGENET_1K_NORMALIZATION_PARAMETERS.channel_standard_deviations,
        ),
    ]
)

## Path to directory containing data to use for for numerical range calibration during quantization
## Data is used also used to compare accuracy of fp32 and int8 models
dataset = ImageNet_Mini_Quadric.Dataset(transform=transforms)
subset_of_dataset = Subset(dataset, range(100))

calibration_dataloader = CalibrationDataLoader(
    DataLoader(subset_of_dataset, batch_size=1, shuffle=True), ["input"]
)

3. Quantization

Theoretical aspects of quantization used

What is quantization?

Quantization in Machine Learning (ML) is the process of converting data in FP32 (floating point 32 bits) to a smaller precision like INT8 (Integer 8 bit) and perform all critical operations like Convolution in INT8 and at the end, convert the lower precision output to higher precision in FP32.

What tools should I use?

This process can be achieved using multiple libraries like Pytorch (GraphFX or Eager mode), ONNX Runtime, Apache TVM or Tensorflow. Testing different frameworks, we've obtained best results using ONNX Runtime framework. Therefore, we'll be using it in this tutorial. Before jumping into the code, let's have a quick overview about the theoretical aspects of quantization.

How it works?

During quantization, the floating point values are mapped to an 8 bit quantization space of the form:

$ValFP32 = scale * (ValQuantized - ZeroPoint)$

where $scale$ is a positive real number used to map the floating point numbers to a quantization space. For symetric quantization it is calculated as follows: $scale = (DataRangeMax - DataRangeMin) / (QuantizationRangeMax - QuantizationRangeMin)$

$ZeroPoint$ represents zero in the quantization space. It is important that the floating point zero value be exactly representable in quantization space. This is because zero padding is used in many CNNs. If it is not possible to represent 0 uniquely after quantization, it will result in accuracy errors.

Quantization using ONNXRuntime framework

Quantization in ONNXRuntime refers to 8 bit linear quantization of an ONNX model.

There are two ways of quantizing a model: dynamic and static.

1. Dynamic quantization: This method calculates the quantization parameters (scale and zero point) for activations dynamically.
2. Static quantization: leverages calibration data to calculate the quantization parameters of activations. Our quantization tool supports three calibration methods: MinMax, Entropy and Percentile. Please refer to calibrate.py for details.

The main difference between dynamic and static quantization is how the scale and zero point of activations are calculated. For static quantization, they are calculated in advance (offline) using a calibration data set. For dynamic quantization, they are calculated on-the-fly (online) and are specific for each forward pass.

In general, it is recommended to use dynamic quantization for RNNs and transformer-based models, and static quantization for CNN models. Therefore, we'll continue by adopting static quantization.
For more technical details, please refer to the official documentation of ONNXRuntime.

quantized_onnx_model: QuantizedONNXModel = quantize_onnx_model(
    onnx_model_path, subset_of_dataset, asymmetric_activation=False
)

2026-06-19 03:53 - INFO - sdk - quantize - ONNX model shapes inferred.
2026-06-19 03:53 - DEBUG - sdk - quantize - Forcing node types: []
2026-06-19 03:53 - DEBUG - sdk - quantize - ONNX Node types excluded from quantization: ['Softmax', 'Sigmoid', 'QuadricCustomOp']
2026-06-19 03:53 - DEBUG - sdk - quantize - ONNX Node names excluded from quantization: []
2026-06-19 03:53 - INFO - sdk - quantize - Starting quantization...
WARNING:root:Please use QuantFormat.QDQ for activation type QInt8 and weight type QInt8. Or it will lead to bad performance on x64.
2026-06-19 03:53 - INFO - sdk - quantize - Quantization completed! Quantized model saved to /quadric/sdk-cli/examples/quantization/ResNet_OpSet16_optimized_sym_int8_q.onnx
2026-06-19 03:53 - INFO - sdk - quantize - ONNX full precision model size: 44.58MB
2026-06-19 03:53 - INFO - sdk - quantize - ONNX quantized model size: 11.2MB
2026-06-19 03:53 - INFO - sdk - quantize - ONNX model shapes inferred.
2026-06-19 03:53 - INFO - sdk - quantize - ONNX Model with well-defined shapes has been saved at `/quadric/sdk-cli/examples/quantization/ResNet_OpSet16_optimized_sym_int8_q_shaped.onnx`.
2026-06-19 03:53 - DEBUG - sdk - quantize - Checking for FLOAT/FLOAT16 types...
2026-06-19 03:53 - INFO - sdk - quantize - Checking for remaining FLOAT/FLOAT16 types.
2026-06-19 03:53 - INFO - sdk - quantize - Model still has FLOAT/FLOAT16 types after quantization. Creating ranges for floating point tensors using calibration data...
2026-06-19 03:54 - INFO - sdk - quantize - Saved computed tensor ranges to /quadric/sdk-cli/examples/quantization/ResNet_OpSet16_optimized_sym_int8_q_shaped.tranges.
2026-06-19 03:54 - INFO - sdk - quantize - 
╒════════════════════════════════════════════════════════════════════════════════════════╤═══════════════════════════════════════════════════════════════════════════════════════════╕
│ Quantized ONNX Model                                                                   │ Tensor Ranges File                                                                        │
╞════════════════════════════════════════════════════════════════════════════════════════╪═══════════════════════════════════════════════════════════════════════════════════════════╡
│ /quadric/sdk-cli/examples/quantization/ResNet_OpSet16_optimized_sym_int8_q_shaped.onnx │ /quadric/sdk-cli/examples/quantization/ResNet_OpSet16_optimized_sym_int8_q_shaped.tranges │
╘════════════════════════════════════════════════════════════════════════════════════════╧═══════════════════════════════════════════════════════════════════════════════════════════╛

4. Testing the performance of quantized model

The last part of this tutorial is to make sure that quantization was done succesfully by checking the performance of the original model against the quantized model.

Depending on the task, you may choose different metrics like spearman correlation, f1-score, AUC, ROC, L2 norm, IoU etc.

For our classification task, we'll resume only on accuruacy, precision, recall, f1 score and L2 norm.

We will test on the full validation set.

MAX_NUM_SAMPLES_FOR_ACCURACY_CALCULATION = 1

quantization_experiment = QuantizationExperiment(
    floating_point_onnx_model_path=onnx_model_path,
    quantized_onnx_model=quantized_onnx_model,
    performance_tracker=ClassifierPerformanceTracker(),
)

## Set `export_path` if you'd like to save a copy of your experiment's results to disk
## E.g. `export_path=resnet18_quantization_experiment_report.txt`
run_quantization_experiment(
    quantization_experiment,
    calibration_dataloader,
    export_path=None,
    max_num_samples=MAX_NUM_SAMPLES_FOR_ACCURACY_CALCULATION,
)

1: FP32100.00% <> INT8100.00%: 100%|█████████████| 1/1 [00:00<00:00,  1.72it/s]


Used 1 image samples for model accuracy comparison.
Original FP32 model accuracy (Top-1): 100.00%
Quantized INT8 model accuracy (Top-1): 100.00%
Change in Top-1 model accuracy due to quantization: 0.00%

<div class="alert alert-block alert-warning"> <b>Note:</b> The <b>reported accuracy</b> for ResNet18, evaluated on ImageNet, is reported at this link: https://pytorch.org/vision/stable/models.html </div>

## NOTE: `ToTensor` is needed at a minimum to convert from `PIL.Image` to `torch.Tensor`
dataset_without_transforms = QuadricCalibration.Dataset(transform=Compose([ToTensor()]))

calibration_dataloader = CalibrationDataLoader(
    DataLoader(dataset_without_transforms, batch_size=1, shuffle=True), ["input"]
)

quantization_experiment = QuantizationExperiment(
    floating_point_onnx_model_path=onnx_model_path,
    quantized_onnx_model=quantized_onnx_model,
    performance_tracker=ClassifierPerformanceTracker(),
)


def recover_original_image_from_onnxruntime_format(
    image_formatted_for_onnxruntime: Dict[str, np.ndarray]
) -> Image:
    """Recover the original image loaded from disk as a PIL.Image from the onnxruntime formatted numpy array."""
    original_image_as_numpy = np.moveaxis(
        np.uint8(image_formatted_for_onnxruntime["input"][0] * 255.0), 0, -1
    )
    return Image.fromarray(original_image_as_numpy)


## NOTE: Code below is a near identical copy to the source code of the `run_quantization` function
## Only differences are we are performing the transformations outside of the CalibrationDataloader so that we
## can visualize the original image
num_samples_to_display = 6
while len(quantization_experiment) < num_samples_to_display:
    # Load and unpack next sample
    next_sample = calibration_dataloader.get_next(return_targets=True)
    image_formatted_for_onnxruntime: Dict[str, np.ndarray] = next_sample[0]
    ground_truth_label = int(next_sample[1])

    original_image = recover_original_image_from_onnxruntime_format(image_formatted_for_onnxruntime)

    # Perform image transforms outside of the dataloader so that we may visualize the original image
    transformed_image = np.expand_dims(transforms(original_image).numpy(), axis=0)
    transformed_image_formatted_for_onnxruntime = {"input": transformed_image}

    # Run inference
    quantization_experiment.run(
        ([], transformed_image_formatted_for_onnxruntime),
        ground_truth_label=ground_truth_label,
        ground_truth_class_map=OPTIMIZED_IMAGENET_1K_LABELS.class_map,
    )

    quantization_experiment_report = quantization_experiment.report(
        labels=list(OPTIMIZED_IMAGENET_1K_LABELS.class_map.keys()),
        target_names=list(OPTIMIZED_IMAGENET_1K_LABELS.class_map.values()),
    )

    classifier_layouter = ClassifierLayouter(original_image, "", "extraction")

    for order, infer_type in enumerate(["Original FP32", "Quantized INT8"]):
        classifier_layouter.add_data(
            quantization_experiment_report[order + 5][-1],
            f"{infer_type} Model - onnxruntime top5 %",
        )

    classifier_layouter.display()

tvm_logger.setLevel(logging.WARNING)
warnings.filterwarnings("ignore")
sys.tracebacklimit = 0

from tvm.contrib.epu.chimera_job.chimera_job import ChimeraJob

output = None
array_size = 32

tvm_logger.setLevel(logging.WARNING)
warnings.filterwarnings("ignore")
sys.tracebacklimit = 0

cgc_job = ChimeraJob(
    model_p=str(quantization_experiment.quantized_onnx_model.model_path),
    trange_file=str(quantization_experiment.quantized_onnx_model.tensor_ranges_path),
)
cgc_job.compile()

2026-06-19 03:54 - INFO - epu - chimera_job - START==================================onnx_ingest
2026-06-19 03:54 - INFO - epu - chimera_job - Numerical ranges provided
2026-06-19 03:54 - INFO - epu - codegen - START===============================optimize_relay
2026-06-19 03:54 - INFO - epu - codegen - START====================quantize_to_cpu_runnable_fx
2026-06-19 03:54 - INFO - epu - fx - 

Source name              Op                      Output 0 Range           Output 0 Frac Bits
-----------------------  ----------------------  ---------------------  --------------------
output_DequantizeLinear  contrib.epu.dequantize  [-10.5101f, 36.0347f]                    25

2026-06-19 03:54 - INFO - epu - codegen - START====================build_cpu_runnable_fx_relay
2026-06-19 03:54 - INFO - epu - codegen - START=======================quantize_to_chimera_fx
2026-06-19 03:54 - INFO - epu - codegen - START=================================relay_to_tir
2026-06-19 03:54 - INFO - epu - codegen - START===========================relay_to_epu_relay
2026-06-19 03:54 - INFO - epu - codegen - START==============================adapt_and_order
2026-06-19 03:54 - INFO - epu - mac_counter - 
2026-06-19 03:54 - INFO - epu - mac_counter - ============================================================
2026-06-19 03:54 - INFO - epu - mac_counter - MAC Operation Count Summary
2026-06-19 03:54 - INFO - epu - mac_counter - ============================================================
2026-06-19 03:54 - INFO - epu - mac_counter -   conv2d: 236,027,904 ops (118,013,952 MACs) - /conv1/Conv_quant
2026-06-19 03:54 - INFO - epu - mac_counter -   conv2d: 231,211,008 ops (115,605,504 MACs) - /layer1/layer1.0/conv1/Conv_quant
2026-06-19 03:54 - INFO - epu - mac_counter -   conv2d: 231,211,008 ops (115,605,504 MACs) - /layer1/layer1.0/conv2/Conv_quant
2026-06-19 03:54 - INFO - epu - mac_counter -   conv2d: 231,211,008 ops (115,605,504 MACs) - /layer1/layer1.1/conv1/Conv_quant
2026-06-19 03:54 - INFO - epu - mac_counter -   conv2d: 231,211,008 ops (115,605,504 MACs) - /layer1/layer1.1/conv2/Conv_quant
2026-06-19 03:54 - INFO - epu - mac_counter -   conv2d: 115,605,504 ops (57,802,752 MACs) - /layer2/layer2.0/conv1/Conv_quant
2026-06-19 03:54 - INFO - epu - mac_counter -   conv2d: 231,211,008 ops (115,605,504 MACs) - /layer2/layer2.0/conv2/Conv_quant
2026-06-19 03:54 - INFO - epu - mac_counter -   conv2d: 12,845,056 ops (6,422,528 MACs) - /layer2/layer2.0/downsample/downsample.0/Conv_quant
2026-06-19 03:54 - INFO - epu - mac_counter -   conv2d: 231,211,008 ops (115,605,504 MACs) - /layer2/layer2.1/conv1/Conv_quant
2026-06-19 03:54 - INFO - epu - mac_counter -   conv2d: 231,211,008 ops (115,605,504 MACs) - /layer2/layer2.1/conv2/Conv_quant
2026-06-19 03:54 - INFO - epu - mac_counter -   conv2d: 115,605,504 ops (57,802,752 MACs) - /layer3/layer3.0/conv1/Conv_quant
2026-06-19 03:54 - INFO - epu - mac_counter -   conv2d: 231,211,008 ops (115,605,504 MACs) - /layer3/layer3.0/conv2/Conv_quant
2026-06-19 03:54 - INFO - epu - mac_counter -   conv2d: 12,845,056 ops (6,422,528 MACs) - /layer3/layer3.0/downsample/downsample.0/Conv_quant
2026-06-19 03:54 - INFO - epu - mac_counter -   conv2d: 231,211,008 ops (115,605,504 MACs) - /layer3/layer3.1/conv1/Conv_quant
2026-06-19 03:54 - INFO - epu - mac_counter -   conv2d: 231,211,008 ops (115,605,504 MACs) - /layer3/layer3.1/conv2/Conv_quant
2026-06-19 03:54 - INFO - epu - mac_counter -   conv2d: 115,605,504 ops (57,802,752 MACs) - /layer4/layer4.0/conv1/Conv_quant
2026-06-19 03:54 - INFO - epu - mac_counter -   conv2d: 231,211,008 ops (115,605,504 MACs) - /layer4/layer4.0/conv2/Conv_quant
2026-06-19 03:54 - INFO - epu - mac_counter -   conv2d: 12,845,056 ops (6,422,528 MACs) - /layer4/layer4.0/downsample/downsample.0/Conv_quant
2026-06-19 03:54 - INFO - epu - mac_counter -   conv2d: 231,211,008 ops (115,605,504 MACs) - /layer4/layer4.1/conv1/Conv_quant
2026-06-19 03:54 - INFO - epu - mac_counter -   conv2d: 231,211,008 ops (115,605,504 MACs) - /layer4/layer4.1/conv2/Conv_quant
2026-06-19 03:54 - INFO - epu - mac_counter - ------------------------------------------------------------
2026-06-19 03:54 - INFO - epu - mac_counter - Total: 3,627,122,688 ops (1,813,561,344 MACs)
2026-06-19 03:54 - INFO - epu - mac_counter - ============================================================
2026-06-19 03:54 - INFO - epu - mac_counter - 
2026-06-19 03:54 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-06-19 03:54 - INFO - epu - codegen - START=============================plan_lrm_virtual
2026-06-19 03:55 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-06-19 03:55 - INFO - epu - codegen - START===============================lrm_alloc_loop
2026-06-19 03:55 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-06-19 03:55 - INFO - epu - codegen - START================================lrm_splitting
2026-06-19 03:55 - INFO - epu - codegen - START==============================ext_split_relay
2026-06-19 03:56 - INFO - epu - codegen - START====================================build_tir
2026-06-19 03:56 - INFO - epu - chimera_job - Compilation of ResNet_OpSet16_optimized_sym_int8_q_shaped_QC_U_1d7_16MB_4kB_128GBps_128GBps_16_OFF_x1_x1 successful

The first thing we can check is the FX Canonicalized Result. In this stage, we replace any NON-MAC compute that remains in neural networks with the appropriate Fixed Point representaitons.

Finally we can run the entire Architecture-specific lowering passes and check the resulting outputs.

## inspect the fx canonicalized relay runtime resullt
result_fxquant = ClassifyResult("fxquant", OPTIMIZED_IMAGENET_1K_LABELS.class_map, softmax=True)
relay_output = cgc_job.run_relay_inf_session(inputs={"input": transformed_image})
result_fxquant.output = relay_output["output"]
result_fxquant.calculate_stats()
print(result_fxquant)

## inspect the architecture specific Chimera ISS resullt
result_iss = ClassifyResult("iss", OPTIMIZED_IMAGENET_1K_LABELS.class_map, softmax=True)
cgc_output = cgc_job.run_inference_harness(inputs={"input": transformed_image})
result_iss.output = cgc_output["output"]
result_iss.calculate_stats()
print(result_iss)

2026-06-19 03:56 - INFO - epu - iss_testing - Found tranges for input: <tvm.contrib.epu.interval.Interval object at 0x7f0794457bb0>
2026-06-19 03:56 - INFO - epu - iss_testing - Started Executing Quantized Relay...
2026-06-19 03:56 - INFO - epu - iss_testing - Done 0:00:09.470533


---------------------fxquant----------------------
espresso_maker...............................0.681
Polaroid_camera..............................0.083
toaster......................................0.083
dishwasher...................................0.034
microwave....................................0.019
Top1: espresso_maker 0.681



2026-06-19 03:56 - INFO - epu - iss_testing - Found tranges for input: <tvm.contrib.epu.interval.Interval object at 0x7f07944560e0>
FILM 12/12: 100%|███████████████████████████████████████████████████| 12/12 [00:17<00:00,  1.45s/it]
2026-06-19 03:56 - INFO - epu - iss_testing - Found tranges for input: <tvm.contrib.epu.interval.Interval object at 0x7f080ed1ada0>
2026-06-19 03:56 - INFO - epu - iss_testing - Started Executing Onnxruntime...
2026-06-19 03:56 - INFO - epu - iss_testing - Done 0:00:00.590089


-----------------------iss------------------------
espresso_maker...............................0.681
Polaroid_camera..............................0.083
toaster......................................0.083
dishwasher...................................0.034
microwave....................................0.019
Top1: espresso_maker 0.681

Optional Study

Now that we have several different representations of the network:

• Original FP32 ONNX
• INT8-quantized ONNX
• Fully Architecture-Lowered Assembly We can study and compare different combinations of these numerical representations of the graph.

Run the quantized onnx runtime session and compare it to the generated assembly running in ISS

NUM_IMAGES = 2
THREADS = 2

all_images = []
all_inputs = []
for _ in range(NUM_IMAGES):
    # Load and unpack next sample
    next_sample = calibration_dataloader.get_next(return_targets=True)
    image_formatted_for_onnxruntime: Dict[str, np.ndarray] = next_sample[0]
    ground_truth_label = int(next_sample[1])

    original_image_as_numpy = np.moveaxis(
        np.uint8(image_formatted_for_onnxruntime["input"][0] * 255.0), 0, -1
    )
    original_image = Image.fromarray(original_image_as_numpy)

    # Perform image transforms outside of the dataloader so that we may visualize the original image
    transformed_image = np.expand_dims(transforms(original_image).numpy(), axis=0)
    transformed_image_formatted_for_onnxruntime = {"input": transformed_image}

    all_images.append(original_image_as_numpy)
    all_inputs.append(transformed_image_formatted_for_onnxruntime)


all_results = cgc_job.run_batch_inference_harness(inputs=all_inputs, threads=THREADS)
all_results_ort = cgc_job.run_batch_ort_harness(inputs=all_inputs)

Processing: 100%|████████████████████████████████| 2/2 [00:16<00:00,  8.10s/it]
100%|████████████████████████████████████████████| 2/2 [00:00<00:00,  4.03it/s]

for image_numpy, result_iss, result_ort in zip(all_images, all_results, all_results_ort):
    classifier_layouter = ClassifierLayouter(image_numpy, "")

    classifier_layouter.add_data(result_ort["output"], "ort int8 top 5 (%)")
    classifier_layouter.add_data(result_iss["output"], "iss top 5 (%)")

    classifier_layouter.display()