Model Demo: VGG-16 A8W4 Quantization

NOTE: The Jupyter Notebook below is included in the Chimera SDK and can be run interactively by running the following CLI command:

$ quadric sdk notebook

From the Jupyter Notebook window in your browser, select the notebook named /quadric/sdk-cli/examples/models/vgg16/vgg16_quantize.ipynb.

VGG-16 A8W4 Quantization with the Quadric SDK

This notebook quantizes VGG-16's three fully-connected classifier Gemm layers to A8W4 (INT8 activations, INT4 weights) while keeping the convolutional backbone at INT8, then compares accuracy and on-target cycles against the INT8 (A8W8) baseline. The classifier holds the bulk of VGG-16's weights, so int4-ing it is where the memory and cycle savings come from.

Why hand-build A8W4?

We want an INT8 QOperator backbone with INT4 on only the three classifier Gemm weights — a configuration ONNX Runtime's quantize_static can't produce directly:

• weight_type is global — you can't apply INT4 to just the classifier Gemms while keeping the rest INT8.
• INT4 is QDQ-only — requesting INT4 converts the entire model to the QDQ format, dropping the INT8 QOperator (QLinearConv / QGemm) backbone.

We did try to coax the config out of quantize_static — quantizing the backbone to INT8 and the classifier Gemms to INT4 in two steps — but Top-1 dropped by roughly 20 points. We didn't isolate the exact cause; two things change versus the direct swap, and either (or both) could be responsible:

• a redundant requantization of the classifier's input (int8 → float → int8), and
• the ReLUs un-fuse — the QOperator path folds each inter-classifier Relu into the preceding QGemm's output requantization, but the QDQ path leaves them as standalone float nodes, changing how those activations are quantized.

So instead we keep ONNX Runtime's INT8 QOperator model as the A8W8 baseline and surgically swap the three QGemms for INT4 QDQ subgraphs — reusing each QGemm's own scales and re-deriving the INT4 weights from the original FP32 tensors — which gives the exact A8W4 configuration with no accuracy penalty.

Model: VGG-16 (ImageNet-1K, 224×224, classifier = 3 FC Gemms: 25088 → 4096 → 4096 → 1000)

1. Setup

Import the SDK, download the prebuilt FP32 vgg16_fp32.onnx from Quadric's public model store (on first run), and build the ImageNet-Mini dataset used for both calibration and accuracy evaluation.

from pathlib import Path
from urllib.request import urlretrieve

import numpy as np
from onnxruntime import InferenceSession
from torch.utils.data import Subset
from torchvision.transforms import CenterCrop, Compose, Normalize, Resize, ToTensor
from tvm.contrib.epu.chimera_job.chimera_job import ChimeraJob
from tvm.contrib.epu.chimera_job.hw_config import HWConfig
from tvm.contrib.epu.chimera_job.plot import get_max_total_cycles

from sdk_cli.utils.datasets import ImageNet_Mini_Quadric
from sdk_cli.utils.datasets.ImageNet import IMAGENET_1K_NORMALIZATION_PARAMETERS

from vgg16_helpers import build_vgg_quant_models

onnx_model_path = "vgg16_fp32.onnx"
S3_BASE = "https://sdk-cli-models.s3.us-east-2.amazonaws.com"
if not Path(onnx_model_path).exists():
    urlretrieve(f"{S3_BASE}/{onnx_model_path}", onnx_model_path)

model_input_size = (224, 224)
transforms = Compose(
    [
        Resize(256),
        CenterCrop(model_input_size[0]),
        ToTensor(),
        Normalize(
            IMAGENET_1K_NORMALIZATION_PARAMETERS.channel_means,
            IMAGENET_1K_NORMALIZATION_PARAMETERS.channel_standard_deviations,
        ),
    ]
)

imagenet_dataset = ImageNet_Mini_Quadric.Dataset(transform=transforms)
subset_of_dataset = Subset(imagenet_dataset, range(100))

2. Quantization

A single build_vgg_quant_models(...) call produces the three models this notebook compares:

• A8W8 — the INT8 QOperator baseline (QLinearConv / QGemm), calibrated on 100 ImageNet-Mini images.
• A8W4 — the classifier Gemm weights swapped to per-output-channel INT4 (INT8 activations, INT8 output, FP32 bias), saved as a QDQ model that ONNX Runtime can execute.
• A8W4 custom-op — the INT4 Gemms matched to the Quadric channelwiseQuantMatMul custom op (weights packed to V8I4) for the Chimera Graph Compiler.

The A8W8 and A8W4 models drive the accuracy comparison; the A8W8 and A8W4 custom-op models drive the cycle comparison. All the graph transforms live in vgg16_helpers.py.

models = build_vgg_quant_models(onnx_model_path, subset_of_dataset)

print("A8W8 (int8)     :", models.a8w8.model_path)
print("A8W4 (int4 QDQ) :", models.a8w4.model_path)
print("A8W4 custom op  :", models.a8w4_customop_path)

2026-08-20 14:34 - INFO - sdk - quantize - ONNX model to quantize is defined in OpSet 11 , but the Chimera Graph Compiler (CGC) currently only supports models defined in OpSets: [12, 13, 14, 15, 16]. Converting to OpSet 12.
2026-08-20 14:34 - INFO - sdk - quantize - ONNX model shapes inferred.
2026-08-20 14:35 - DEBUG - sdk - quantize - Forcing node types: []
2026-08-20 14:35 - DEBUG - sdk - quantize - ONNX Node types excluded from quantization: ['Softmax', 'Sigmoid', 'QuadricCustomOp']
2026-08-20 14:35 - DEBUG - sdk - quantize - ONNX Node names excluded from quantization: ['Softmax_37']
2026-08-20 14:35 - INFO - sdk - quantize - Starting quantization...
WARNING:root:Please use QuantFormat.QDQ for activation type QInt8 and weight type QInt8. Or it will lead to bad performance on x64.
2026-08-20 14:36 - INFO - sdk - quantize - Quantization completed! Quantized model saved to /quadric/sdk-cli/examples/models/vgg16/vgg16_fp32_OpSet12_optimized_asym_int8_q.onnx
2026-08-20 14:36 - INFO - sdk - quantize - ONNX full precision model size: 527.8MB
2026-08-20 14:36 - INFO - sdk - quantize - ONNX quantized model size: 132.0MB
2026-08-20 14:36 - INFO - sdk - quantize - ONNX model shapes inferred.
2026-08-20 14:36 - INFO - sdk - quantize - ONNX Model with well-defined shapes has been saved at `/quadric/sdk-cli/examples/models/vgg16/vgg16_fp32_OpSet12_optimized_asym_int8_q_shaped.onnx`.
2026-08-20 14:36 - DEBUG - sdk - quantize - Checking for FLOAT/FLOAT16 types...
2026-08-20 14:36 - INFO - sdk - quantize - Checking for remaining FLOAT/FLOAT16 types.
2026-08-20 14:36 - INFO - sdk - quantize - Model still has FLOAT/FLOAT16 types after quantization. Creating ranges for floating point tensors using calibration data...
2026-08-20 14:37 - INFO - sdk - quantize - Saved computed tensor ranges to /quadric/sdk-cli/examples/models/vgg16/vgg16_fp32_OpSet12_optimized_asym_int8_q_shaped.tranges.
2026-08-20 14:37 - INFO - sdk - quantize - 
╒═════════════════════════════════════════════════════════════════════════════════════════════╤════════════════════════════════════════════════════════════════════════════════════════════════╕
│ Quantized ONNX Model                                                                        │ Tensor Ranges File                                                                             │
╞═════════════════════════════════════════════════════════════════════════════════════════════╪════════════════════════════════════════════════════════════════════════════════════════════════╡
│ /quadric/sdk-cli/examples/models/vgg16/vgg16_fp32_OpSet12_optimized_asym_int8_q_shaped.onnx │ /quadric/sdk-cli/examples/models/vgg16/vgg16_fp32_OpSet12_optimized_asym_int8_q_shaped.tranges │
╘═════════════════════════════════════════════════════════════════════════════════════════════╧════════════════════════════════════════════════════════════════════════════════════════════════╛


Swapped 3 QGemms -> INT4 QDQ (opset 21) -> vgg16_a8w4_gemm_qdq.onnx


2026-08-20 14:59 - INFO - sdk - quantize - Saved computed tensor ranges to vgg16_a8w4_gemm_qdq.tranges.


A8W8 (int8)     : /quadric/sdk-cli/examples/models/vgg16/vgg16_fp32_OpSet12_optimized_asym_int8_q_shaped.onnx
A8W4 (int4 QDQ) : vgg16_a8w4_gemm_qdq.onnx
A8W4 custom op  : vgg16_gemm_customop.onnx

3. Accuracy — Top-1 / Top-5 (FP32 / A8W8 / A8W4)

Run the FP32, INT8 (A8W8), and INT4-Gemm (A8W4) models in ONNX Runtime over the evaluation set and report Top-1 / Top-5 for each. Comparing A8W8 vs A8W4 isolates the accuracy cost of INT4 on the classifier. (Accuracy is measured in ONNX Runtime on the host CPU; on-target cycles come from the ISS in the next section.)

MAX_EVAL = 1000
eval_dataset = Subset(imagenet_dataset, range(min(MAX_EVAL, len(imagenet_dataset))))

sessions = {
    "FP32": InferenceSession(str(onnx_model_path)),
    "A8W8": InferenceSession(str(models.a8w8.model_path)),
    "A8W4": InferenceSession(str(models.a8w4.model_path)),
}
input_names = {k: s.get_inputs()[0].name for k, s in sessions.items()}

n = 0
top1 = {k: 0 for k in sessions}
top5 = {k: 0 for k in sessions}
for idx in range(len(eval_dataset)):
    img, label = eval_dataset[idx]
    x = np.expand_dims(img.numpy(), 0).astype("float32")
    label = int(label)
    for k, sess in sessions.items():
        out = sess.run(None, {input_names[k]: x})[0].ravel()
        top1[k] += int(out.argmax() == label)
        top5[k] += int(label in out.argsort()[-5:])
    n += 1

for k in sessions:
    print(f"{k}:  Top-1 {100*top1[k]/n:5.2f}%   Top-5 {100*top5[k]/n:5.2f}%   ({n} images)")

FP32:  Top-1 76.20%   Top-5 93.10%   (1000 images)
A8W8:  Top-1 75.60%   Top-5 93.10%   (1000 images)
A8W4:  Top-1 75.50%   Top-5 93.00%   (1000 images)

4. Compile & Cycles (A8W8 vs A8W4)

Compile the A8W8 model and the A8W4 custom-op model with the CGC (Chimera Graph Compiler) and run them on the ISS (Instruction Set Simulator), then report cycles / latency / FPS and the overall cycle reduction from int4-ing the classifier Gemms.

Hardware configuration:

• Product: QC-U (Quadric Chimera Processor)
• Clock: 1.5 GHz
• MACs per PE: 16
• DDR bandwidth: 32 GB/s (read + write)
• OCM: 16 MB, LRM: 4 kB, 1 core (defaults)

img0, _ = subset_of_dataset[0]
x0 = np.expand_dims(img0.numpy(), 0).astype("float32")

## Target HW: QC-U, 1.5 GHz, 16 MACs/PE, 32 GBps DDR (read + write).
## Other fields keep their defaults (ocm_size=16MB, lrm_size=4kB, num_cores=1).
hw_config = HWConfig(
    product="QC-U",
    clock_freq_ghz=1.5,
    macs_per_pe=16,
    ext_rd_bw="32GBps",
    ext_wr_bw="32GBps",
)

## --- A8W8 (int8 model) ---
cgc_job = ChimeraJob(
    model_p=str(models.a8w8.model_path),
    trange_file=str(models.a8w8.tensor_ranges_path),
    hw_config=hw_config,
)
cgc_job.compile(quiet=True)
cgc_job.run_inference_harness(inputs={"input": x0}, compare_ort=False)
stats_a8w8 = cgc_job.parse_profile_results()
cycles_a8w8 = get_max_total_cycles(cgc_job.profile_dict, cgc_job.hw_config.effective_core_count)

## --- A8W4 (int4-Gemm custom-op model) ---
cgc_job_a8w4 = ChimeraJob(
    model_p=models.a8w4_customop_path,
    trange_file=str(models.a8w4.tensor_ranges_path),
    hw_config=hw_config,
)
cgc_job_a8w4.compile(quiet=True)
cgc_job_a8w4.run_inference_harness(inputs={"input": x0}, compare_ort=False)
stats_a8w4 = cgc_job_a8w4.parse_profile_results()
cycles_a8w4 = get_max_total_cycles(
    cgc_job_a8w4.profile_dict, cgc_job_a8w4.hw_config.effective_core_count
)

reduction = 100.0 * (cycles_a8w8 - cycles_a8w4) / cycles_a8w8

2026-08-20 15:14 - INFO - epu - iss_testing - Found tranges for input: <tvm.contrib.epu.interval.Interval object at 0x7db6f15f9240>
FILM 11/11: 100%|███████████████████████████████████████████████████| 11/11 [00:50<00:00,  4.58s/it]
2026-08-20 15:16 - INFO - epu - iss_testing - Found tranges for input: <tvm.contrib.epu.interval.Interval object at 0x7db6f24785e0>
FILM 14/14: 100%|███████████████████████████████████████████████████| 14/14 [00:42<00:00,  3.03s/it]

print(
    f"[A8W8]  Cycles: {cycles_a8w8:,}   Latency (ms): {stats_a8w8.get('Latency (ms)')}   FPS: {stats_a8w8.get('FPS')}"
)
print(
    f"[A8W4]  Cycles: {cycles_a8w4:,}   Latency (ms): {stats_a8w4.get('Latency (ms)')}   FPS: {stats_a8w4.get('FPS')}"
)
verb = "reduce" if reduction >= 0 else "increase"
print(f"\n Cycles {verb} by around {abs(reduction):.1f}% overall (A8W8 -> A8W4).")

[A8W8]  Cycles: 7,270,089   Latency (ms): 4.85   FPS: 206.32
[A8W4]  Cycles: 4,589,218   Latency (ms): 3.06   FPS: 326.85

 Cycles reduce by around 36.9% overall (A8W8 -> A8W4).

Summary

Techniques

Key takeaways

1. INT4 on the classifier is essentially lossless — A8W4 tracks A8W8 within ~0.1% Top-1 (75.5% vs 75.6% over 1000 images), because the classifier's per-channel weight distribution quantizes well to 4 bits.
2. The cycle win is large — int4-ing the three FC Gemms cuts total cycles by ~36.6% (7.33M → 4.65M), since the classifier's weights dominate VGG-16.
3. The config needed a surgical build — quantize_static can't target INT4 to just the Gemms without converting the whole model to QDQ (and un-fusing the ReLUs), so we quantize to INT8 and swap only the three Gemms.