NOTE: The Jupyter Notebook below is included in the Chimera SDK and can be run interactively by running the following CLI command:
$ quadric sdk notebook
From the Jupyter Notebook window in your browser, select the notebook named /quadric/sdk-cli/examples/models/yolo/yolov5/yolov5.ipynb.
YOLOv5 Object Detection on Chimera GPNPU
YOLOv5, from Ultralytics, is an anchor-based single-stage detector that automatically learns anchor-box priors from the training data. This notebook compiles the yolov5n variant end-to-end: download the pre-built backbone + head ONNX, quantize the backbone to INT8, compile with the Chimera Graph Compiler (CGC), and run ISS + ORT INT8 inferences side-by-side.
Why split the graph?
YOLOv5's head mixes anchor decoding and per-scale detection ops that are cheaper to keep on the host CPU; the backbone is a dense stack of convolutions that compiles well to the Chimera GPNPU. The ONNX is pre-split at the head boundary — CGC handles the convolutional backbone and ONNX Runtime handles the classical tail.
Model: YOLOv5n (COCO, 640×640, 1.9M params, 4.5 GFLOPs)
Note — split pipeline.
- On the Chimera GPNPU: the backbone
- On the host CPU: the detection head ONNX, NMS, and bounding-box visualization
The full pipeline can run end-to-end on the GPNPU — see
yolox_e2e_chipy.ipynborretina_net_e2e_chipy.ipynbfor examples that compose CCL custom ops with ChiPy to keep everything on-chip.
1. Setup
Imports and per-run configuration. The S3 ONNX download, calibration dataset, input loading, and bounding-box visualization all live in yolov5_helpers.py.
import gc
import numpy as np
import onnx
from onnxruntime import InferenceSession
from PIL import Image
from sdk_cli.lib.inference import InferenceEngine, batch_inference
from sdk_cli.lib.quantize import QuantizedONNXModel, quantize_onnx_model
from sdk_cli.node_builtins.classical.yolo_postprocessing import (
boxes_detections,
get_postprocess_handle,
)
from sdk_cli.utils.models.yolo import YOLOModelVariant
from tvm.contrib.epu.chimera_job.chimera_job import ChimeraJob
from tvm.contrib.epu.chimera_job.hw_config import DEFAULT_32_ARRAY_SIZE
from yolov5_helpers import (
DEFAULT_IMAGE_PATHS,
INPUT_SIZE,
build_calibration_dataset,
display_detections,
download_model,
load_images,
)
%matplotlib inline
2. Model Selection
Ultralytics ships YOLOv5 in several sizes. This notebook compiles yolov5n (nano) for a fast round-trip through the SDK.
| Model | Size (px) | mAPval 50-95 | mAPval 50 | CPU b=1 (ms) | Params (M) | FLOPs @640 (B) |
|---|---|---|---|---|---|---|
| YOLOv5n | 640 | 28.0 | 45.7 | 45 | 1.9 | 4.5 |
| YOLOv5s | 640 | 37.4 | 56.8 | 98 | 7.2 | 16.5 |
| YOLOv5m | 640 | 45.4 | 64.1 | 224 | 21.2 | 49.0 |
| YOLOv5l | 640 | 49.0 | 67.3 | 430 | 46.5 | 109.1 |
| YOLOv5n6 | 1280 | 36.0 | 54.4 | 153 | 3.2 | 4.6 |
3. Model Download
download_model() fetches the pre-built YOLOv5n backbone + head ONNX from the public sdk-cli-models S3 bucket. The backbone is the convolutional stack; the head holds the anchor-decode + per-scale detection ops and runs on the host through ORT.
backbone_onnx_file, head_onnx_file = download_model()
Downloading yolov5n-backbone.onnx ...
Backbone: yolov5n-backbone.onnx (7.5 MB)
Downloading yolov5n-head.onnx ...
Head: yolov5n-head.onnx (0.4 MB)
4. Quantization
The backbone is quantized to INT8 with asymmetric activations using the COCO-like subset of QuadricCalibration for tensor-range statistics. The QuantizedONNXModel return value carries both the quantized ONNX and its companion .tranges file.
calibration_dataset = build_calibration_dataset(INPUT_SIZE)
quantized_onnx_model: QuantizedONNXModel = quantize_onnx_model(
backbone_onnx_file,
calibration_dataset,
asymmetric_activation=True,
)
print(f"Quantized ONNX: {quantized_onnx_model.model_path.name}")
print(f"Tensor ranges: {quantized_onnx_model.tensor_ranges_path.name}")
gc.collect()
2026-06-19 03:53 - INFO - sdk - quantize - ONNX model shapes inferred.
2026-06-19 03:53 - DEBUG - sdk - quantize - Forcing node types: []
2026-06-19 03:53 - DEBUG - sdk - quantize - ONNX Node types excluded from quantization: ['Softmax', 'Sigmoid', 'QuadricCustomOp']
2026-06-19 03:53 - DEBUG - sdk - quantize - ONNX Node names excluded from quantization: ['/model/model/model.20/m/m.0/cv1/act/Mul', '/model/model/model.4/cv1/act/Mul', '/model/model/model.6/m/m.2/cv2/act/Sigmoid', '/model/model/model.18/act/Mul', '/model/model/model.20/cv3/act/Sigmoid', '/model/model/model.17/m/m.0/cv2/act/Sigmoid', '/model/model/model.6/m/m.0/cv2/act/Mul', '/model/model/model.23/m/m.0/cv1/act/Sigmoid', '/model/model/model.17/cv1/act/Mul', '/model/model/model.2/m/m.0/cv2/act/Mul', '/model/model/model.4/m/m.1/cv1/act/Mul', '/model/model/model.6/m/m.2/cv1/act/Sigmoid', '/model/model/model.17/cv3/act/Sigmoid', '/model/model/model.8/cv2/act/Mul', '/model/model/model.4/cv2/act/Sigmoid', '/model/model/model.20/cv2/act/Sigmoid', '/model/model/model.3/act/Mul', '/model/model/model.4/m/m.0/cv1/act/Sigmoid', '/model/model/model.7/act/Sigmoid', '/model/model/model.5/act/Mul', '/model/model/model.6/m/m.0/cv1/act/Mul', '/model/model/model.23/cv1/act/Mul', '/model/model/model.6/m/m.0/cv2/act/Sigmoid', '/model/model/model.13/cv1/act/Mul', '/model/model/model.20/m/m.0/cv1/act/Sigmoid', '/model/model/model.1/act/Sigmoid', '/model/model/model.17/cv2/act/Mul', '/model/model/model.17/m/m.0/cv1/act/Mul', '/model/model/model.13/cv3/act/Sigmoid', '/model/model/model.20/m/m.0/cv2/act/Sigmoid', '/model/model/model.4/m/m.0/cv2/act/Mul', '/model/model/model.23/cv3/act/Sigmoid', '/model/model/model.20/cv1/act/Mul', '/model/model/model.20/m/m.0/cv2/act/Mul', '/model/model/model.23/cv3/act/Mul', '/model/model/model.4/m/m.1/cv1/act/Sigmoid', '/model/model/model.23/m/m.0/cv2/act/Mul', '/model/model/model.6/m/m.0/cv1/act/Sigmoid', '/model/model/model.1/act/Mul', '/model/model/model.6/cv2/act/Sigmoid', '/model/model/model.4/m/m.1/cv2/act/Sigmoid', '/model/model/model.8/m/m.0/cv2/act/Mul', '/model/model/model.6/cv1/act/Sigmoid', '/model/model/model.17/cv3/act/Mul', '/model/model/model.10/act/Mul', '/model/model/model.20/cv2/act/Mul', '/model/model/model.2/cv2/act/Mul', '/model/model/model.4/cv3/act/Sigmoid', '/model/model/model.8/m/m.0/cv2/act/Sigmoid', '/model/model/model.9/cv1/act/Sigmoid', '/model/model/model.6/cv2/act/Mul', '/model/model/model.9/cv1/act/Mul', '/model/model/model.23/cv2/act/Mul', '/model/model/model.17/cv1/act/Sigmoid', '/model/model/model.20/cv3/act/Mul', '/model/model/model.21/act/Sigmoid', '/model/model/model.6/m/m.1/cv2/act/Mul', '/model/model/model.8/cv1/act/Mul', '/model/model/model.5/act/Sigmoid', '/model/model/model.14/act/Sigmoid', '/model/model/model.2/cv1/act/Sigmoid', '/model/model/model.8/m/m.0/cv1/act/Mul', '/model/model/model.20/cv1/act/Sigmoid', '/model/model/model.6/cv3/act/Mul', '/model/model/model.10/act/Sigmoid', '/model/model/model.2/m/m.0/cv2/act/Sigmoid', '/model/model/model.13/m/m.0/cv2/act/Mul', '/model/model/model.7/act/Mul', '/model/model/model.6/m/m.1/cv1/act/Sigmoid', '/model/model/model.9/cv2/act/Sigmoid', '/model/model/model.9/cv2/act/Mul', '/model/model/model.4/m/m.1/cv2/act/Mul', '/model/model/model.2/cv3/act/Mul', '/model/model/model.6/m/m.2/cv1/act/Mul', '/model/model/model.6/m/m.2/cv2/act/Mul', '/model/model/model.23/m/m.0/cv2/act/Sigmoid', '/model/model/model.4/cv1/act/Sigmoid', '/model/model/model.17/cv2/act/Sigmoid', '/model/model/model.23/cv1/act/Sigmoid', '/model/model/model.6/m/m.1/cv1/act/Mul', '/model/model/model.4/cv3/act/Mul', '/model/model/model.2/m/m.0/cv1/act/Mul', '/model/model/model.14/act/Mul', '/model/model/model.8/cv2/act/Sigmoid', '/model/model/model.2/cv3/act/Sigmoid', '/model/model/model.17/m/m.0/cv2/act/Mul', '/model/model/model.0/act/Mul', '/model/model/model.2/cv2/act/Sigmoid', '/model/model/model.3/act/Sigmoid', '/model/model/model.8/cv1/act/Sigmoid', '/model/model/model.0/act/Sigmoid', '/model/model/model.6/cv3/act/Sigmoid', '/model/model/model.13/cv2/act/Mul', '/model/model/model.18/act/Sigmoid', '/model/model/model.13/m/m.0/cv1/act/Sigmoid', '/model/model/model.4/cv2/act/Mul', '/model/model/model.13/m/m.0/cv1/act/Mul', '/model/model/model.4/m/m.0/cv2/act/Sigmoid', '/model/model/model.13/m/m.0/cv2/act/Sigmoid', '/model/model/model.6/cv1/act/Mul', '/model/model/model.8/m/m.0/cv1/act/Sigmoid', '/model/model/model.23/m/m.0/cv1/act/Mul', '/model/model/model.13/cv2/act/Sigmoid', '/model/model/model.17/m/m.0/cv1/act/Sigmoid', '/model/model/model.4/m/m.0/cv1/act/Mul', '/model/model/model.2/m/m.0/cv1/act/Sigmoid', '/model/model/model.2/cv1/act/Mul', '/model/model/model.21/act/Mul', '/model/model/model.8/cv3/act/Sigmoid', '/model/model/model.8/cv3/act/Mul', '/model/model/model.6/m/m.1/cv2/act/Sigmoid', '/model/model/model.23/cv2/act/Sigmoid', '/model/model/model.13/cv1/act/Sigmoid', '/model/model/model.13/cv3/act/Mul']
2026-06-19 03:53 - INFO - sdk - quantize - Starting quantization...
WARNING:root:Please use QuantFormat.QDQ for activation type QInt8 and weight type QInt8. Or it will lead to bad performance on x64.
2026-06-19 03:53 - INFO - sdk - quantize - Quantization completed! Quantized model saved to /quadric/sdk-cli/examples/models/yolo/yolov5/yolov5n-backbone_OpSet16_optimized_asym_int8_q.onnx
2026-06-19 03:53 - INFO - sdk - quantize - ONNX full precision model size: 7.18MB
2026-06-19 03:53 - INFO - sdk - quantize - ONNX quantized model size: 1.94MB
2026-06-19 03:53 - INFO - sdk - quantize - ONNX model shapes inferred.
2026-06-19 03:53 - INFO - sdk - quantize - ONNX Model with well-defined shapes has been saved at `/quadric/sdk-cli/examples/models/yolo/yolov5/yolov5n-backbone_OpSet16_optimized_asym_int8_q_shaped.onnx`.
2026-06-19 03:53 - DEBUG - sdk - quantize - Checking for FLOAT/FLOAT16 types...
2026-06-19 03:53 - INFO - sdk - quantize - Checking for remaining FLOAT/FLOAT16 types.
2026-06-19 03:53 - INFO - sdk - quantize - Model still has FLOAT/FLOAT16 types after quantization. Creating ranges for floating point tensors using calibration data...
2026-06-19 03:53 - INFO - sdk - quantize - Saved computed tensor ranges to /quadric/sdk-cli/examples/models/yolo/yolov5/yolov5n-backbone_OpSet16_optimized_asym_int8_q_shaped.tranges.
2026-06-19 03:53 - INFO - sdk - quantize -
╒═════════════════════════════════════════════════════════════════════════════════════════════════════════╤════════════════════════════════════════════════════════════════════════════════════════════════════════════╕
│ Quantized ONNX Model │ Tensor Ranges File │
╞═════════════════════════════════════════════════════════════════════════════════════════════════════════╪════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
│ /quadric/sdk-cli/examples/models/yolo/yolov5/yolov5n-backbone_OpSet16_optimized_asym_int8_q_shaped.onnx │ /quadric/sdk-cli/examples/models/yolo/yolov5/yolov5n-backbone_OpSet16_optimized_asym_int8_q_shaped.tranges │
╘═════════════════════════════════════════════════════════════════════════════════════════════════════════╧════════════════════════════════════════════════════════════════════════════════════════════════════════════╛
Quantized ONNX: yolov5n-backbone_OpSet16_optimized_asym_int8_q_shaped.onnx
Tensor ranges: yolov5n-backbone_OpSet16_optimized_asym_int8_q_shaped.tranges
1508
5. Compilation
ChimeraJob wraps the full compile pipeline: graph analyze, kernel selection, memory planning, and code generation.
hw_config = DEFAULT_32_ARRAY_SIZE
cgc_job = ChimeraJob(
model_p=str(quantized_onnx_model.model_path),
**hw_config.to_dict(),
)
cgc_job.compile(quiet=True)
/tmp/ipykernel_2644/657128748.py:3: DeprecationWarning: Specifying hardware configuration through individual parameters is deprecated. Please use hw_config parameter instead. Example: hw_cfg = HWConfig(product='QC-U', ocm_size='16MB'); ChimeraJob('model.onnx', hw_config=hw_cfg)
cgc_job = ChimeraJob(
6. Inference
batch_inference runs the compiled backbone against multiple images in parallel — once through ORT INT8 (reference) and once through ISS INT8 (the Chimera GPNPU's cycle-accurate simulator). Both drive the same head ONNX on the host for the classical tail.
all_image_paths, all_images = load_images(INPUT_SIZE, DEFAULT_IMAGE_PATHS)
num_images = len(all_images)
engines = {
InferenceEngine.CHIMERA_ORT_INT8: cgc_job,
InferenceEngine.CHIMERA_ISS_INT8: cgc_job,
}
head_session = InferenceSession(onnx.load(str(head_onnx_file)).SerializeToString())
outputs_per_engine = {}
threads = min(num_images, 6)
for engine, job in engines.items():
outputs_per_engine[engine] = batch_inference(
engine, job, all_images, head_session=head_session, threads=threads
)
2026-06-19 03:58 - WARNING - epu - chimera_job - ORT is not threadsafe -- forcing single threaded batch execution
100%|████████████████████████████████████████████| 3/3 [00:00<00:00, 5.23it/s]
Processing: 100%|████████████████████████████████| 3/3 [00:51<00:00, 17.25s/it]
7. Run Statistics
plot_run_statistics() renders the per-kernel cycle breakdown for the compiled backbone — useful for spotting hot kernels when tuning the target configuration.
print(cgc_job)
cgc_job.plot_run_statistics()
╒═════════════════════╤═════════════════════════════════════════════════════════════════════════════════════════════════════════╕
│ Module Name │ yolov5n_backbone_OpSet16_optimized_asym_int8_q_shaped_QC_U_1d7_16MB_4kB_128GBps_128GBps_16_OFF_x1_x1 │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ ONNX File │ /quadric/sdk-cli/examples/models/yolo/yolov5/yolov5n-backbone_OpSet16_optimized_asym_int8_q_shaped.onnx │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Product Target │ QC-U │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Number of Cores │ 1 │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ ISS Clock Frequency │ 1.700 │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ L2M Size │ 16MB │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ LRM Size │ 4kB │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ External Read BW │ 128GBps │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ External Write BW │ 128GBps │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ MACS per PE │ 16 │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Max L2M │ 7.435MB │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Max LRM │ 1.625kB │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Max Temp Ext Bytes │ 0.000MB │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Network GMACs │ 2.234 │
╘═════════════════════╧═════════════════════════════════════════════════════════════════════════════════════════════════════════╛
NOTE: CGC has used general convolution for some nodes, which may result in suboptimal performance. For performance-critical applications, please contact Quadric support to explore optimization strategies.
General convolution nodes: /model/model/model.0/conv/Conv_quant
For more details, see: https://app.quadric.ai/docs/latest/chimera-software-user-guide/chimera-graph-compiler-cgc/graph-optimizations-performed-by-cgc#general-convolution
╒════╤════════╤═════════════════════════════════════════╤══════════════════╤══════════════════════════╤═══════╕
│ │ Type │ Name │ shape │ type │ mse │
╞════╪════════╪═════════════════════════════════════════╪══════════════════╪══════════════════════════╪═══════╡
│ 0 │ Input │ inputs0 │ [1, 3, 640, 640] │ tensor[FixedPoint32<27>] │ n/a │
├────┼────────┼─────────────────────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 1 │ Output │ /model/model/model.24/m.0/Conv_output_0 │ [1, 255, 80, 80] │ tensor[FixedPoint32<26>] │ 0.020 │
├────┼────────┼─────────────────────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 2 │ Output │ /model/model/model.24/m.1/Conv_output_0 │ [1, 255, 40, 40] │ tensor[FixedPoint32<26>] │ 0.012 │
├────┼────────┼─────────────────────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 3 │ Output │ /model/model/model.24/m.2/Conv_output_0 │ [1, 255, 20, 20] │ tensor[FixedPoint32<27>] │ 0.008 │
╘════╧════════╧═════════════════════════════════════════╧══════════════════╧══════════════════════════╧═══════╛
Post-ISS Report 1.7 GHz ***
Fully placed-and-routed gate simulation:
╒══════════════════════════════════╤═════════╕
│ Latency (ms) │ 2.31 │
├──────────────────────────────────┼─────────┤
│ FPS │ 433.47 │
├──────────────────────────────────┼─────────┤
│ Average Power @ 3nm SSGNP (mW) │ 1115.57 │
├──────────────────────────────────┼─────────┤
│ FPS per Watt @ 3nm SSGNP (FPS/W) │ 388.56 │
├──────────────────────────────────┼─────────┤
│ Ext Rd Bytes (MB) │ 6.55 │
├──────────────────────────────────┼─────────┤
│ Ext Wr Bytes (MB) │ 8.17 │
├──────────────────────────────────┼─────────┤
│ Avg Ext Rd BW (GBps) │ 2.77 │
├──────────────────────────────────┼─────────┤
│ Avg Ext Wr BW (GBps) │ 3.46 │
├──────────────────────────────────┼─────────┤
│ MAC Utilization │ 3.48% │
╘══════════════════════════════════╧═════════╛
*** Data generated using 7nm SSGNP gatesim and scaled to 3nm
[SDK-CLI] : TotalCycles: 3,921,882
[SDK-CLI] : Executions/second: 433.47
compute : ▇▇▇▇▇▇▇▇▇▇▇ 576.686K
data_array : ▇▇▇▇▇ 300.393K
mac : ▇▇▇▇▇▇▇ 355.311K
data_external: ▇▇▇ 162.506K
data_ocm : ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 2.526M
for more information check run directory: /quadric/sdk-cli/examples/models/yolo/yolov5/ccl_build/yolov5n_backbone_OpSet16_optimized_asym_int8_q_shaped_QC_U_1d7_16MB_4kB_128GBps_128GBps_16_OFF_x1_x1/run/20260619_035822_010cd3
2026-06-19 03:59 - INFO - epu - chimera_job - Combined plots generated and saved to:
/quadric/sdk-cli/examples/models/yolo/yolov5/ccl_build/yolov5n_backbone_OpSet16_optimized_asym_int8_q_shaped_QC_U_1d7_16MB_4kB_128GBps_128GBps_16_OFF_x1_x1/run/20260619_035822_010cd3/data/yolov5n_backbone_OpSet16_optimized_asym_int8_q_shaped_QC_U_1d7_16MB_4kB_128GBps_128GBps_16_OFF_x1_x1.combined.png
'/quadric/sdk-cli/examples/models/yolo/yolov5/ccl_build/yolov5n_backbone_OpSet16_optimized_asym_int8_q_shaped_QC_U_1d7_16MB_4kB_128GBps_128GBps_16_OFF_x1_x1/run/20260619_035822_010cd3/data'
Decode the network outputs and overlay the detections on the input image.
postprocess_handle = get_postprocess_handle(YOLOModelVariant.YOLOv5)
detections_per_engine = {}
for engine, all_outputs in outputs_per_engine.items():
detections_per_engine[engine] = [
boxes_detections(
postprocess_handle,
np.array(Image.open(path)),
outputs,
INPUT_SIZE,
)[0]
for outputs, path in zip(all_outputs, all_image_paths)
]
display_detections(all_image_paths, detections_per_engine)
Summary
| Model | YOLOv5n (COCO, 640×640, 1.9M params, 4.5 GFLOPs) |
| Pipeline | ONNX download (pre-split) → quantize (INT8 asymmetric) → CGC compile → ISS + ORT inference → YOLOv5 NMS → bbox overlay |
| Compiled on GPNPU | Convolutional backbone |
| On host CPU | YOLOv5 head ONNX, classical NMS, bounding-box visualization |
| Calibration | QuadricCalibration COCO-like subset |
Key takeaways
- Splitting the ONNX at the head boundary keeps CGC focused on GPNPU-friendly convolutional ops while the anchor-decode + NMS tail runs on the host.
- The SDK exposes the same
ChimeraJobhandle to bothCHIMERA_ORT_INT8(ONNX Runtime reference) andCHIMERA_ISS_INT8(ISS), making side-by-side validation a one-dict change. - Asymmetric INT8 quantization on a COCO-like calibration subset preserves detection quality at the default YOLOv5 score threshold.
Citation
@software{yolov5,
title = {YOLOv5 by Ultralytics},
author = {Glenn Jocher},
year = {2020},
version = {7.0},
license = {AGPL-3.0},
url = {https://github.com/ultralytics/yolov5},
doi = {10.5281/zenodo.3908559}
}
