NOTE: The Jupyter Notebook below is included in the Chimera SDK and can be run interactively by running the following CLI command:
$ quadric sdk notebook
From the Jupyter Notebook window in your browser, select the notebook named /quadric/sdk-cli/examples/models/mask_rcnn/detector/faster_rcnn.ipynb.
Faster R-CNN for Object Detection
Abstract
State-of-the-art object detection networks depend on region proposal algorithms to hypothesize object locations. Advances like SPPnet and Fast R-CNN have reduced the running time of these detection networks, exposing region proposal computation as a bottleneck. In Faster R-CNN, a Region Proposal Network (RPN) that shares full-image convolutional features with the detection network is introduced, enabling nearly cost-free region proposals. An RPN is a fully convolutional network that simultaneously predicts object bounds and objectness scores at each position. The RPN is trained end-to-end to generate high-quality region proposals, which are used by Fast R-CNN for detection. RPN and Fast R-CNN are further merged into a single network by sharing their convolutional features---using the recently popular terminology of neural networks with 'attention' mechanisms, the RPN component tells the unified network where to look. In ILSVRC and COCO 2015 competitions, Faster R-CNN and RPN were the foundations of the 1st-place winning entries in several tracks.
Please refer papers for details.
- Shaoqing Ren, Kaiming He, Ross Girshick, Jian Sun, "Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks"
Set-Up
import matplotlib.pyplot as plt
import numpy as np
import onnx
import os
from pathlib import Path
from PIL import Image
import torch, torchvision
from torch.utils.data import Subset
from torchvision.transforms import Compose, Normalize, ToTensor
import warnings
from examples.models.zoo.zoo_utils import onnx_check_and_simplify
from tvm.contrib.epu.chimera_job.chimera_job import ChimeraJob
import tvm.contrib.epu.chimera_job.constants as sdk_constants
from sdk_cli.lib.inference import InferenceEngine, batch_inference
from sdk_cli.lib.quantize import (
QuantizedONNXModel,
quantize_onnx_model,
)
from sdk_cli.utils.models.rcnn import RCNNModelVariant
from sdk_cli.node_builtins.classical.rcnn_postprocessing import (
get_rcnn_parameters,
get_rcnn_postprocessor,
rcnn_postprocessing,
)
from sdk_cli.node_builtins.outputs.bbox_label_visualizer import draw_bbox
from sdk_cli.utils.transforms import ResizePad
from sdk_cli.utils.datasets import QuadricCalibration
from sdk_cli.utils.datasets.COCO import COCO91CLASSES
import tvm.contrib.epu.graphutils as gutils
warnings.filterwarnings("ignore")
Model Selection
In this notebook, the following models can be experimented. We are going to use faster_rcnn_800x800 here.
- faster_rcnn_800x800
- faster_rcnn_800x1344
MODEL_NAME = RCNNModelVariant.faster_rcnn_800x800
rcnn_parameters = get_rcnn_parameters(MODEL_NAME)
model_input_size = rcnn_parameters.model_input_size
ONNX Export
Load Model and Export Full ONNX
outputs0: boxesoutputs1: labelsoutputs2: scores
onnx_file = MODEL_NAME + ".onnx"
model = torchvision.models.detection.fasterrcnn_resnet50_fpn_v2(pretrained=True)
x = torch.rand(1, 3, *model_input_size[::-1])
output_edges = ["boxes", "labels", "scores"]
torch.onnx.export(
model,
x, # ONNX requires fixed input size
onnx_file,
do_constant_folding=True,
input_names=["inputs0"],
output_names=output_edges,
opset_version=sdk_constants.DEFAULT_ONNX_OPSET,
)
## load your predefined ONNX model
model = onnx_check_and_simplify(onnx.load(onnx_file))
onnx.save(model, onnx_file)
print(f"Simplified {MODEL_NAME} onnx is saved as {onnx_file}")
Simplified faster_rcnn_800x800 onnx is saved as faster_rcnn_800x800.onnx
Extract Backbone ONNX
## load the ONNX model
model = onnx.load(onnx_file)
backbone_onnx_file = MODEL_NAME + "-backbone.onnx"
util = gutils.CustomOpReplacer(model)
sub_graph, _ = util.extract_subgraph_by_name_matching(["/backbone/.*"])
onnx.save(sub_graph, backbone_onnx_file)
print(f"{MODEL_NAME}'s backbone onnx is saved as {backbone_onnx_file}")
faster_rcnn_800x800's backbone onnx is saved as faster_rcnn_800x800-backbone.onnx
Quantization and Compilation
Quantization
dataset_mean, dataset_std = [0.485, 0.456, 0.406], [0.229, 0.224, 0.225]
transforms = Compose(
[
ResizePad(model_input_size),
ToTensor(),
Normalize(dataset_mean, dataset_std),
]
)
## Path to directory containing data to use for for numerical range calibration during quantization
## Data is used also used to compare accuracy of fp32 and int8 models
dataset = QuadricCalibration.Dataset(transform=transforms)
## NOTE: `coco-like` is the 0th index target for `QuadricCalibration.Dataset`
coco_like_data_indices = [index for index, target in enumerate(dataset.targets) if target == 0]
coco_like_subset_of_dataset = Subset(dataset, coco_like_data_indices)
quantized_onnx_model: QuantizedONNXModel = quantize_onnx_model(
backbone_onnx_file,
coco_like_subset_of_dataset,
asymmetric_activation=True,
)
print(
f"quantized onnx: {quantized_onnx_model.model_path}, tranges file: {quantized_onnx_model.tensor_ranges_path}"
)
2026-06-19 03:53 - INFO - sdk - quantize - ONNX model shapes inferred.
2026-06-19 03:53 - DEBUG - sdk - quantize - Forcing node types: []
2026-06-19 03:53 - DEBUG - sdk - quantize - ONNX Node types excluded from quantization: ['Softmax', 'Sigmoid', 'QuadricCustomOp']
2026-06-19 03:53 - DEBUG - sdk - quantize - ONNX Node names excluded from quantization: []
2026-06-19 03:53 - INFO - sdk - quantize - Starting quantization...
WARNING:root:Please use QuantFormat.QDQ for activation type QInt8 and weight type QInt8. Or it will lead to bad performance on x64.
2026-06-19 03:54 - INFO - sdk - quantize - Quantization completed! Quantized model saved to /quadric/sdk-cli/examples/models/mask_rcnn/detector/faster_rcnn_800x800-backbone_OpSet16_optimized_asym_int8_q.onnx
2026-06-19 03:54 - INFO - sdk - quantize - ONNX full precision model size: 102.38MB
2026-06-19 03:54 - INFO - sdk - quantize - ONNX quantized model size: 25.75MB
2026-06-19 03:54 - INFO - sdk - quantize - ONNX model shapes inferred.
2026-06-19 03:54 - INFO - sdk - quantize - ONNX Model with well-defined shapes has been saved at `/quadric/sdk-cli/examples/models/mask_rcnn/detector/faster_rcnn_800x800-backbone_OpSet16_optimized_asym_int8_q_shaped.onnx`.
2026-06-19 03:54 - DEBUG - sdk - quantize - Checking for FLOAT/FLOAT16 types...
2026-06-19 03:54 - INFO - sdk - quantize - Checking for remaining FLOAT/FLOAT16 types.
2026-06-19 03:54 - INFO - sdk - quantize - Model still has FLOAT/FLOAT16 types after quantization. Creating ranges for floating point tensors using calibration data...
2026-06-19 03:54 - INFO - sdk - quantize - Saved computed tensor ranges to /quadric/sdk-cli/examples/models/mask_rcnn/detector/faster_rcnn_800x800-backbone_OpSet16_optimized_asym_int8_q_shaped.tranges.
2026-06-19 03:54 - INFO - sdk - quantize -
╒════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╤═══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╕
│ Quantized ONNX Model │ Tensor Ranges File │
╞════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╪═══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
│ /quadric/sdk-cli/examples/models/mask_rcnn/detector/faster_rcnn_800x800-backbone_OpSet16_optimized_asym_int8_q_shaped.onnx │ /quadric/sdk-cli/examples/models/mask_rcnn/detector/faster_rcnn_800x800-backbone_OpSet16_optimized_asym_int8_q_shaped.tranges │
╘════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╧═══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╛
quantized onnx: /quadric/sdk-cli/examples/models/mask_rcnn/detector/faster_rcnn_800x800-backbone_OpSet16_optimized_asym_int8_q_shaped.onnx, tranges file: /quadric/sdk-cli/examples/models/mask_rcnn/detector/faster_rcnn_800x800-backbone_OpSet16_optimized_asym_int8_q_shaped.tranges
Compilation
cgc_job = ChimeraJob(
model_p=str(quantized_onnx_model.model_path),
trange_file=str(quantized_onnx_model.tensor_ranges_path),
)
cgc_job.compile(quiet=True)
print(cgc_job)
╒═════════════════════╤════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╕
│ Module Name │ faster_rcnn_800x800_backbone_OpSet16_optimized_asym_int8_q_shaped_QC_U_1d7_16MB_4kB_128GBps_128GBps_16_OFF_x1_x1 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ ONNX File │ /quadric/sdk-cli/examples/models/mask_rcnn/detector/faster_rcnn_800x800-backbone_OpSet16_optimized_asym_int8_q_shaped.onnx │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Product Target │ QC-U │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Number of Cores │ 1 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ ISS Clock Frequency │ 1.700 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ L2M Size │ 16MB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ LRM Size │ 4kB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ External Read BW │ 128GBps │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ External Write BW │ 128GBps │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ MACS per PE │ 16 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Max L2M │ 15.976MB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Max LRM │ 3.000kB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Max Temp Ext Bytes │ 36.621MB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Network GMACs │ 88.381 │
╘═════════════════════╧════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╛
╒════╤════════╤═════════════════════════════════════════════════════════════╤════════════════════╤══════════════════════════╤═══════╕
│ │ Type │ Name │ shape │ type │ mse │
╞════╪════════╪═════════════════════════════════════════════════════════════╪════════════════════╪══════════════════════════╪═══════╡
│ 0 │ Input │ /transform/Unsqueeze_12_output_0 │ [1, 3, 800, 800] │ tensor[FixedPoint32<29>] │ n/a │
├────┼────────┼─────────────────────────────────────────────────────────────┼────────────────────┼──────────────────────────┼───────┤
│ 1 │ Output │ /backbone/fpn/layer_blocks.0/layer_blocks.0.0/Conv_output_0 │ [1, 256, 200, 200] │ tensor[FixedPoint32<28>] │ n/a │
├────┼────────┼─────────────────────────────────────────────────────────────┼────────────────────┼──────────────────────────┼───────┤
│ 2 │ Output │ /backbone/fpn/layer_blocks.1/layer_blocks.1.0/Conv_output_0 │ [1, 256, 100, 100] │ tensor[FixedPoint32<29>] │ n/a │
├────┼────────┼─────────────────────────────────────────────────────────────┼────────────────────┼──────────────────────────┼───────┤
│ 3 │ Output │ /backbone/fpn/layer_blocks.2/layer_blocks.2.0/Conv_output_0 │ [1, 256, 50, 50] │ tensor[FixedPoint32<29>] │ n/a │
├────┼────────┼─────────────────────────────────────────────────────────────┼────────────────────┼──────────────────────────┼───────┤
│ 4 │ Output │ /backbone/fpn/layer_blocks.3/layer_blocks.3.0/Conv_output_0 │ [1, 256, 25, 25] │ tensor[FixedPoint32<29>] │ n/a │
├────┼────────┼─────────────────────────────────────────────────────────────┼────────────────────┼──────────────────────────┼───────┤
│ 5 │ Output │ /backbone/fpn/extra_blocks/MaxPool_output_0 │ [1, 256, 13, 13] │ tensor[FixedPoint32<29>] │ n/a │
╘════╧════════╧═════════════════════════════════════════════════════════════╧════════════════════╧══════════════════════════╧═══════╛
Demo
Inference as Batch<a id='Inference-as-Batch'></a>
ChimeraJob supports batch execution for both ort and iss. This invokes specified number of threads to execute inference in parallel.
Here, we invoke multiple threads for images so that the inference on the target images finishes faster.
all_image_paths = [
"../../../common/calibration/face/daniel_maverick.png",
"../../../common/calibration/face/sales_squad_jani.jpg",
"../../../common/calibration/coco-like/33823288584_1d21cf0a26_k.jpeg",
]
all_images = []
for image_path in all_image_paths:
all_images.append(np.expand_dims(transforms(Image.open(image_path)), axis=0))
NUM_IMAGES = len(all_images)
engines = {
InferenceEngine.CHIMERA_ORT_INT8: cgc_job,
InferenceEngine.CHIMERA_ISS_INT8: cgc_job,
}
outputs_per_inference_engine = {}
THREADS = min(NUM_IMAGES, 6)
for inference_engine, engine in engines.items():
all_backbone_outputs = batch_inference(
inference_engine,
engine,
all_images,
threads=THREADS,
)
outputs_per_inference_engine[inference_engine] = all_backbone_outputs
2026-06-19 03:57 - WARNING - epu - chimera_job - ORT is not threadsafe -- forcing single threaded batch execution
100%|████████████████████████████████████████████| 3/3 [00:03<00:00, 1.11s/it]
Processing: 100%|███████████████████████████████| 3/3 [09:45<00:00, 195.31s/it]
Run Statistics for Faster RCNN Backbone
print(cgc_job)
cgc_job.plot_run_statistics()
╒═════════════════════╤════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╕
│ Module Name │ faster_rcnn_800x800_backbone_OpSet16_optimized_asym_int8_q_shaped_QC_U_1d7_16MB_4kB_128GBps_128GBps_16_OFF_x1_x1 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ ONNX File │ /quadric/sdk-cli/examples/models/mask_rcnn/detector/faster_rcnn_800x800-backbone_OpSet16_optimized_asym_int8_q_shaped.onnx │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Product Target │ QC-U │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Number of Cores │ 1 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ ISS Clock Frequency │ 1.700 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ L2M Size │ 16MB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ LRM Size │ 4kB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ External Read BW │ 128GBps │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ External Write BW │ 128GBps │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ MACS per PE │ 16 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Max L2M │ 15.976MB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Max LRM │ 3.000kB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Max Temp Ext Bytes │ 36.621MB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Network GMACs │ 88.381 │
╘═════════════════════╧════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╛
╒════╤════════╤═════════════════════════════════════════════════════════════╤════════════════════╤══════════════════════════╤═══════╕
│ │ Type │ Name │ shape │ type │ mse │
╞════╪════════╪═════════════════════════════════════════════════════════════╪════════════════════╪══════════════════════════╪═══════╡
│ 0 │ Input │ /transform/Unsqueeze_12_output_0 │ [1, 3, 800, 800] │ tensor[FixedPoint32<29>] │ n/a │
├────┼────────┼─────────────────────────────────────────────────────────────┼────────────────────┼──────────────────────────┼───────┤
│ 1 │ Output │ /backbone/fpn/layer_blocks.0/layer_blocks.0.0/Conv_output_0 │ [1, 256, 200, 200] │ tensor[FixedPoint32<28>] │ 0.000 │
├────┼────────┼─────────────────────────────────────────────────────────────┼────────────────────┼──────────────────────────┼───────┤
│ 2 │ Output │ /backbone/fpn/layer_blocks.1/layer_blocks.1.0/Conv_output_0 │ [1, 256, 100, 100] │ tensor[FixedPoint32<29>] │ 0.000 │
├────┼────────┼─────────────────────────────────────────────────────────────┼────────────────────┼──────────────────────────┼───────┤
│ 3 │ Output │ /backbone/fpn/layer_blocks.2/layer_blocks.2.0/Conv_output_0 │ [1, 256, 50, 50] │ tensor[FixedPoint32<29>] │ 0.000 │
├────┼────────┼─────────────────────────────────────────────────────────────┼────────────────────┼──────────────────────────┼───────┤
│ 4 │ Output │ /backbone/fpn/layer_blocks.3/layer_blocks.3.0/Conv_output_0 │ [1, 256, 25, 25] │ tensor[FixedPoint32<29>] │ 0.000 │
├────┼────────┼─────────────────────────────────────────────────────────────┼────────────────────┼──────────────────────────┼───────┤
│ 5 │ Output │ /backbone/fpn/extra_blocks/MaxPool_output_0 │ [1, 256, 13, 13] │ tensor[FixedPoint32<29>] │ 0.000 │
╘════╧════════╧═════════════════════════════════════════════════════════════╧════════════════════╧══════════════════════════╧═══════╛
Post-ISS Report 1.7 GHz ***
Fully placed-and-routed gate simulation:
╒══════════════════════════════════╤═════════╕
│ Latency (ms) │ 9.84 │
├──────────────────────────────────┼─────────┤
│ FPS │ 101.64 │
├──────────────────────────────────┼─────────┤
│ Average Power @ 3nm SSGNP (mW) │ 2221.76 │
├──────────────────────────────────┼─────────┤
│ FPS per Watt @ 3nm SSGNP (FPS/W) │ 45.75 │
├──────────────────────────────────┼─────────┤
│ Ext Rd Bytes (MB) │ 129.69 │
├──────────────────────────────────┼─────────┤
│ Ext Wr Bytes (MB) │ 139.48 │
├──────────────────────────────────┼─────────┤
│ Avg Ext Rd BW (GBps) │ 12.87 │
├──────────────────────────────────┼─────────┤
│ Avg Ext Wr BW (GBps) │ 13.84 │
├──────────────────────────────────┼─────────┤
│ MAC Utilization │ 32.25% │
╘══════════════════════════════════╧═════════╛
*** Data generated using 7nm SSGNP gatesim and scaled to 3nm
[SDK-CLI] : TotalCycles: 16,725,643
[SDK-CLI] : Executions/second: 101.64
compute : ▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 2.605M
data_array : ▇▇▇▇▇▇▇▇▇ 1.669M
mac : ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 8.772M
data_external: ▇▇▇▇▇▇▇▇▇▇ 1.841M
data_ocm : ▇▇▇▇▇▇▇▇▇ 1.592M
for more information check run directory: /quadric/sdk-cli/examples/models/mask_rcnn/detector/ccl_build/faster_rcnn_800x800_backbone_OpSet16_optimized_asym_int8_q_shaped_QC_U_1d7_16MB_4kB_128GBps_128GBps_16_OFF_x1_x1/run/20260619_035729_c86e83
2026-06-19 04:07 - INFO - epu - chimera_job - Combined plots generated and saved to:
/quadric/sdk-cli/examples/models/mask_rcnn/detector/ccl_build/faster_rcnn_800x800_backbone_OpSet16_optimized_asym_int8_q_shaped_QC_U_1d7_16MB_4kB_128GBps_128GBps_16_OFF_x1_x1/run/20260619_035729_c86e83/data/faster_rcnn_800x800_backbone_OpSet16_optimized_asym_int8_q_shaped_QC_U_1d7_16MB_4kB_128GBps_128GBps_16_OFF_x1_x1.combined.png
'/quadric/sdk-cli/examples/models/mask_rcnn/detector/ccl_build/faster_rcnn_800x800_backbone_OpSet16_optimized_asym_int8_q_shaped_QC_U_1d7_16MB_4kB_128GBps_128GBps_16_OFF_x1_x1/run/20260619_035729_c86e83/data'
Display Inference Results
Postprocessing
At first, post-processing for inference results is done here to retrieve bounding boxes and segmentation masks.
faster_rcnn_postprocessor = get_rcnn_postprocessor(rcnn_parameters.postprocess_model)
detections_per_inference_engine = {}
for inference_engine, all_outputs in outputs_per_inference_engine.items():
all_detections = []
all_masks = []
for outputs, image in zip(all_outputs, all_image_paths):
detections = rcnn_postprocessing(
outputs,
model_input_size,
Image.open(image).size,
postprocessor=faster_rcnn_postprocessor,
score_threshold=rcnn_parameters.score_threshold,
)[0]
all_detections.append(detections)
detections_per_inference_engine[inference_engine] = all_detections
Display Detected Objects
The following shows detected objects with bounding boxes.
%matplotlib inline
pic_len = len(engines) + 1
for i in range(len(all_image_paths)):
ax, idx = {}, 1
fig = plt.figure(figsize=(6 * pic_len, 6), tight_layout=True)
ax[idx] = fig.add_subplot(int("1%s%s" % (pic_len, idx)))
ax[idx].imshow(Image.open(all_image_paths[i]))
ax[idx].set_title("Original Image")
ax[idx].axis("off")
for inference_engine, bboxes in detections_per_inference_engine.items():
idx += 1
ax[idx] = fig.add_subplot(int("1%s%s" % (pic_len, idx)))
detected_frame = draw_bbox(
np.array(Image.open(all_image_paths[i])), bboxes[i], classes=COCO91CLASSES
)
ax[idx].imshow(detected_frame)
ax[idx].set_title(f"{MODEL_NAME}: {str(inference_engine).upper()}")
ax[idx].axis("off")
fig.show()
