Model Demo: BEVFormer End-to-End 3D Detection

NOTE: The Jupyter Notebook below is included in the Chimera SDK and can be run interactively by running the following CLI command:

$ quadric sdk notebook

From the Jupyter Notebook window in your browser, select the notebook named /quadric/sdk-cli/examples/models/bevformer/e2e/bevformer_e2e_guide.ipynb.

BEVFormer INT8 — Full End-to-End on GPNPU

BEVFormer (Li et al., ECCV 2022) is a camera-only 3D object detection model for autonomous driving. It takes 6 surround-view camera images and produces 3D bounding boxes in bird's-eye view.

This notebook walks through compiling and running the entire BEVFormer-Tiny INT8 pipeline on the Quadric GPNPU as a single kernel, from raw camera images to 3D detection outputs:

6 camera images [6, 3, 480, 800]
    │
    ▼
┌─────────────────────────────────────────────┐
│  ResNet-50 + FPN (ONNX → CGC)              │
│  backboneBridge (NCHW→NHWC + quantize, CCL)│
│  3-Layer Encoder (TSA → SCA → FFN, CCL)    │
│  6-Layer Decoder (SA → XAttn → FFN, CCL)   │  ← Single GPNPU Kernel
│  cls_branch + reg_branch (CCL)             │
└─────────────────────────────────────────────┘
    │  cls [900, 10]  reg [900, 10]
    ▼
┌─────────────────────────┐
│  Post-Processing (host) │
│  sigmoid, denormalize,  │
│  top-k, thresholding    │
└─────────────────────────┘
    │
    ▼
  3D Bounding Boxes
  (position, size, rotation, class, score)

A Note on the Current ChiPy Workflow

You may notice that this example re-implements the model's forward logic in PyTorch (nn.Module classes) rather than directly importing the original model. It also requires manually extracting and passing weights from the checkpoint. This is a known limitation of the current ChiPy workflow — in the future, ChiPy will be able to accept PyTorch modules directly and handle checkpoint weights automatically, eliminating this boilerplate.

That said, even without those improvements, this ChiPy-based workflow provides meaningful advantages over the traditional ONNX → CGC path:

• Cleaner custom op boundaries. A model like BEVFormer has complex attention mechanisms (deformable attention, multi-head self-attention, voxel scatter-add) that produce a tangled web of ONNX ops. Identifying where to draw custom op boundaries in an ONNX graph is difficult and error-prone. With ChiPy, each custom op is an explicit nn.Module with a clear forward() — the boundaries are defined in Python, not reverse-engineered from a graph.

• Easier validation and debugging. You can comment out individual modules, swap in reference implementations, or test sub-chains in isolation — all in Python. With ONNX, the equivalent requires graph cutting tools, manual node manipulation, and re-exporting.

• Built-in calibration. Each op carries a CalibrationPoint that records min/max during a single CPU forward pass. Fixed-point frac_bits are determined automatically — no separate calibration scripts or manual tuning.

Model Architecture

Quantization Strategy (v4 Checkpoint)

• Backbone Conv2d: per-tensor weight quantization (55 layers)
• Encoder attention + FFN: per-channel weight quantization (30 layers)
• Decoder attention: per-tensor weight quantization (54 layers)
• Decoder FFN + detection heads: per-channel weight quantization (48 layers)
• All activations: per-tensor symmetric INT8

Step 1: Download Model Data and Setup

The checkpoint and test vectors are stored on S3. Run this cell once to download them.

from pathlib import Path
from urllib.request import urlretrieve

CKPT_DIR = Path("../checkpoint")
CKPT_DIR.mkdir(exist_ok=True)
S3_BASE = "https://sdk-cli-models.s3.us-east-2.amazonaws.com/bevformer/checkpoint"

for name in [
    "full_model.pth",
    "test_vectors.safetensors",
    "geometry_test_vectors.safetensors",
    "quant_params.pth",
]:
    local = CKPT_DIR / name
    if local.exists():
        print(f"  {name} ({local.stat().st_size / 1e6:.0f} MB) — cached")
    else:
        print(f"  Downloading {name}...")
        urlretrieve(f"{S3_BASE}/{name}", str(local))
        print(f"  {name} ({local.stat().st_size / 1e6:.0f} MB) — done")

  Downloading full_model.pth...
  full_model.pth (38 MB) — done
  Downloading test_vectors.safetensors...
  test_vectors.safetensors (861 MB) — done
  Downloading geometry_test_vectors.safetensors...
  geometry_test_vectors.safetensors (110 MB) — done
  Downloading quant_params.pth...
  quant_params.pth (1 MB) — done

from pathlib import Path

BASE = Path(".").resolve()
ONNX_DIR = BASE / "onnx_exports"

from bevformer_e2e import load_data

ckpt, tv, geo_tv, images_np = load_data()
print(f"Input images:      {images_np.shape}  (6 cameras x 3 channels x 480 x 800)")
print(f"Checkpoint:        {len(ckpt)} keys")
print(f"Test vectors:      {len(tv)} tensors")
print(f"Geometry vectors:  {len(geo_tv)} tensors")

Input images:      (6, 3, 480, 800)  (6 cameras x 3 channels x 480 x 800)
Checkpoint:        1815 keys
Test vectors:      266 tensors
Geometry vectors:  126 tensors

Step 2: Export Backbone ONNX, Prepare Inputs, and Build Calibrated Chain

The ResNet-50 + FPN backbone is exported as a QLinearConv ONNX model:

1. FP32 ONNX — dequantized weights, standard graph topology
2. INT8 ONNX — onnxruntime.quantize_static with exact checkpoint scales via TensorQuantOverrides
3. Tranges — activation ranges captured by running the INT8 model with test vectors

Then prepare() extracts weights from the checkpoint and build_calibrated_chain() assembles the full E2E chain (backbone + encoder + decoder + heads) and runs a single CPU forward pass with CalibrationPoints to determine frac_bits for all encoder/decoder/head layers.

from export_resnet_fpn_onnx import export_backbone_onnx
from bevformer_e2e import prepare, build_calibrated_chain
from encoder import prepare_encoder_inputs
from decoder import prepare_decoder_inputs

## Export backbone ONNX
onnx_path, tranges_path = export_backbone_onnx(ONNX_DIR, ckpt, images_np)
print(f"\nONNX:    {onnx_path.name}  ({onnx_path.stat().st_size / 1e6:.1f} MB)")
print(f"Tranges: {tranges_path.name}")

## Prepare weights and inputs
prep = prepare(ckpt)
enc_inputs = prepare_encoder_inputs(tv, geo_tv, ckpt)
dec_inputs = prepare_decoder_inputs(tv, prep["ref_pts"])

## Build chain and calibrate frac_bits in one pass
print("\nBuilding and calibrating chain...")
chain = build_calibrated_chain(onnx_path, prep, images_np, enc_inputs, dec_inputs)
print("Done.")

[1] Exporting FP32 ONNX ...
  Input:  [6, 3, 480, 800]  Output: [6, 256, 15, 25]
  -> resnet_fpn_fp32.onnx  (98.4 MB)

[2] Quantizing to INT8 ONNX (per-tensor w_scale) ...
  Overrides: 55 x_scales + 55 w_scales (per-tensor) from checkpoint


WARNING:root:Please use QuantFormat.QDQ for activation type QInt8 and weight type QInt8. Or it will lead to bad performance on x64.
2026-06-19 04:17 - INFO - sdk - quantize - ONNX model shapes inferred.
2026-06-19 04:17 - INFO - sdk - quantize - ONNX Model with well-defined shapes has been saved at `/quadric/sdk-cli/examples/models/bevformer/e2e/onnx_exports/resnet_fpn_int8.onnx`.


  -> resnet_fpn_int8.onnx  (24.8 MB)  ops=['DequantizeLinear', 'MaxPool', 'QLinearAdd', 'QLinearConv', 'QuantizeLinear', 'Relu']

[3] Writing tranges (from INT8 model run with test vectors) ...
  Captured actual ranges for 222 INT8 model tensors
  -> resnet_fpn_int8.tranges  (100 entries: 100 from data, 0 from scales)

Artifacts in onnx_exports/
  resnet_fpn_fp32.onnx                           98.4 MB
  resnet_fpn_int8.onnx                           24.8 MB
  resnet_fpn_int8.tranges                        0.0 MB

ONNX:    resnet_fpn_int8.onnx  (24.8 MB)
Tranges: resnet_fpn_int8.tranges

Building and calibrating chain...
Done.

Step 3: Run ChiPy (CPU) Pipeline

Run the full ChiPy pipeline on CPU. This executes the same chain that will be compiled for the GPNPU (ONNX backbone + backboneBridge + encoder/decoder/heads) but evaluated in Python. The ChiPy (CPU) output serves as the reference for validating ChiPy (GPNPU) accuracy.

from bevformer_e2e import run_chipy_pipeline, compare_detections_summary
from postprocess import print_detections

## Run ChiPy (CPU) pipeline (same chain as ChiPy (GPNPU), executed on CPU)
print("Running ChiPy (CPU) pipeline...")
det_chipy, cls_chipy, bbox_chipy, ref_pts = run_chipy_pipeline(
    chain, images_np, prep, enc_inputs, dec_inputs
)
print(f"  {len(det_chipy['scores'])} detections\n")
print("ChiPy (CPU) detections:")
print_detections(det_chipy)

Running ChiPy (CPU) pipeline...
  94 detections

ChiPy (CPU) detections:

============================================================
Detections: 94 total
============================================================
  #  1  car                   score=0.9111  pos=(5.5, -12.5, -1.4)  size=(1.8, 4.3, 1.6)  rot=-0.04rad
  #  2  car                   score=0.8250  pos=(-7.3, -28.5, -0.7)  size=(1.9, 4.3, 1.7)  rot=-3.11rad
  #  3  car                   score=0.8031  pos=(-8.3, -8.9, -0.8)  size=(2.0, 4.6, 1.7)  rot=-1.61rad
  #  4  car                   score=0.8031  pos=(-19.3, -5.3, 0.3)  size=(1.9, 4.5, 1.5)  rot=-1.56rad
  #  5  car                   score=0.7968  pos=(-17.3, -10.3, -0.1)  size=(1.9, 4.3, 1.8)  rot=1.66rad
  #  6  car                   score=0.7046  pos=(0.0, 6.7, -1.0)  size=(1.8, 4.4, 1.6)  rot=3.12rad
  #  7  car                   score=0.6710  pos=(-6.0, -28.4, -1.0)  size=(1.9, 4.7, 1.7)  rot=3.12rad
  #  8  car                   score=0.5106  pos=(-1.9, -5.1, -1.1)  size=(1.9, 4.5, 1.7)  rot=-3.12rad
  #  9  pedestrian            score=0.4739  pos=(9.7, 16.1, -1.1)  size=(0.6, 0.7, 1.7)  rot=1.71rad
  # 10  car                   score=0.3662  pos=(-19.0, -17.7, 0.2)  size=(1.9, 4.5, 1.6)  rot=-1.55rad
  ... and 84 more

Step 4: Run Full E2E on ISS — 4-Core Multicore

Run the single-kernel chain on 4 GPNPU cores. The multicore CCL ops (matmulDqPCQBiasOut, addMC, layerNormWidthMC, deformableAttention, etc.) automatically split work across cores:

import logging

from bevformer_e2e import run_iss_pipeline
from tvm.contrib.epu.chimera_job.hw_config import HWConfig

## Suppress verbose TVM/compiler logs during ChiPy (GPNPU) compilation
logging.getLogger("epu").setLevel(logging.WARNING)

hw_config_4c = HWConfig(num_cores=4, macs_per_pe=16, ocm_size="8MB")

## NOTE: do NOT wrap this in contextlib.redirect_stdout(io.StringIO()). The ISS
## runs as a subprocess that streams a large volume to stdout; capturing it under
## the Jupyter kernel deadlocks the cell. Let it stream normally.
print("Running full E2E on ISS — 4-core (~60 min)...")
det_iss_4c, cls_iss_4c, reg_iss_4c, _ = run_iss_pipeline(
    chain,
    images_np,
    prep,
    enc_inputs,
    dec_inputs,
    tranges_path=tranges_path,
    hw_config=hw_config_4c,
)

print(f"ISS 4-core E2E: {len(det_iss_4c['scores'])} detections above 0.1\n")
print_detections(det_iss_4c)

Running full E2E on ISS — 4-core (~60 min)...
  Single-kernel E2E: 534 parameters
  HWConfig: QC-U 8MB 16MAC x4
  Compiling and running on ISS...


/usr/local/lib/python3.10/dist-packages/tvm/relay/backend/contrib/epu/iss_testing.py:119: RuntimeWarning: invalid value encountered in cast
  input_tensor = (input_tensor * (1 << epu_fx_util.frac_bits_from_range(trange))).astype(


ISS 4-core E2E: 98 detections above 0.1


============================================================
Detections: 98 total
============================================================
  #  1  car                   score=0.8955  pos=(5.4, -12.5, -1.4)  size=(1.8, 4.5, 1.7)  rot=-0.04rad
  #  2  car                   score=0.8280  pos=(-7.4, -28.4, -0.8)  size=(1.9, 4.4, 1.7)  rot=-3.08rad
  #  3  car                   score=0.8260  pos=(-8.3, -8.9, -0.8)  size=(2.0, 4.7, 1.7)  rot=-1.59rad
  #  4  car                   score=0.8233  pos=(-19.4, -5.2, 0.2)  size=(1.9, 4.5, 1.5)  rot=-1.56rad
  #  5  car                   score=0.7677  pos=(-17.4, -10.3, -0.1)  size=(1.9, 4.4, 1.9)  rot=1.65rad
  #  6  car                   score=0.7158  pos=(-0.1, 6.8, -1.0)  size=(1.8, 4.4, 1.6)  rot=-3.11rad
  #  7  car                   score=0.6410  pos=(-6.1, -28.5, -1.1)  size=(1.9, 4.7, 1.7)  rot=2.73rad
  #  8  pedestrian            score=0.4544  pos=(9.8, 16.1, -1.1)  size=(0.6, 0.7, 1.7)  rot=1.72rad
  #  9  car                   score=0.4029  pos=(-0.0, -47.6, -1.9)  size=(1.9, 4.5, 1.6)  rot=-0.02rad
  # 10  car                   score=0.3741  pos=(-2.0, -5.2, -1.2)  size=(1.8, 4.4, 1.6)  rot=-3.13rad
  ... and 88 more

Step 5: Validate 4-Core vs ChiPy (CPU)

Compare the 4-core GPNPU output against the CPU reference from Step 3.

import json
from pathlib import Path

MIN_IOU_MATCH = 8  # require at least 8/10 top detections to match by BEV IoU

print("=" * 60)
print("ChiPy (4-core GPNPU) vs ChiPy (CPU) pipeline")
print("=" * 60)
iou_matched_4c, mean_iou_4c = compare_detections_summary(
    det_iss_4c, det_chipy, "ChiPy (4-core)", "ChiPy (CPU)"
)

passed_4c = iou_matched_4c >= MIN_IOU_MATCH
status_4c = "PASS" if passed_4c else "FAIL"
print(f"\n{'=' * 60}")
print(
    f"[{status_4c}] 4-core E2E: BEV IoU match = {iou_matched_4c}/10"
    f" (threshold: {MIN_IOU_MATCH})"
)
assert passed_4c, f"4-core validation failed: {iou_matched_4c}/10 < {MIN_IOU_MATCH}"

## Cycle count summary. 4-core writes per-core profile_core{0..3}.json —
## core 0's TotalCycles is the synced wall-clock representative.
print(f"\n{'=' * 60}")
print("Cycle count summary")
print("=" * 60)
build_dir = Path(".").resolve() / "ccl_build" / "bevformer_e2e_c4_8mb" / "build"
profile_path = build_dir / "profile_core0.json"
if profile_path.exists():
    ops = json.load(profile_path.open())
    total = sum(o["data"]["TotalCycles"] for o in ops)
    print(f"  4-core: {total:>14,} cycles")
else:
    print("  4-core: profile not found (run Step 4 first)")

============================================================
ChiPy (4-core GPNPU) vs ChiPy (CPU) pipeline
============================================================
  ChiPy (4-core)  98 detections
  ChiPy (CPU)     94 detections
  Top-10 label match: 8/10
  Top-10 score diff:  max=0.071004  mean=0.026697
  Top-10 BEV IoU match: 10/10 (mean IoU=0.867)

============================================================
[PASS] 4-core E2E: BEV IoU match = 10/10 (threshold: 8)

============================================================
Cycle count summary
============================================================
  4-core:     62,502,966 cycles

Step 6: Visualize Detections — ISS INT8 vs Torch Model

Three Figure-7-style views, all on the test_vectors scene (the images the model actually ran on). Each shows the 6 surround cameras (FRONT_LEFT | FRONT | FRONT_RIGHT over BACK_LEFT | BACK | BACK_RIGHT) with 3D boxes projected through the model's own lidar2img, plus a top-down BEV.

Boxes are converted from the decode format [cx, cy, cz, w, l, h, rot] to the visualize format [cx, cy, w, l, cz, h, rot] via to_viz. The ISS view uses a lower threshold (0.2) because it produces sparser detections.

import numpy as np
import matplotlib.pyplot as plt
from visualize import plot_combined_view, unnormalize_images


def to_viz(det):
    """decode_detections [cx,cy,cz,w,l,h,rot,...] -> visualize [cx,cy,w,l,cz,h,rot,...]."""
    b = np.asarray(det["bboxes"], np.float32)
    return {**det, "bboxes": b[:, [0, 1, 4, 3, 2, 5, 6, 7, 8]]}


def filter_by_score(det, thr):
    """Keep detections with score >= thr."""
    m = np.asarray(det["scores"]) >= thr
    return {k: np.asarray(v)[m] for k, v in det.items()}


THR = 0.3
cameras = unnormalize_images(images_np)
lidar2img = geo_tv["frame0:metadata:lidar2img"].float().numpy()
det_iss = filter_by_score(to_viz(det_iss_4c), THR)
det_cpu = filter_by_score(to_viz(det_chipy), THR)

## 1) ISS INT8 (deployed GPNPU) with live per-layer refinement
plot_combined_view(
    raw_cameras=cameras,
    lidar2img=lidar2img,
    lidar_pcd=None,
    det_primary=det_iss,
    label_primary="ISS INT8 (GPNPU)",
    color_primary="#d62728",
    suptitle=f"ISS INT8 / GPNPU (score >= {THR})",
)
plt.show()

## 2) ISS (red) vs ChiPy CPU (blue) — same chain, kernel vs reference execution
plot_combined_view(
    raw_cameras=cameras,
    lidar2img=lidar2img,
    lidar_pcd=None,
    det_primary=det_iss,
    det_compare=det_cpu,
    label_primary="ISS INT8 (GPNPU)",
    label_compare="ChiPy (CPU)",
    color_primary="#d62728",
    color_compare="#1f77b4",
    suptitle=f"ISS vs ChiPy CPU (score >= {THR})",
)
plt.show()