Model Demo: SegFormer Semantic Segmentation

NOTE: The Jupyter Notebook below is included in the Chimera SDK and can be run interactively by running the following CLI command:

$ quadric sdk notebook

From the Jupyter Notebook window in your browser, select the notebook named /quadric/sdk-cli/examples/models/segformer/segformer.ipynb.

SegFormer Semantic Segmentation on Chimera GPNPU

SegFormer pairs a hierarchical Mix Transformer (MiT) backbone with a lightweight all-MLP decode head to turn an image into a dense per-pixel label map. This notebook compiles a Cityscapes-trained SegFormer-B0 end-to-end with the CGC (Chimera Graph Compiler), runs it on the ISS (Instruction Set Simulator), and compares the on-chip segmentation against an ONNX Runtime reference.

Why efficient attention on-chip?

At street-scene resolution, ordinary self-attention is expensive, so SegFormer reduces the spatial size of the keys and values in each transformer stage. CGC keeps that efficient attention — the query/key/value projections, the softmax, and the value aggregation — on the Chimera GPNPU. The one piece it hands back to the developer is the small requantization that follows the key and value projections, which we supply as a compact CCL (Chimera Compute Language) kernel. Everything compiles into a single program that maps a street image to a label map with no host round-trips.

Pipeline

Model: SegFormer-B0 (MiT-B0 backbone, Cityscapes, 512×512 input, 19 classes, 7.84 GMACs)

1. Setup

We need ChimeraJob (the compile-and-run entry point) and a handful of helpers that keep this notebook focused on the pipeline: model download, mmsegmentation-style preprocessing, and the Cityscapes visualization.

import re
from pathlib import Path

from tvm.contrib.epu.chimera_job.chimera_job import ChimeraJob
from tvm.contrib.epu.chimera_job.hw_config import HWConfig

from segformer_helpers import (
    ATTENTION_STUBS,
    compare_segmentations,
    download_model,
    load_input,
    to_class_map,
)

%matplotlib inline

2. Download the Quantized Model

The model has already been exported to ONNX and quantized to symmetric int8 with the Quadric SDK. We fetch the quantized graph and its tensor-ranges file from S3 and go straight to compilation — no calibration step is needed at run time.

model_path, tranges_path = download_model()

Downloading segformer_192x432_cityscapes_sym_int8_q.onnx ...
Downloading segformer_192x432_cityscapes_sym_int8_q.onnx.tranges ...
Quantized SegFormer ready: segformer_192x432_cityscapes_sym_int8_q.onnx

3. The Attention Post-Projection Kernel

SegFormer's efficient attention lands on the GPNPU as a first-class operator. CGC compiles the projections, the QKᵀ scores, the softmax, and the value aggregation directly. The only subgraph it factors out is the requantization that follows the key and value projections — a DequantizeLinear → QuantizeLinear pair. CGC emits a typed stub with the exact scales as template parameters, and we provide the body: a single symmetric rescale over the projected tensor. Because the model is symmetric int8, the requant is just a multiply by input_scale / output_scale.

kernel_src = Path(ATTENTION_STUBS).read_text()
post_k = re.search(
    r"// Compute of the subgraph of ops following K-projection\..*?\n    \}",
    kernel_src,
    re.S,
)
print(post_k.group(0))

// Compute of the subgraph of ops following K-projection.
    template <std::int32_t patchHeight,
              std::int32_t patchWidth,
              std::int32_t replicateFactor,
              std::int32_t postKFuncId,
              typename NDArrayTypeK,
              typename OcmAllocatorType,
              FracRepType dequantize_0_input_scale_frac_bits,
              FracRepType quantize_0_output_scale_frac_bits,
              std::enable_if_t<postKFuncId == 1, int> = 0>
    INLINE void postKFunc(std::int32_t firstPartitionHead,
                          NDArrayTypeK& NDArrayK,
                          OcmAllocatorType& ocmAllocator,
                          FixedPoint32<dequantize_0_input_scale_frac_bits> dequantize_0_input_scale,
                          std::int8_t dequantize_0_input_zero_point,
                          FixedPoint32<quantize_0_output_scale_frac_bits> quantize_0_output_scale,
                          std::int32_t quantize_0_output_zero_point) {
      constexpr FracRepType scaleFB = 28;
      constexpr std::int32_t requantScaleShift =
          scaleFB - (dequantize_0_input_scale_frac_bits - quantize_0_output_scale_frac_bits);
      qVar_t<FixedPoint32<scaleFB>> requantScale = FXDirectAssign(
          math::fxDiv<requantScaleShift>(dequantize_0_input_scale.value, quantize_0_output_scale.value));

      for (std::size_t i = 0; i < NDArrayTypeK::size(); ++i) {
        NDArrayK[i] = nn::quantizeLinear(NDArrayK[i], requantScale);
      }

      (void)firstPartitionHead;
      (void)ocmAllocator;
      (void)dequantize_0_input_zero_point;
      (void)quantize_0_output_zero_point;
    }

4. Compile with CGC

We target QC-U with an 8 MB on-chip memory budget. SegFormer's working set is larger than that, so CGC automatically tiles the graph and streams tensors through OCM — here it splits the run into 103 tiles. We pair the tight budget with disable_prefetch=True, which keeps more of that OCM available for live tensors. The quantized ONNX, its tensor ranges, and the attention kernel are all handed to ChimeraJob; compile() runs the full CGC flow and builds the on-chip program.

cgc_job = ChimeraJob(
    model_p=str(model_path),
    hw_config=HWConfig(product="QC-U", ocm_size="8MB"),
    trange_file=str(tranges_path),
    attn_stub_src_path=ATTENTION_STUBS,
    disable_prefetch=True,
)
cgc_job.compile(quiet=True)
print(cgc_job)

╒═════════════════════╤═══════════════════════════════════════════════════════════════════════════════════════╕
│ Module Name         │ segformer_192x432_cityscapes_sym_int8_q_QC_U_1d7_8MB_4kB_128GBps_128GBps_16_OFF_x1_x1 │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────┤
│ ONNX File           │ segformer_192x432_cityscapes_sym_int8_q.onnx                                          │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────┤
│ Product Target      │ QC-U                                                                                  │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────┤
│ Number of Cores     │ 1                                                                                     │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────┤
│ ISS Clock Frequency │ 1.700                                                                                 │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────┤
│ L2M Size            │ 8MB                                                                                   │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────┤
│ LRM Size            │ 4kB                                                                                   │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────┤
│ External Read BW    │ 128GBps                                                                               │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────┤
│ External Write BW   │ 128GBps                                                                               │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────┤
│ MACS per PE         │ 16                                                                                    │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────┤
│ Max L2M             │ 8.000MB                                                                               │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────┤
│ Max LRM             │ 3.000kB                                                                               │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────┤
│ Max Temp Ext Bytes  │ 20.750MB                                                                              │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────┤
│ Network GMACs       │ 7.838                                                                                 │
╘═════════════════════╧═══════════════════════════════════════════════════════════════════════════════════════╛
NOTE: CGC has used general convolution for some nodes, which may result in suboptimal performance. For performance-critical applications, please contact Quadric support to explore optimization strategies.
General convolution nodes: /backbone/layers.0.0/projection/Conv_quant, /backbone/layers.0.1.0/attn/sr/Conv_quant, /backbone/layers.0.1.1/attn/sr/Conv_quant
For more details, see: https://app.quadric.ai/docs/latest/chimera-software-user-guide/chimera-graph-compiler-cgc/graph-optimizations-performed-by-cgc#general-convolution


╒════╤════════╤════════╤══════════════════╤══════════════════════════╤═══════╕
│    │ Type   │ Name   │ shape            │ type                     │ mse   │
╞════╪════════╪════════╪══════════════════╪══════════════════════════╪═══════╡
│  0 │ Input  │ input  │ [1, 3, 512, 512] │ tensor[FixedPoint32<30>] │ n/a   │
├────┼────────┼────────┼──────────────────┼──────────────────────────┼───────┤
│  1 │ Output │ output │ [1, 1, 512, 512] │ tensor[int32]            │ n/a   │
╘════╧════════╧════════╧══════════════════╧══════════════════════════╧═══════╛

5. Run: Host CPU and Chimera GPNPU

We load a street scene, preprocess it the way the training pipeline did (resize to 512×512, normalize with ImageNet statistics), and run it two ways: through ONNX Runtime on the host CPU as a reference, and through the compiled program on the Chimera GPNPU via the ISS.

original_rgb, input_tensor = load_input()
inputs = {"input": input_tensor}

cpu_output = cgc_job.run_onnx_inf_session(inputs)
gpnpu_output = cgc_job.run_inference_harness(inputs, compare_ort=False)

cpu_map = to_class_map(cpu_output["output"])
gpnpu_map = to_class_map(gpnpu_output["output"])
print(
    "Segmentation map:", gpnpu_map.shape, "labels", int(gpnpu_map.min()), "-", int(gpnpu_map.max())
)

2026-07-26 21:37 - INFO - epu - iss_testing - Found tranges for input: <tvm.contrib.epu.interval.Interval object at 0x712913f3fbe0>
/usr/local/lib/python3.10/dist-packages/tvm/relay/backend/contrib/epu/iss_testing.py:119: RuntimeWarning: invalid value encountered in cast
  input_tensor = (input_tensor * (1 << epu_fx_util.frac_bits_from_range(trange))).astype(
2026-07-26 21:37 - INFO - epu - iss_testing - Started Executing Onnxruntime...
2026-07-26 21:37 - INFO - epu - iss_testing - Done 0:00:00.227925
2026-07-26 21:37 - INFO - epu - iss_testing - Found tranges for input: <tvm.contrib.epu.interval.Interval object at 0x712a8e566f50>
FILM 103/103: 100%|███████████████████████████████████████████████| 103/103 [03:19<00:00,  1.94s/it]


Segmentation map: (512, 512) labels 0 - 13

6. Visualization

Both runs produce a 512×512 map of Cityscapes class ids. We colorize each with the standard Cityscapes palette, overlay it on the input, and highlight where the two disagree.

agreement = compare_segmentations(
    original_rgb,
    cpu_map,
    gpnpu_map,
    save_path="figures/segformer_comparison.png",
)
print(f"Host CPU vs Chimera GPNPU per-pixel agreement: {agreement * 100:.2f}%")

Host CPU vs Chimera GPNPU per-pixel agreement: 88.55%

7. Run Statistics

CGC records a per-layer cycle and memory-traffic profile for the ISS run. This is the same data DevStudio surfaces, and it is where you would look to find the hotspots worth tuning.

profile_path = cgc_job.plot_run_statistics()
print("Profile written to", Path(profile_path).name)

2026-07-26 21:41 - INFO - epu - chimera_job - Combined plots generated and saved to: 
/quadric/sdk-cli/examples/models/segformer/ccl_build/segformer_192x432_cityscapes_sym_int8_q_QC_U_1d7_8MB_4kB_128GBps_128GBps_16_OFF_x1_x1/run/20260726_213759_d1bd38/data/segformer_192x432_cityscapes_sym_int8_q_QC_U_1d7_8MB_4kB_128GBps_128GBps_16_OFF_x1_x1.combined.png


Profile written to data

Summary

Key takeaways

1. Efficient attention compiles as a first-class operator. SegFormer's spatial-reduction attention runs on the Chimera GPNPU — projections, softmax, and value aggregation included.
2. The developer surface is tiny. The only hand-written code is a single symmetric requantization loop; CGC supplies the exact scales as template parameters.
3. Dense segmentation stays on-chip. A street image becomes a 512×512 Cityscapes label map in one compiled program, matching the ONNX Runtime reference closely.

Citation

@article{xie2021segformer,
  title={SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers},
  author={Xie, Enze and Wang, Wenhai and Yu, Zhiding and Anandkumar, Anima and Alvarez, Jose M and Luo, Ping},
  journal={Advances in Neural Information Processing Systems},
  volume={34},
  year={2021}
}