Unlock this feature

This feature isn’t part of your plan yet

Contact sales to get upgraded to the full DevStudio experience.

Unlock this feature

This feature isn't part of your plan yet.

Tutorial: MatMulNBits Custom Op Replacement


NOTE: The Jupyter Notebook below is included in the Chimera SDK and can be run interactively by running the following CLI command:

$ quadric sdk notebook

From the Jupyter Notebook window in your browser, select the notebook named /quadric/sdk-cli/examples/custom_op/MatMulNBits_custom_op_tutorial.ipynb.


MatMulNBits with the shipped plugin bundle

MatMulNBits is ONNX Runtime's 4-bit weight matmul (com.microsoft domain): the weights are packed two per byte with a float scale per block of block_size values along K. It is not in CGC's convert map. The plugin's shipped library carries a bundle for it, matmulnbits-const: it matches every MatMulNBits[bits=4] node, dequantizes the packed weights into one dense float B with the dequant_matmulnbits builtin, and binds the node to quadric_plugin::matMulNBitsStub. The library applies to every compile, so nothing in this notebook names the bundle, rewrites the graph or passes a header. The one line that differs from a plain compile is which ChimeraJob is imported.

Model: matmulnbits_single_layer.onnx, one node, Y = A @ B with K=384 and N=1152.

#partwhat it does
1SetupPaths, and the one node with its attributes.
2Tensor rangesThe .tranges file CGC sizes the fixed-point input and output from.
3CompileThe plugin's ChimeraJob: the shipped bundle replaces the node, then CGC compiles.
4ISS against ONNX RuntimeThe same random input through both; RMSE, correlation and max abs error.

1. Setup

Imports the plugin's ChimeraJob and reads the model. Creates _plugin_out_matmulnbits/ beside the notebook for the tensor ranges and the substituted graph. Prints the model's opset and the MatMulNBits node's K, N, bits and block_size attributes and its inputs: the activation A, the packed weights B and the per-block scales.

import json

import numpy as np
import onnx
import onnxruntime as ort
from onnx import helper
from tvm.contrib.epu.chimera_job.hw_config import HWConfig

from sdk_cli.cgc_plugin.chimera_job import ChimeraJob
from sdk_cli.utils.model_helpers import example_path

model_path = example_path("custom_op/matmulnbits_single_layer.onnx")
work_dir = example_path("custom_op") / "_plugin_out_matmulnbits"
work_dir.mkdir(exist_ok=True)
tranges_path = work_dir / "matmulnbits_single_layer.tranges"

model = onnx.load(str(model_path))
node = next(n for n in model.graph.node if n.op_type == "MatMulNBits")
attrs = {a.name: helper.get_attribute_value(a) for a in node.attribute}
K, N = attrs["K"], attrs["N"]
opset = next(o.version for o in model.opset_import if o.domain in ("", "ai.onnx"))
print(f"{model_path.name}: opset {opset}, {len(model.graph.node)} node")
print(
    f"{node.op_type} ({node.domain}): K={K} N={N} bits={attrs['bits']} block_size={attrs['block_size']}"
)
print(f"inputs {list(node.input)} -> outputs {list(node.output)}")
matmulnbits_single_layer.onnx: opset 21, 1 node
MatMulNBits (com.microsoft): K=384 N=1152 bits=4 block_size=128
inputs ['A', 'B', 'scales'] -> outputs ['Y']

2. Tensor ranges

The model is not calibrated, so the ranges are written here. A is declared in [-10, 10]. Y's range is 1.5 times the largest output magnitude ONNX Runtime produces over ten random inputs drawn from that range. The two ranges are written as JSON to the .tranges path.

session = ort.InferenceSession(str(model_path), providers=["CPUExecutionProvider"])
rng = np.random.default_rng(0)
max_abs_y = 0.0
for _ in range(10):
    sample = rng.uniform(-10.0, 10.0, (1, 1, K)).astype(np.float32)
    max_abs_y = max(max_abs_y, float(np.abs(session.run(None, {"A": sample})[0]).max()))

tranges = {"A": [-10.0, 10.0], "Y": [-1.5 * max_abs_y, 1.5 * max_abs_y]}
tranges_path.write_text(json.dumps(tranges, indent=2))
print(f"A range: {tranges['A']}")
print(f"Y range: [{tranges['Y'][0]:.4f}, {tranges['Y'][1]:.4f}]")
A range: [-10.0, 10.0]
Y range: [-4576.8446, 4576.8446]

3. Compile

Builds the plugin's ChimeraJob on the original model with a QC-N HWConfig and the ranges. The constructor applies the shipped library: matmulnbits-const replaces the node, and the dequantized weights become the customOp's one constant operand. onnx_ort_override_p keeps the ONNX Runtime reference on the original graph. Prints plugin_result, the substituted graph's nodes, and the job summary after compile(). The assertion pins the matched bundle.

cgc_job = ChimeraJob(
    str(model_path),
    hw_config=HWConfig(product="QC-N"),
    trange_file=str(tranges_path),
    bypass_quantization_checks=True,
    onnx_ort_override_p=str(model_path),
    plugin_out_dir=str(work_dir),
)
print(cgc_job.plugin_result)
assert cgc_job.plugin_result.matched_plugins == ("matmulnbits-const",), cgc_job.plugin_result

substituted = onnx.load(cgc_job.plugin_result.model_path)
for n in substituted.graph.node:
    print(f"{n.op_type} ({n.name}): inputs={list(n.input)} -> outputs={list(n.output)}")

cgc_job.compile(quiet=True)
print(cgc_job)
2026-10-23 00:55 - INFO - epu - chimera_job - Overriding onnx model ort execution with the onnx model: /quadric/sdk-cli/examples/custom_op/matmulnbits_single_layer.onnx


cgc plugin: 1 customOp(s) from matmulnbits-const -> /quadric/sdk-cli/examples/custom_op/_plugin_out_matmulnbits
QuadricCustomOp (CustomOp/quadric_plugin::matMulNBitsStub0): inputs=['A', 'B'] -> outputs=['Y']

╒═════════════════════╤═══════════════════════════════════════════════════════════════════════════════════════════╕
│ Module Name         │ matmulnbits_single_layer_QC_N_1d7_16MB_4kB_128GBps_128GBps_16_OFF_x1_x1                   │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ ONNX File           │ /quadric/sdk-cli/examples/custom_op/_plugin_out_matmulnbits/matmulnbits_single_layer.onnx │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ Custom Ops          │ /quadric/sdk-cli/examples/custom_op/_plugin_out_matmulnbits/custom_ops.hpp                │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ Product Target      │ QC-N                                                                                      │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ Number of Cores     │ 1                                                                                         │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ ISS Clock Frequency │ 1.700                                                                                     │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ L2M Size            │ 16MB                                                                                      │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ LRM Size            │ 4kB                                                                                       │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ External Read BW    │ 128GBps                                                                                   │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ External Write BW   │ 128GBps                                                                                   │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ MACS per PE         │ 16                                                                                        │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ Max L2M             │ 0.000MB                                                                                   │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ Max LRM             │ 0.000kB                                                                                   │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ Max Temp Ext Bytes  │ 0.004MB                                                                                   │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ Network GMACs       │                                                                                           │
╘═════════════════════╧═══════════════════════════════════════════════════════════════════════════════════════════╛

╒════╤════════╤════════╤══════════════╤══════════════════════════╤═══════╕
│    │ Type   │ Name   │ shape        │ type                     │ mse   │
╞════╪════════╪════════╪══════════════╪══════════════════════════╪═══════╡
│  0 │ Input  │ A      │ [1, 1, 384]  │ tensor[FixedPoint32<27>] │ n/a   │
├────┼────────┼────────┼──────────────┼──────────────────────────┼───────┤
│  1 │ Output │ Y      │ [1, 1, 1152] │ tensor[FixedPoint32<16>] │ n/a   │
╘════╧════════╧════════╧══════════════╧══════════════════════════╧═══════╛

4. ISS against ONNX Runtime

Draws one input from the declared range and runs it through ONNX Runtime on the original model and through the ISS on the compiled one. Prints both output ranges, the RMSE, the correlation and the max abs error. The assertion requires RMSE below 1e-3.

inputs = {"A": rng.uniform(-10.0, 10.0, (1, 1, K)).astype(np.float32)}
ort_y = cgc_job.run_onnx_inf_session(inputs=inputs)["Y"]
iss_y = np.asarray(cgc_job.run_inference_harness(inputs=inputs, compare_ort=False)["Y"]).reshape(
    ort_y.shape
)

diff = (iss_y - ort_y).ravel()
rmse = float(np.sqrt(np.mean(diff**2)))
corr = float(np.corrcoef(ort_y.ravel(), iss_y.ravel())[0, 1])
print(f"ORT output range: [{ort_y.min():.4f}, {ort_y.max():.4f}]")
print(f"ISS output range: [{iss_y.min():.4f}, {iss_y.max():.4f}]")
print(f"RMSE:          {rmse:.4f}")
print(f"Correlation:   {corr:.6f}")
print(f"Max abs error: {np.abs(diff).max():.4f}")

assert rmse < 1e-3, f"ISS differs from ONNX Runtime: RMSE {rmse:.4e}"
2026-10-23 00:55 - INFO - epu - iss_testing - Found tranges for input: <tvm.contrib.epu.interval.Interval object at 0x7fefb13a1de0>
2026-10-23 00:55 - INFO - epu - iss_testing - Started Executing Onnxruntime...
2026-10-23 00:55 - INFO - epu - chimera_job - running ort reference with overriden model: /quadric/sdk-cli/examples/custom_op/matmulnbits_single_layer.onnx
2026-10-23 00:55 - INFO - epu - iss_testing - Done 0:00:00.039360
2026-10-23 00:55 - INFO - epu - iss_testing - Found tranges for input: <tvm.contrib.epu.interval.Interval object at 0x7ff14c2871c0>
FILM 1/1: 100%|██████████████████████████████████████████████████████| 1/1 [00:00<00:00, 491.08it/s]


ORT output range: [-2196.6162, 2027.6128]
ISS output range: [-2196.6162, 2027.6128]
RMSE:          0.0001
Correlation:   1.000000
Max abs error: 0.0004

Sign in to your account

Don't have an account? 
By signing in, you are agreeing to our Terms of Use and Privacy Policy.
Quadric // One architecture. Every algorithm.

Develop.

Simulate.

Profile.

Collaborate.