NOTE: The Jupyter Notebook below is included in the Chimera SDK and can be run interactively by running the following CLI command:
$ quadric sdk notebook
From the Jupyter Notebook window in your browser, select the notebook named /quadric/sdk-cli/examples/custom_op/MatMulNBits_custom_op_tutorial.ipynb.
MatMulNBits with the shipped plugin bundle
MatMulNBits is ONNX Runtime's 4-bit weight matmul (com.microsoft domain): the weights are packed two per byte with a float scale per block of block_size values along K. It is not in CGC's convert map. The plugin's shipped library carries a bundle for it, matmulnbits-const: it matches every MatMulNBits[bits=4] node, dequantizes the packed weights into one dense float B with the dequant_matmulnbits builtin, and binds the node to quadric_plugin::matMulNBitsStub. The library applies to every compile, so nothing in this notebook names the bundle, rewrites the graph or passes a header. The one line that differs from a plain compile is which ChimeraJob is imported.
Model: matmulnbits_single_layer.onnx, one node, Y = A @ B with K=384 and N=1152.
| # | part | what it does |
|---|---|---|
| 1 | Setup | Paths, and the one node with its attributes. |
| 2 | Tensor ranges | The .tranges file CGC sizes the fixed-point input and output from. |
| 3 | Compile | The plugin's ChimeraJob: the shipped bundle replaces the node, then CGC compiles. |
| 4 | ISS against ONNX Runtime | The same random input through both; RMSE, correlation and max abs error. |
1. Setup
Imports the plugin's ChimeraJob and reads the model. Creates _plugin_out_matmulnbits/ beside the notebook for the tensor ranges and the substituted graph. Prints the model's opset and the MatMulNBits node's K, N, bits and block_size attributes and its inputs: the activation A, the packed weights B and the per-block scales.
import json
import numpy as np
import onnx
import onnxruntime as ort
from onnx import helper
from tvm.contrib.epu.chimera_job.hw_config import HWConfig
from sdk_cli.cgc_plugin.chimera_job import ChimeraJob
from sdk_cli.utils.model_helpers import example_path
model_path = example_path("custom_op/matmulnbits_single_layer.onnx")
work_dir = example_path("custom_op") / "_plugin_out_matmulnbits"
work_dir.mkdir(exist_ok=True)
tranges_path = work_dir / "matmulnbits_single_layer.tranges"
model = onnx.load(str(model_path))
node = next(n for n in model.graph.node if n.op_type == "MatMulNBits")
attrs = {a.name: helper.get_attribute_value(a) for a in node.attribute}
K, N = attrs["K"], attrs["N"]
opset = next(o.version for o in model.opset_import if o.domain in ("", "ai.onnx"))
print(f"{model_path.name}: opset {opset}, {len(model.graph.node)} node")
print(
f"{node.op_type} ({node.domain}): K={K} N={N} bits={attrs['bits']} block_size={attrs['block_size']}"
)
print(f"inputs {list(node.input)} -> outputs {list(node.output)}")
matmulnbits_single_layer.onnx: opset 21, 1 node
MatMulNBits (com.microsoft): K=384 N=1152 bits=4 block_size=128
inputs ['A', 'B', 'scales'] -> outputs ['Y']
2. Tensor ranges
The model is not calibrated, so the ranges are written here. A is declared in [-10, 10]. Y's range is 1.5 times the largest output magnitude ONNX Runtime produces over ten random inputs drawn from that range. The two ranges are written as JSON to the .tranges path.
session = ort.InferenceSession(str(model_path), providers=["CPUExecutionProvider"])
rng = np.random.default_rng(0)
max_abs_y = 0.0
for _ in range(10):
sample = rng.uniform(-10.0, 10.0, (1, 1, K)).astype(np.float32)
max_abs_y = max(max_abs_y, float(np.abs(session.run(None, {"A": sample})[0]).max()))
tranges = {"A": [-10.0, 10.0], "Y": [-1.5 * max_abs_y, 1.5 * max_abs_y]}
tranges_path.write_text(json.dumps(tranges, indent=2))
print(f"A range: {tranges['A']}")
print(f"Y range: [{tranges['Y'][0]:.4f}, {tranges['Y'][1]:.4f}]")
A range: [-10.0, 10.0]
Y range: [-4576.8446, 4576.8446]
3. Compile
Builds the plugin's ChimeraJob on the original model with a QC-N HWConfig and the ranges. The constructor applies the shipped library: matmulnbits-const replaces the node, and the dequantized weights become the customOp's one constant operand. onnx_ort_override_p keeps the ONNX Runtime reference on the original graph. Prints plugin_result, the substituted graph's nodes, and the job summary after compile(). The assertion pins the matched bundle.
cgc_job = ChimeraJob(
str(model_path),
hw_config=HWConfig(product="QC-N"),
trange_file=str(tranges_path),
bypass_quantization_checks=True,
onnx_ort_override_p=str(model_path),
plugin_out_dir=str(work_dir),
)
print(cgc_job.plugin_result)
assert cgc_job.plugin_result.matched_plugins == ("matmulnbits-const",), cgc_job.plugin_result
substituted = onnx.load(cgc_job.plugin_result.model_path)
for n in substituted.graph.node:
print(f"{n.op_type} ({n.name}): inputs={list(n.input)} -> outputs={list(n.output)}")
cgc_job.compile(quiet=True)
print(cgc_job)
2026-10-23 00:55 - INFO - epu - chimera_job - Overriding onnx model ort execution with the onnx model: /quadric/sdk-cli/examples/custom_op/matmulnbits_single_layer.onnx
cgc plugin: 1 customOp(s) from matmulnbits-const -> /quadric/sdk-cli/examples/custom_op/_plugin_out_matmulnbits
QuadricCustomOp (CustomOp/quadric_plugin::matMulNBitsStub0): inputs=['A', 'B'] -> outputs=['Y']
╒═════════════════════╤═══════════════════════════════════════════════════════════════════════════════════════════╕
│ Module Name │ matmulnbits_single_layer_QC_N_1d7_16MB_4kB_128GBps_128GBps_16_OFF_x1_x1 │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ ONNX File │ /quadric/sdk-cli/examples/custom_op/_plugin_out_matmulnbits/matmulnbits_single_layer.onnx │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ Custom Ops │ /quadric/sdk-cli/examples/custom_op/_plugin_out_matmulnbits/custom_ops.hpp │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ Product Target │ QC-N │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ Number of Cores │ 1 │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ ISS Clock Frequency │ 1.700 │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ L2M Size │ 16MB │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ LRM Size │ 4kB │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ External Read BW │ 128GBps │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ External Write BW │ 128GBps │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ MACS per PE │ 16 │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ Max L2M │ 0.000MB │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ Max LRM │ 0.000kB │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ Max Temp Ext Bytes │ 0.004MB │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ Network GMACs │ │
╘═════════════════════╧═══════════════════════════════════════════════════════════════════════════════════════════╛
╒════╤════════╤════════╤══════════════╤══════════════════════════╤═══════╕
│ │ Type │ Name │ shape │ type │ mse │
╞════╪════════╪════════╪══════════════╪══════════════════════════╪═══════╡
│ 0 │ Input │ A │ [1, 1, 384] │ tensor[FixedPoint32<27>] │ n/a │
├────┼────────┼────────┼──────────────┼──────────────────────────┼───────┤
│ 1 │ Output │ Y │ [1, 1, 1152] │ tensor[FixedPoint32<16>] │ n/a │
╘════╧════════╧════════╧══════════════╧══════════════════════════╧═══════╛
4. ISS against ONNX Runtime
Draws one input from the declared range and runs it through ONNX Runtime on the original model and through the ISS on the compiled one. Prints both output ranges, the RMSE, the correlation and the max abs error. The assertion requires RMSE below 1e-3.
inputs = {"A": rng.uniform(-10.0, 10.0, (1, 1, K)).astype(np.float32)}
ort_y = cgc_job.run_onnx_inf_session(inputs=inputs)["Y"]
iss_y = np.asarray(cgc_job.run_inference_harness(inputs=inputs, compare_ort=False)["Y"]).reshape(
ort_y.shape
)
diff = (iss_y - ort_y).ravel()
rmse = float(np.sqrt(np.mean(diff**2)))
corr = float(np.corrcoef(ort_y.ravel(), iss_y.ravel())[0, 1])
print(f"ORT output range: [{ort_y.min():.4f}, {ort_y.max():.4f}]")
print(f"ISS output range: [{iss_y.min():.4f}, {iss_y.max():.4f}]")
print(f"RMSE: {rmse:.4f}")
print(f"Correlation: {corr:.6f}")
print(f"Max abs error: {np.abs(diff).max():.4f}")
assert rmse < 1e-3, f"ISS differs from ONNX Runtime: RMSE {rmse:.4e}"
2026-10-23 00:55 - INFO - epu - iss_testing - Found tranges for input: <tvm.contrib.epu.interval.Interval object at 0x7fefb13a1de0>
2026-10-23 00:55 - INFO - epu - iss_testing - Started Executing Onnxruntime...
2026-10-23 00:55 - INFO - epu - chimera_job - running ort reference with overriden model: /quadric/sdk-cli/examples/custom_op/matmulnbits_single_layer.onnx
2026-10-23 00:55 - INFO - epu - iss_testing - Done 0:00:00.039360
2026-10-23 00:55 - INFO - epu - iss_testing - Found tranges for input: <tvm.contrib.epu.interval.Interval object at 0x7ff14c2871c0>
FILM 1/1: 100%|██████████████████████████████████████████████████████| 1/1 [00:00<00:00, 491.08it/s]
ORT output range: [-2196.6162, 2027.6128]
ISS output range: [-2196.6162, 2027.6128]
RMSE: 0.0001
Correlation: 1.000000
Max abs error: 0.0004
