NOTE: The Jupyter Notebook below is included in the Chimera SDK and can be run interactively by running the following CLI command:
$ quadric sdk notebook
From the Jupyter Notebook window in your browser, select the notebook named /quadric/sdk-cli/examples/models/modernbert/modernbert_base/modernbert_gpnpu.ipynb.
ModernBERT-base Masked Language Model on Chimera GPNPU
ModernBERT-base (answerdotai/ModernBERT-base) is an encoder-only transformer: 22 layers, hidden 768, 12 heads, GeGLU FFN, RoPE, global attention every third layer and a ±64-token sliding window elsewhere, with a masked-language-model head that predicts the token under each [MASK] from a 50,368-token vocabulary.
This notebook takes the INT8 export written by modernbert_quant_pipeline.ipynb (in output/ next to this file, downloaded when it is not already there), compiles it for a single-core QC-N with 1 MB OCM (16 MACs/PE) through CGC's custom-op path, runs it on the ISS, and checks the predictions end to end against ONNX Runtime on fill-in-the-blank text.
It runs in the Chimera SDK environment (the one ChimeraJob lives in), not in the quantization notebook's .venv: the quantization step needs its own torch/neural_compressor stack, the compile needs the SDK, and the two do not share a kernel. Run the quantization notebook first, then this one.
input_ids [1,512]
|
+- int8 Gather + dequant + embedding LN .......... mbEmbedNorm (LN + quantize, one op)
|
== Encoder layer x22 ===================================================
|| attn LN + quantize ................. mbLayerNormQuantOp [none @ layer 0] ||
|| Wqkv [768 -> 2304] int8 ............ mbQkvOp ||
|| head split -> RoPE -> masked flash . mbAttnOp (int8 in, int8 out) ||
|| Wo [768 -> 768] .................... mbAttnWoOp (per-channel scale -> int16) ||
|| residual add ....................... mbResidualAddOp ||
|| mlp LN + quantize .................. mbLayerNormQuantOp ||
|| Wi + GeGLU + SmoothQuant + Wo ...... mbFfnOp ||
|| residual add ....................... mbResidualAddOp ||
=========================================================================
|
+- final LN + quantize ............... mbLayerNormQuantOp
+- head dense + GeLU ................. mbHeadDenseOp
+- head LN + quantize ................ mbLayerNormQuantOp
+- decoder [768 -> 50368] + bias ..... mbLogitsOp (int8 matmul, dequantize, bias)
|
logits [1,512,50368]
Quantization
The compile consumes the INT8 export as exported, together with its calibration ranges (output/model.onnx.tranges): SmoothQuant static INT8 with fused int8 matmuls where the quantization step chose them, QDQ elsewhere, a per-channel int8 MLP down-projection weight and a per-tensor int8 embedding table. The graph is not re-quantized here; the quantization notebook reports its pseudo-perplexity against FP32 (about 4.37 vs 4.06 on WikiText-2).
1. Setup
Two modules live next to this notebook: modernbert_custom_ops.py stamps the custom ops, and modernbert_helpers.py compiles, runs the ISS and validates. The export directory and the URL it is fetched from are set here; use_export points the compile at it.
import logging
import sys
from pathlib import Path
HERE = Path.cwd().resolve()
if str(HERE) not in sys.path:
sys.path.insert(0, str(HERE))
import modernbert_custom_ops as custom_ops
import modernbert_helpers as mb
logging.getLogger("epu").setLevel(logging.ERROR)
logging.getLogger("epu").handlers = [logging.NullHandler()]
OCM_SIZE = "1MB" # QC-N, single core, 16 MACs/PE
## The export to compile, and where to get it when this directory is empty.
EXPORT_DIR = HERE / "output"
EXPORT_URL = "https://sdk-cli-models.s3.us-east-2.amazonaws.com/modernbert"
mb.use_export(EXPORT_DIR)
print(f"kernels : {mb.KERNELS_HEADER.name}")
print(f"model : {mb.QUANTIZED_MODEL}")
print(f"tranges : {mb.TRANGES}")
print(f"stub : {mb.STUB_MODEL}")
kernels : modernbert_kernels.hpp
model : /quadric/sdk-cli/examples/models/modernbert/modernbert_base/output/model.onnx
tranges : /quadric/sdk-cli/examples/models/modernbert/modernbert_base/output/model.onnx.tranges
stub : /quadric/sdk-cli/examples/models/modernbert/modernbert_base/ccl_build/modernbert_base_stub.onnx
2. The quantized export
The export is three files in output/, plus the quantization notebook's FP32 export, which the validation below compares against as well when it is there. Whatever is missing is downloaded from EXPORT_URL, so this notebook runs on its own; right after the quantization notebook the three are already in place and nothing is downloaded.
| File | What |
|---|---|
model.onnx + model.onnx.data | the INT8 graph after the quantization notebook's clean pass |
model.onnx.tranges | calibration ranges; the two activations the graph never quantizes per tensor (the attention-Wo and MLP-Wo inputs) take their scale from here, and the residual stream and head are sized from it |
onnx_fp32/model.onnx | the FP32 export, kept by the quantization notebook as its reference (optional here) |
mb.fetch_export(EXPORT_DIR, EXPORT_URL)
for p in (mb.QUANTIZED_MODEL, mb.QUANTIZED_MODEL.with_suffix(".onnx.data"), mb.TRANGES):
assert p.exists(), f"missing {p}"
print(f" {p.name:40s} {p.stat().st_size / 1e6:8.1f} MB")
fp32 = mb.FP32_MODEL
print(
f" {'onnx_fp32/model.onnx':40s} {'present' if fp32.exists() else 'absent (INT8-only validation)'}"
)
import onnx
model = onnx.load(str(mb.QUANTIZED_MODEL), load_external_data=False)
head = custom_ops.head_projection(model)
print(
f"\n{len(model.graph.node)} nodes, opset {model.opset_import[0].version}; "
f"{custom_ops.NUM_LAYERS} layers, seq {custom_ops.SEQ_LEN}, hidden {custom_ops.HIDDEN}; "
f"final projection {head['matmul'].name} -> {head['width']} logits"
)
fetching model.onnx
fetching model.onnx.data
fetching model.onnx.tranges
model.onnx 0.7 MB
model.onnx.data 189.4 MB
model.onnx.tranges 0.1 MB
onnx_fp32/model.onnx absent (INT8-only validation)
1433 nodes, opset 17; 22 layers, seq 512, hidden 768; final projection /decoder/MatMul_quant -> 50368 logits
3. Custom ops
build_stub replaces each region with a QuadricCustomOp node whose constants (weights, folded scales, LayerNorm gammas, the fixed-point formats) are packed from the ONNX and the calibration ranges. Every edge and node name is validated up front, and the calibrated magnitudes are checked against each kernel's fixed-point format so a range overflow fails here rather than silently on the ISS.
stub = mb.ensure_stub(rebuild=True)
import collections
stamped = onnx.load(str(stub), load_external_data=False)
ops = collections.Counter()
for n in stamped.graph.node:
if n.op_type == custom_ops.CUSTOM_OP_TYPE:
attr = next(a for a in n.attribute if a.name == "ccl_func_name")
name = onnx.helper.get_attribute_value(attr)
ops[(name.decode() if isinstance(name, bytes) else name).split("<")[0]] += 1
else:
ops["native " + n.op_type] += 1
print(f"{stub.name}: {sum(ops.values())} nodes")
for k, v in sorted(ops.items(), key=lambda kv: -kv[1]):
print(f" {v:4d} {k}")
validated 183 edge and 336 node names against the export
WARNING: kScoreFracBits = 24 holds +/-128, but /model/layers.15/attn/Mul_4_output_0 calibrates to 99.9051 (78% of the format)
WARNING: kWoOutFracBits = 15 holds +/-65536, but /model/layers.15/mlp/Wo/MatMul_output_0 calibrates to 45860.1 (70% of the format)
WARNING: the residual stream outgrows fx16 (+/-32768) at 13 boundaries, worst /model/layers.17/Add_output_0 at 45870.4: the residual-add stamps lower their format there, and mbLayerNormQuantOp reinterprets the narrower input at 16 fractional bits
cloned 146 shared mask-chain nodes across 22 layers
/quadric/sdk-cli/examples/models/modernbert/modernbert_base/ccl_build/modernbert_base_stub.onnx: 183 nodes, 181 custom ops
modernbert_base_stub.onnx: 183 nodes
46 modernbert::mbLayerNormQuantOp
44 modernbert::mbResidualAddOp
22 modernbert::mbQkvOp
22 modernbert::mbAttnOp
22 modernbert::mbAttnWoOp
22 modernbert::mbFfnOp
1 native Gather
1 native DequantizeLinear
1 modernbert::mbLayerNormOp
1 modernbert::mbHeadDenseOp
1 modernbert::mbLogitsOp
4. Compile with CGC
ChimeraJob compiles the native nodes and the custom ops from modernbert_kernels.hpp, placing everything for a QC-N with 1 MB OCM. The full model compiles in about six minutes.
job, t_compile = mb.compile_stub(stub, ocm_size=OCM_SIZE)
print(f"COMPILE OK in {t_compile} s")
COMPILE OK in 547 s
5. Run on the ISS and validate
A paragraph of fill-in-the-blank sentences (one [MASK] each) is tokenized with the checkpoint's tokenizer and run both on the ISS and through ONNX Runtime on the un-stamped INT8 graph, on the same inputs. Two checks make the gate:
- Logits error: the cosine similarity of the ISS and ORT logits over the live tokens is the headline number; the gate is rms(ISS − ORT) / std(ORT) ≤ 0.2, which corresponds to a cosine of about 0.98. For scale, ORT INT8 itself sits near rms/std 0.5 (cosine 0.92) against the FP32 export on this text, so compiling adds about a third of the error the quantization already carries.
- Predictions: the ISS top-1 token must agree with ORT's at every masked position where ORT's own top-1 / top-2 margin is at least 2.0 logits. Below that margin the model is near-tied between two candidates, which an error of the size in (1) can legitimately flip; those positions are listed with both margins rather than gated.
The attention kernel attends over the whole 512-token window, so the ISS and its ORT reference are both fed an all-ones mask. The FP32 export is shown twice at every masked position, once with the same all-ones mask (comparable) and once with the tokenizer's padding mask (the model's intended behaviour), so the effect of the padding on a short text is visible in the tables. The top-5 predictions of every source are printed side by side. The full-model ISS run takes about 25 minutes.
mb.validate raises when either check fails, so a divergence stops the notebook here rather than printing a [FAIL] line into a scrolling log.
result = mb.validate(job=job, ocm_size=OCM_SIZE)
text: 89 live tokens, 8 masked position(s)
note: no FP32 export at /quadric/sdk-cli/examples/models/modernbert/modernbert_base/output/onnx_fp32/model.onnx; comparing against ORT INT8 only
FILM 183/183: 100%|███████████████████████████████████████████████| 183/183 [20:53<00:00, 6.85s/it]
ISS run: 1255 s, 568270854 cycles
ISS vs ort_int8 : cosine 0.989275 | rms/std 0.1693 | top-1 agree 100.000% on 8 masked tokens (100.000% on the 5 with margin >= 2)
[PASS] ModernBERT-base on the ISS vs ORT INT8: logits cosine 0.9893, rms/std 0.1693 (gate <= 0.2), top-1 agrees on 100.000% of the 5 confident masked tokens (gate 100%), 0 flip(s) among 8 masked tokens
Predictions at the masked positions:
[MASK] at token 6 ...The capital of France is[MASK]. Water boils at one hundred degrees...
ISS 'Paris' (24.2), 'Nice' (20.7), 'Lyon' (18.3), 'Nancy' (17.7), 'Brussels' (16.6)
ort_int8 'Paris' (23.6), 'Nice' (20.7), 'Lyon' (18.3), 'Nancy' (17.1), 'Geneva' (16.0)
[MASK] at token 15 .... Water boils at one hundred degrees[MASK] at sea level. The quick brown fox...
ISS 'north' (17.0), 'zero' (15.9), ',' (15.8), 'south' (15.6), 'and' (15.6)
ort_int8 'north' (17.0), 'below' (15.6), 'C' (15.5), 'over' (15.2), ',' (15.2)
[MASK] at token 28 ... The quick brown fox jumps over the lazy[MASK]. She parked the[MASK] in the garage...
ISS 'river' (26.2), 'dog' (20.7), 'bridge' (19.5), 'frog' (18.6), 'eye' (18.5)
ort_int8 'river' (25.6), 'dog' (21.3), 'bridge' (20.7), 'frog' (18.6), 'eye' (18.5)
[MASK] at token 33 ... over the lazy[MASK]. She parked the[MASK] in the garage before going inside. Photos...
ISS 'car' (20.1), 'vehicle' (16.8), 'van' (16.6), 'truck' (16.5), 'bike' (15.6)
ort_int8 'car' (19.5), 'truck' (16.5), 'van' (16.0), 'bike' (15.6), 'vehicle' (15.5)
[MASK] at token 48 .... Photosynthesis lets plants turn sunlight into[MASK]. The museum opens at nine in the...
ISS 'energy' (21.7), 'food' (21.5), 'oxygen' (20.5), 'nutrients' (19.9), 'color' (19.2)
ort_int8 'energy' (21.7), 'oxygen' (21.1), 'food' (20.8), 'nutrients' (19.9), 'water' (19.6)
[MASK] at token 57 .... The museum opens at nine in the[MASK] and closes at five. He ordered a...
ISS 'morning' (27.3), 'afternoon' (24.7), 'evening' (24.3), 'mornings' (19.9), 'day' (19.4)
ort_int8 'morning' (26.7), 'afternoon' (24.7), 'evening' (23.7), 'day' (18.8), 'mornings' (18.7)
[MASK] at token 68 ... at five. He ordered a cup of[MASK] and a slice of toast for breakfast....
ISS 'coffee' (24.0), 'tea' (22.2), 'water' (20.9), 'milk' (20.8), 'soup' (19.3)
ort_int8 'coffee' (24.0), 'tea' (21.6), 'milk' (20.1), 'water' (19.6), 'soup' (18.7)
[MASK] at token 84 .... Mount Everest is the tallest[MASK] on Earth.[SEP][PAD][PAD][PAD][PAD]...
ISS 'mountain' (24.7), 'peak' (21.7), 'building' (19.2), 'tree' (18.9), 'summit' (18.9)
ort_int8 'mountain' (24.7), 'peak' (21.7), 'summit' (19.5), 'rock' (18.9), 'tree' (18.9)
Any text works, as long as it carries at least one [MASK]:
result = mb.validate("Paris is the [MASK] of France.", job=job, ocm_size=OCM_SIZE,
strict=False)
strict=False reports a divergence instead of raising: on a text of your own, a disagreement is what you came to look at.
6. Performance
print(job)
info = result.get("info") or {}
if info.get("cycles"):
print(
f"\n{info['cycles'] / 1e6:,.0f} M cycles for one 512-token sequence "
f"= {info['cycles'] / 1e9 * 1e3:,.0f} ms at 1 GHz"
)
╒═════════════════════╤═════════════════════════════════════════════════════════════════════════════════════════════════╕
│ Module Name │ modernbert_base_stub_QC_N_1d0_1MB_4kB_8GBps_8GBps_16_OFF_x1_x1 │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────┤
│ ONNX File │ /quadric/sdk-cli/examples/models/modernbert/modernbert_base/ccl_build/modernbert_base_stub.onnx │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Custom Ops │ /quadric/sdk-cli/examples/models/modernbert/modernbert_base/modernbert_kernels.hpp │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Product Target │ QC-N │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Number of Cores │ 1 │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────┤
│ ISS Clock Frequency │ 1.000 │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────┤
│ L2M Size │ 1MB │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────┤
│ LRM Size │ 4kB │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────┤
│ External Read BW │ 8GBps │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────┤
│ External Write BW │ 8GBps │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────┤
│ MACS per PE │ 16 │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Max L2M │ 0.938MB │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Max LRM │ 0.312kB │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Max Temp Ext Bytes │ 98.750MB │
├─────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Network GMACs │ │
╘═════════════════════╧═════════════════════════════════════════════════════════════════════════════════════════════════╛
╒════╤════════╤════════════════╤═════════════════╤══════════════════════════╤═══════╕
│ │ Type │ Name │ shape │ type │ mse │
╞════╪════════╪════════════════╪═════════════════╪══════════════════════════╪═══════╡
│ 0 │ Input │ input_ids │ [1, 512] │ tensor[int32] │ n/a │
├────┼────────┼────────────────┼─────────────────┼──────────────────────────┼───────┤
│ 1 │ Input │ attention_mask │ [1, 512] │ tensor[int32] │ n/a │
├────┼────────┼────────────────┼─────────────────┼──────────────────────────┼───────┤
│ 2 │ Output │ logits │ [1, 512, 50368] │ tensor[FixedPoint32<16>] │ n/a │
╘════╧════════╧════════════════╧═════════════════╧══════════════════════════╧═══════╛
Post-ISS Report 1.0 GHz ***
Fully placed-and-routed gate simulation:
╒══════════════════════════════════╤═════════╕
│ Latency (ms) │ 568.27 │
├──────────────────────────────────┼─────────┤
│ FPS │ 1.76 │
├──────────────────────────────────┼─────────┤
│ Average Power @ 3nm SSGNP (mW) │ 181.13 │
├──────────────────────────────────┼─────────┤
│ FPS per Watt @ 3nm SSGNP (FPS/W) │ 9.72 │
├──────────────────────────────────┼─────────┤
│ Ext Rd Bytes (MB) │ 1174.28 │
├──────────────────────────────────┼─────────┤
│ Ext Wr Bytes (MB) │ 583.03 │
├──────────────────────────────────┼─────────┤
│ Avg Ext Rd BW (GBps) │ 2.02 │
├──────────────────────────────────┼─────────┤
│ Avg Ext Wr BW (GBps) │ 1 │
╘══════════════════════════════════╧═════════╛
*** Data generated using 7nm SSGNP gatesim and scaled to 3nm
[SDK-CLI] : TotalCycles: 568,270,854
[SDK-CLI] : Executions/second: 1.76
compute : ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 179.032M
data_array : ▇▇▇▇▇▇▇▇▇▇ 36.09M
mac : ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 110.692M
data_external: ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 89.246M
data_ocm : ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 151.554M
for more information check run directory: /quadric/sdk-cli/examples/models/modernbert/modernbert_base/ccl_build/modernbert_base_stub_QC_N_1d0_1MB_4kB_8GBps_8GBps_16_OFF_x1_x1/run/20260922_030133_c2af22
568 M cycles for one 512-token sequence = 568 ms at 1 GHz
Summary
| Model | ModernBERT-base masked language model (50,368-token vocabulary) |
| Input | 512 tokens, batch 1 |
| Target | Chimera GPNPU QC-N, single core, 16 MACs/PE, 1 MB OCM, 4 kB LRM |
| Quantization | INT8 export from modernbert_quant_pipeline.ipynb (SmoothQuant, symmetric), calibration .tranges |
| Custom ops | mbEmbedNorm, mbLayerNormQuantOp, mbQkvOp, mbAttnOp, mbAttnWoOp, mbResidualAddOp, mbFfnOp, mbHeadDenseOp, mbLogitsOp |
| Validation | ISS vs ONNX Runtime (INT8, same inputs, all-ones mask): logits cosine reported, gate rms/std ≤ 0.2 and top-1 agreement at every masked position where ORT's margin ≥ 2.0; FP32 shown for scale, with and without the padding mask |
Citation
@article{warner2024modernbert,
title = {Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for
Fast, Memory Efficient, and Long Context Finetuning and Inference},
author = {Warner, Benjamin and Chaffin, Antoine and Clavié, Benjamin and
Weller, Orion and Hallström, Oskar and Taghadouini, Said and
Gallagher, Alexis and Biswas, Raja and Ladhak, Faisal and
Aarsen, Tom and Cooper, Nathan and Adams, Griffin and
Howard, Jeremy and Poli, Iacopo},
journal = {arXiv preprint arXiv:2412.13663},
year = {2024}
}
