Tutorial: ModernBERT-base INT8 Quantization

NOTE: The Jupyter Notebook below is included in the Chimera SDK and can be run interactively by running the following CLI command:

$ quadric sdk notebook

From the Jupyter Notebook window in your browser, select the notebook named /quadric/sdk-cli/examples/models/modernbert/modernbert_base/modernbert_quant_pipeline.ipynb.

INT8 Quantization for ModernBERT-base with the Quadric SDK

Runs the full static INT8 quantization pipeline end to end, in 4 steps, entirely from this notebook -- no shell scripts, no subprocess calls, no log files to scrape. Each step is a plain Python function in src/modernbert_pipeline.py; the per-matmul treatment for each matmul in the graph is controlled by src/modernbert_quant_config.py.

Self-contained: this folder has no dependency on any particular location on disk -- unzip it anywhere and run. There is no local checkpoint or dataset to point at -- answerdotai/ModernBERT-base and its calibration/eval corpus (WikiText-2) are both fetched from the HuggingFace Hub the first time you run Steps 1-3.

1. Setup

Run the cell below once to create a .venv, install dependencies, and register a Jupyter kernel named modernbert-quant.

Then select that kernel -- shown either as the Python environment ending in .venv, or as the Jupyter kernel modernbert-quant, depending on your editor; both are the same environment. Run the confirmation cell after that.

!python3.10 -m venv .venv
!.venv/bin/pip install --quiet --upgrade pip
!.venv/bin/pip install --quiet -r requirements.txt ipykernel
!.venv/bin/python -m ipykernel install --user --name modernbert-quant --display-name "modernbert-quant"

WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPSConnection(host='pypi.ngc.nvidia.com', port=443): Failed to resolve 'pypi.ngc.nvidia.com' ([Errno -2] Name or service not known)")': /pip/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPSConnection(host='pypi.ngc.nvidia.com', port=443): Failed to resolve 'pypi.ngc.nvidia.com' ([Errno -2] Name or service not known)")': /pip/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPSConnection(host='pypi.ngc.nvidia.com', port=443): Failed to resolve 'pypi.ngc.nvidia.com' ([Errno -2] Name or service not known)")': /pip/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPSConnection(host='pypi.ngc.nvidia.com', port=443): Failed to resolve 'pypi.ngc.nvidia.com' ([Errno -2] Name or service not known)")': /pip/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPSConnection(host='pypi.ngc.nvidia.com', port=443): Failed to resolve 'pypi.ngc.nvidia.com' ([Errno -2] Name or service not known)")': /pip/
Installed kernelspec modernbert-quant in /github/home/.local/share/jupyter/kernels/modernbert-quant

import os
import sys
from pathlib import Path

PIPELINE_DIR = Path.cwd()
SRC_DIR = PIPELINE_DIR / "src"
sys.path.insert(0, str(SRC_DIR))

MODEL_ID = "answerdotai/ModernBERT-base"  # HuggingFace Hub id -- no local checkpoint needed
## Everything this notebook writes. MODERNBERT_OUTPUT_DIR redirects it -- CI sets it, so
## that a compile run next door keeps its own output/ to itself.
OUTPUT_DIR = PIPELINE_DIR / os.environ.get("MODERNBERT_OUTPUT_DIR", "output")
SEQ_LEN = 512

import onnx, onnxruntime, torch, transformers, datasets  # fails fast with a clear import error if the kernel isn't modernbert-quant

print(f"Kernel Python: {sys.version.split()[0]}")
print(
    f"torch {torch.__version__}, onnxruntime {onnxruntime.__version__}, transformers {transformers.__version__}, datasets {datasets.__version__}"
)
print(f"pipeline dir:  {PIPELINE_DIR}  (src/ has {len(list(SRC_DIR.glob('*.py')))} scripts)")
print(f"model id:      {MODEL_ID}  (fetched from the HF Hub on first use)")

/quadric/sdk-cli/examples/models/modernbert/modernbert_base/.venv/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm


Kernel Python: 3.10.12
torch 2.5.1+cu124, onnxruntime 1.23.2, transformers 4.55.4, datasets 5.0.1
pipeline dir:  /quadric/sdk-cli/examples/models/modernbert/modernbert_base  (src/ has 4 scripts)
model id:      answerdotai/ModernBERT-base  (fetched from the HF Hub on first use)

2. Pipeline Overview

Every matmul in the graph lands in exactly one treatment, driven by src/modernbert_quant_config.py's pattern lists (see that file's module docstring for the full per-matmul breakdown).

Step 2 is one call over eight internal passes: seven that build the graph, plus a finalize pass that clears out the intermediates they wrote. Their order isn't a free choice -- cleanup_graph's shape-inference pass corrupts this checkpoint's declared output type if the FP16 passes have already run, and each QDQ pass needs the calibration ranges the SmoothQuant pass produces -- so they run as one step rather than as eight cells that must be executed in exactly one sequence. Each pass is still importable on its own from modernbert_pipeline when one of them is what's actually misbehaving, and each is a no-op when its pattern list is empty -- so which buckets are populated stays a config question, not a code-path question.

The cell below prints the live config state (which pattern lists are non-empty right now).

import modernbert_quant_config as qc

for name in dir(qc):
    if name.isupper() and isinstance(getattr(qc, name), list):
        val = getattr(qc, name)
        marker = "  <-- non-empty" if val else ""
        print(f"{name:50s} {val}{marker}")

EXCLUDE_NAME_PATTERNS                              ['rotary_emb/MatMul']  <-- non-empty
EXCLUDE_OP_TYPES                                   []
QDQ_ONLY_PATTERNS                                  ['attn/MatMul', 'mlp/Wi/MatMul', 'attn/Wo/MatMul', '/head/dense/MatMul']  <-- non-empty
QDQ_PERCHANNEL_WEIGHT_AXIS1_FP16_ACT_PATTERNS      ['mlp/Wo/MatMul']  <-- non-empty
QDQ_WEIGHT_FP16_ACT_PATTERNS                       []
QUANTIZE_EMBEDDING_PATTERNS                        ['model.embeddings.tok_embeddings.weight']  <-- non-empty

3. Step 1 -- ONNX Export

Loads the checkpoint with AutoModelForMaskedLM and exports it with attn_implementation="eager", a fixed (batch=1, seq_len=512) shape, via raw torch.onnx.export (optimum has no ModernBERT config). Output is logits shaped [1, 512, 50368] (vocab-sized, one distribution per position).

The cell then verifies one property of the export that Step 2 depends on. This checkpoint ties the decoder's output projection to the input embedding table (tie_word_embeddings=True) at the PyTorch level, but torch.onnx.export emits the tied nn.Linear as a MatMul against its own separately-named constant -- so the two are distinct ONNX initializers and can be quantized independently (the decoder as fused int8, the embedding table per-tensor). The assert checks that against the real export rather than trusting the reasoning: if a future torch/transformers emits a genuinely shared initializer, this fails loudly instead of silently quantizing one tensor twice.

import onnx

from modernbert_pipeline import export_onnx

fp32_model_path = export_onnx(
    MODEL_ID, str(OUTPUT_DIR / "onnx_fp32" / "model.onnx"), seq_len=SEQ_LEN
)

## Verify the decoder projection and the embedding table are distinct ONNX initializers, so
## Step 2 can quantize them independently (fused int8 / per-tensor) without touching one twice.
_m = onnx.load(fp32_model_path, load_external_data=False)
_decoder_node = next(
    n for n in _m.graph.node if n.op_type == "MatMul" and n.name.endswith("/decoder/MatMul")
)
_decoder_weight_name = next(inp for inp in _decoder_node.input if inp != _decoder_node.input[0])
assert _decoder_weight_name != "model.embeddings.tok_embeddings.weight", (
    f"/decoder/MatMul's weight ({_decoder_weight_name!r}) is the SAME initializer as the embedding "
    "table -- Step 2 would independently quantize a shared tensor. Revisit their bucket "
    "assignment in modernbert_quant_config.py before continuing."
)
print(
    f"OK: /decoder/MatMul's weight ({_decoder_weight_name!r}) is a distinct initializer from the embedding table."
)
del _m, _decoder_node, _decoder_weight_name

Loading model from: answerdotai/ModernBERT-base


/quadric/sdk-cli/examples/models/modernbert/modernbert_base/.venv/lib/python3.10/site-packages/transformers/modeling_attn_mask_utils.py:196: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
  inverted_mask = torch.tensor(1.0, dtype=dtype) - expanded_mask


Exported to /quadric/sdk-cli/examples/models/modernbert/modernbert_base/ci_output/onnx_fp32/model.onnx (batch=1, seq_len=512)
OK: /decoder/MatMul's weight ('onnx::MatMul_3755') is a distinct initializer from the embedding table.

4. Step 2 -- Quantization (Every Treatment, All Layers)

One call quantizes the graph and applies each matmul's assigned treatment across all layers. Which matmul gets which, entirely from modernbert_quant_config.py:

QOperator has no pattern list of its own -- a matmul lands there by matching none of the others. quantize_smoothquant_int8 holds out every node whose name ends with one of the patterns above, and quantize_static fuses all the rest into a QLinearMatMul. So fused int8 is the default, and the pattern lists are the exceptions to it.

Internally this runs SmoothQuant calibration on WikiText-2, the fused-int8 pass, a defusing pass that removes the redundant rounding where one QOperator matmul's INT8 output feeds straight into another's operand, the two QDQ passes, onnxsim cleanup, and the two FP16-activation passes -- in that order, for the reasons in quantize_model's docstring. Every pass prints as it goes and writes its own subdirectory under output/ while it runs, so a failure is easy to localize. A final pass then consolidates the deliverable at the root of output/ and deletes those intermediates -- several GB of them, all reproducible by re-running:

output/model.onnx          the clean INT8 graph
output/model.onnx.data     its external weights
output/model.onnx.tranges  per-tensor calibration ranges
output/onnx_fp32/          Step 1's FP32 export, kept as Step 3's baseline

The FP32 export is preserved rather than pruned: Step 3 scores the quantized graph against it, so it is a reference, not an intermediate. Pass keep_intermediates=True to keep every pass on disk instead.

To move a matmul between buckets, edit the pattern lists and re-run this cell -- no code change.

from modernbert_pipeline import quantize_model

cleaned_model_path, tranges_path = quantize_model(
    fp32_model_path,
    str(OUTPUT_DIR),
    MODEL_ID,
    seq_len=SEQ_LEN,
    calibration_samples=500,
    smoothquant_alpha=0.7,
    symmetric=True,
)

Token indices sequence length is longer than the specified maximum sequence length for this model (2304667 > 512). Running this sequence through the model will result in indexing errors


Calibrating on 500 512-token windows from wikitext/wikitext-2-raw-v1 (train)
Holding 91 matmul node(s) out of QOperator quantization (later passes give them their own treatment)
Quantizing (SmoothQuant + static INT8, this is the slow part)...


2026-09-22 03:35:26 [INFO] Start smooth model calibration.


SmoothQuant calibrated 10 samples...
SmoothQuant calibrated 20 samples...
SmoothQuant calibrated 30 samples...
SmoothQuant calibrated 40 samples...
SmoothQuant calibrated 50 samples...
SmoothQuant calibrated 60 samples...
SmoothQuant calibrated 70 samples...
SmoothQuant calibrated 80 samples...
SmoothQuant calibrated 90 samples...


2026-09-22 03:35:46 [INFO] Start smooth scales collection.


SmoothQuant calibrated 100 samples...
SmoothQuant calibration complete: 100 samples
Progress: [####################] 100.00%

WARNING:root:Please use QuantFormat.QDQ for activation type QInt8 and weight type QInt8. Or it will lead to bad performance on x64.


Computing tensor ranges for the activations the later passes still need to quantize...
Quantized model: /quadric/sdk-cli/examples/models/modernbert/modernbert_base/ci_output/quantized/model_opt_sym_int8_q.onnx
Tensor ranges:   /quadric/sdk-cli/examples/models/modernbert/modernbert_base/ci_output/quantized/model_opt_sym_int8_q.onnx.tranges
Defuse pass: defused 22 redundant QOperator->QOperator requantization(s): ['/model/layers.0/attn/MatMul_1_quant operand 3', '/model/layers.1/attn/MatMul_1_quant operand 3', '/model/layers.2/attn/MatMul_1_quant operand 3', '/model/layers.3/attn/MatMul_1_quant operand 3', '/model/layers.4/attn/MatMul_1_quant operand 3', '/model/layers.5/attn/MatMul_1_quant operand 3', '/model/layers.6/attn/MatMul_1_quant operand 3', '/model/layers.7/attn/MatMul_1_quant operand 3', '/model/layers.8/attn/MatMul_1_quant operand 3', '/model/layers.9/attn/MatMul_1_quant operand 3', '/model/layers.10/attn/MatMul_1_quant operand 3', '/model/layers.11/attn/MatMul_1_quant operand 3', '/model/layers.12/attn/MatMul_1_quant operand 3', '/model/layers.13/attn/MatMul_1_quant operand 3', '/model/layers.14/attn/MatMul_1_quant operand 3', '/model/layers.15/attn/MatMul_1_quant operand 3', '/model/layers.16/attn/MatMul_1_quant operand 3', '/model/layers.17/attn/MatMul_1_quant operand 3', '/model/layers.18/attn/MatMul_1_quant operand 3', '/model/layers.19/attn/MatMul_1_quant operand 3', '/model/layers.20/attn/MatMul_1_quant operand 3', '/model/layers.21/attn/MatMul_1_quant operand 3']
Matched 67 MatMul node(s) against QDQ_ONLY_PATTERNS=['attn/MatMul', 'mlp/Wi/MatMul', 'attn/Wo/MatMul', '/head/dense/MatMul']
QDQ pass: 67 matmul(s) QDQ'd, saved to /quadric/sdk-cli/examples/models/modernbert/modernbert_base/ci_output/with_qdq/model.onnx
  model.embeddings.tok_embeddings.weight: 154.73 MB FP32 -> 38.68 MB INT8
Embedding pass: 1 embedding table(s) quantized, saved to /quadric/sdk-cli/examples/models/modernbert/modernbert_base/ci_output/with_embedding_qdq/model.onnx
Confirmed static graph. Nodes before cleanup: 1719
Nodes after cleanup: 1345 (374 removed, 21.8%)
Cleanup pass: saved to /quadric/sdk-cli/examples/models/modernbert/modernbert_base/ci_output/cleaned/model.onnx
Pattern list is empty -- passing the model through unchanged.
Matched 22 MatMul node(s) against ['mlp/Wo/MatMul'] (per-output-channel weight + FP16 activation)
Done: 22 matmul(s) given weight-QDQ+FP16-activation treatment, saved to /quadric/sdk-cli/examples/models/modernbert/modernbert_base/ci_output/cleaned/model.onnx
Finalize pass: removed 3.70 GB of intermediates. /quadric/sdk-cli/examples/models/modernbert/modernbert_base/ci_output now holds:
        0.70 MB  model.onnx
      189.38 MB  model.onnx.data
        0.12 MB  model.onnx.tranges
      755.90 MB  onnx_fp32/model.onnx
Step 2 done: quantized graph at /quadric/sdk-cli/examples/models/modernbert/modernbert_base/ci_output/model.onnx

5. Step 3 -- MLM Pseudo-Perplexity Evaluation

Scores INT8 against a FP32 baseline on WikiText-2's test split. Each evaluation window gets an independent 15% [MASK] corruption (fixed seed, so FP32 and INT8 are scored on identical masked positions and are directly comparable); pseudo-perplexity is exp(mean NLL)) of the model recovering the true token at each masked position from bidirectional context.

Runs the full test corpus by default. Set TOKEN_LIMIT to an integer (e.g. 50_000) to truncate the corpus for a fast check -- or leave it alone and set MODERNBERT_EVAL_TOKENS in the environment, which is how CI bounds this step. Results are saved to output/eval_log.txt either way, and returned as results.

from modernbert_pipeline import evaluate_mlm_perplexity

## None -> the full WikiText-2 test corpus; an integer truncates it to that many tokens
## (e.g. 50_000 for a fast check). MODERNBERT_EVAL_TOKENS is the same knob from outside.
TOKEN_LIMIT = int(os.environ.get("MODERNBERT_EVAL_TOKENS") or 0) or None

results = evaluate_mlm_perplexity(
    fp32_model_path,
    cleaned_model_path,
    MODEL_ID,
    seq_len=SEQ_LEN,
    token_limit=TOKEN_LIMIT,
    log_path=str(OUTPUT_DIR / "eval_log.txt"),
)
results

Loading eval split from wikitext/wikitext-2-raw-v1 (test)...


Token indices sequence length is longer than the specified maximum sequence length for this model (288968 > 512). Running this sequence through the model will result in indexing errors
WARNING:transformers.tokenization_utils_base:Token indices sequence length is longer than the specified maximum sequence length for this model (288968 > 512). Running this sequence through the model will result in indexing errors
/quadric/sdk-cli/examples/models/modernbert/modernbert_base/.venv/lib/python3.10/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py:123: UserWarning: Specified provider 'CUDAExecutionProvider' is not in available provider names.Available providers: 'AzureExecutionProvider, CPUExecutionProvider'
  warnings.warn(


Total tokens: 50,000
Windows: 97 (dropped 336 leftover tokens)

--- FP32 baseline ---


Windows (model.onnx): 100%|██████████| 97/97 [00:14<00:00,  6.84it/s]


--- INT8 ---


Windows (model.onnx): 100%|██████████| 97/97 [00:34<00:00,  2.83it/s]

FP32 baseline: /quadric/sdk-cli/examples/models/modernbert/modernbert_base/ci_output/onnx_fp32/model.onnx
INT8 model: /quadric/sdk-cli/examples/models/modernbert/modernbert_base/ci_output/model.onnx
Windows: 97
Masked tokens scored: 7,432 (FP32), 7,432 (INT8)
FP32 pseudo-perplexity: 4.5421
INT8 pseudo-perplexity: 4.8630

Eval log saved to: /quadric/sdk-cli/examples/models/modernbert/modernbert_base/ci_output/eval_log.txt








{'fp32': {'n_masked': 7432,
  'avg_nll': 1.5133793447209316,
  'perplexity': 4.542054054834273},
 'int8': {'n_masked': 7432,
  'avg_nll': 1.5816556026074615,
  'perplexity': 4.86300034617662}}

Summary

Key takeaways

1. Which matmul gets which treatment is configuration, not code. Every matmul lands in exactly one bucket in modernbert_quant_config.py, and each pass is a no-op when its pattern list is empty -- so moving a matmul between fused int8, QDQ-only and per-channel is an edit to one list.
2. Fusing each matmul independently rounds twice. quantize_static cannot see that attn/MatMul_1's INT8 output reaches attn/Wo through nothing but a reshape and a SmoothQuant Mul; the defuse pass reuses the upstream INT8 tensor and its scale instead of quantizing onto a second 8-bit grid.
3. SmoothQuant is what forces mlp/Wo per-channel. It migrates outlier activation channels into that weight's rows, so a single per-tensor scale would spend its whole range on a few rows and flatten the rest.
4. The pass order is load-bearing. cleanup_graph's shape inference corrupts this checkpoint's declared output type once the FP16 passes have run, and each QDQ pass needs the ranges SmoothQuant produced -- which is why Step 2 is one call rather than eight cells to run in exactly one sequence.

Citation

@article{warner2024modernbert,
  title   = {Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for
             Fast, Memory Efficient, and Long Context Finetuning and Inference},
  author  = {Warner, Benjamin and Chaffin, Antoine and Clavié, Benjamin and
             Weller, Orion and Hallström, Oskar and Taghadouini, Said and
             Gallagher, Alexis and Biswas, Raja and Ladhak, Faisal and
             Aarsen, Tom and Cooper, Nathan and Adams, Griffin and
             Howard, Jeremy and Poli, Iacopo},
  journal = {arXiv preprint arXiv:2412.13663},
  year    = {2024}
}