Model Demo: Qwen2.5-0.5B INT8 Quantization Pipeline

NOTE: The Jupyter Notebook below is included in the Chimera SDK and can be run interactively by running the following CLI command:

$ quadric sdk notebook

From the Jupyter Notebook window in your browser, select the notebook named /quadric/sdk-cli/examples/models/qwen/qwen2.5-0.5b/qwen_quant_pipeline.ipynb.

INT8 Quantization for Qwen2.5-0.5B with the Quadric SDK

This notebook walks through the static W8A8 quantization pipeline shipped in examples/models/qwen/qwen2.5-0.5b/. The pipeline takes a HuggingFace Qwen/Qwen2.5-0.5B-Instruct checkpoint and produces a deployment-ready INT8 ONNX model with fixed shapes for autoregressive inference. Everything in this notebook runs on the host CPU; the resulting ONNX is the artifact that would later be handed to CGC (Chimera Graph Compiler) for compilation onto a Chimera GPNPU.

Why a dedicated pipeline?

Off-the-shelf ONNX Runtime static quantization is close to what we want, but Qwen has three quirks that need handling. First, RoPE (Rotary Position Embedding): off-the-shelf, the export bakes sin/cos into the graph as position-fixed constants — fine for a single forward pass but wrong for token-by-token decoding, where each step needs the angle for a new position. The pipeline applies a patched transformers/optimum that lifts sin/cos out as explicit graph inputs, so the runner can supply the right values per step. Second, a handful of ops are precision-sensitive — down_proj, the Q·K attention MatMul, LayerNorm — and must be held out of quantization to preserve perplexity. Third, the static graph has to land at fixed shapes so it can run token-by-token with a KV cache.

1. Setup

The pipeline depends on patched transformers and optimum (the RoPE export is rewritten to use explicit sin/cos inputs), so it can't share the kernel's libraries. Following the same approach as the qwen3_8b demo, the notebook provisions a dedicated qwen_quant_venv and shells into it for every pipeline step — the kernel's own environment is left untouched.

The cell below creates the venv, installs requirements.txt, and runs setup_libs.sh (which clones transformers/optimum at the pinned tags, applies the patches in patches/, and editable-installs them). This is a one-time cost of a few minutes; re-running it is cheap because the clone and patches are skipped when already present.

## Provision the dedicated venv with the patched transformers/optimum (mirrors qwen3_8b).
## Pinned to Python 3.10 to match the patched library versions (see README prerequisites).
## setup_libs.sh clones + patches the libraries with git; the SDK runtime image ships without
## git, so install it if missing (no-op on dev machines that already have it).
!command -v git >/dev/null 2>&1 || (apt-get update -qq && apt-get install -y -qq git)
!python3.10 -m venv qwen_quant_venv
!qwen_quant_venv/bin/pip install --quiet --upgrade pip
!qwen_quant_venv/bin/pip install --quiet -r requirements.txt
!bash -c "source qwen_quant_venv/bin/activate && ./setup_libs.sh"

debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package perl-modules-5.34.
(Reading database ... 17301 files and directories currently installed.)
Preparing to unpack .../00-perl-modules-5.34_5.34.0-3ubuntu1.5_all.deb ...
Unpacking perl-modules-5.34 (5.34.0-3ubuntu1.5) ...
Selecting previously unselected package libgdbm6:amd64.
Preparing to unpack .../01-libgdbm6_1.23-1_amd64.deb ...
Unpacking libgdbm6:amd64 (1.23-1) ...
Selecting previously unselected package libgdbm-compat4:amd64.
Preparing to unpack .../02-libgdbm-compat4_1.23-1_amd64.deb ...
Unpacking libgdbm-compat4:amd64 (1.23-1) ...
Selecting previously unselected package libperl5.34:amd64.
Preparing to unpack .../03-libperl5.34_5.34.0-3ubuntu1.5_amd64.deb ...
Unpacking libperl5.34:amd64 (5.34.0-3ubuntu1.5) ...
Selecting previously unselected package perl.
Preparing to unpack .../04-perl_5.34.0-3ubuntu1.5_amd64.deb ...
Unpacking perl (5.34.0-3ubuntu1.5) ...
Selecting previously unselected package less.
Preparing to unpack .../05-less_590-1ubuntu0.22.04.3_amd64.deb ...
Unpacking less (590-1ubuntu0.22.04.3) ...
Selecting previously unselected package netbase.
Preparing to unpack .../06-netbase_6.3_all.deb ...
Unpacking netbase (6.3) ...
Selecting previously unselected package libcbor0.8:amd64.
Preparing to unpack .../07-libcbor0.8_0.8.0-2ubuntu1_amd64.deb ...
Unpacking libcbor0.8:amd64 (0.8.0-2ubuntu1) ...
Selecting previously unselected package libfido2-1:amd64.
Preparing to unpack .../08-libfido2-1_1.10.0-1_amd64.deb ...
Unpacking libfido2-1:amd64 (1.10.0-1) ...
Selecting previously unselected package libxext6:amd64.
Preparing to unpack .../09-libxext6_2%3a1.3.4-1build1_amd64.deb ...
Unpacking libxext6:amd64 (2:1.3.4-1build1) ...
Selecting previously unselected package libxmuu1:amd64.
Preparing to unpack .../10-libxmuu1_2%3a1.1.3-3_amd64.deb ...
Unpacking libxmuu1:amd64 (2:1.1.3-3) ...
Selecting previously unselected package openssh-client.
Preparing to unpack .../11-openssh-client_1%3a8.9p1-3ubuntu0.15_amd64.deb ...
Unpacking openssh-client (1:8.9p1-3ubuntu0.15) ...
Selecting previously unselected package xauth.
Preparing to unpack .../12-xauth_1%3a1.1-1build2_amd64.deb ...
Unpacking xauth (1:1.1-1build2) ...
Selecting previously unselected package libcurl3-gnutls:amd64.
Preparing to unpack .../13-libcurl3-gnutls_7.81.0-1ubuntu1.24_amd64.deb ...
Unpacking libcurl3-gnutls:amd64 (7.81.0-1ubuntu1.24) ...
Selecting previously unselected package liberror-perl.
Preparing to unpack .../14-liberror-perl_0.17029-1_all.deb ...
Unpacking liberror-perl (0.17029-1) ...
Selecting previously unselected package git-man.
Preparing to unpack .../15-git-man_1%3a2.34.1-1ubuntu1.17_all.deb ...
Unpacking git-man (1:2.34.1-1ubuntu1.17) ...
Selecting previously unselected package git.
Preparing to unpack .../16-git_1%3a2.34.1-1ubuntu1.17_amd64.deb ...
Unpacking git (1:2.34.1-1ubuntu1.17) ...
Selecting previously unselected package patch.
Preparing to unpack .../17-patch_2.7.6-7build2_amd64.deb ...
Unpacking patch (2.7.6-7build2) ...
Setting up libcbor0.8:amd64 (0.8.0-2ubuntu1) ...
Setting up less (590-1ubuntu0.22.04.3) ...
Setting up libxext6:amd64 (2:1.3.4-1build1) ...
Setting up libcurl3-gnutls:amd64 (7.81.0-1ubuntu1.24) ...
Setting up perl-modules-5.34 (5.34.0-3ubuntu1.5) ...
Setting up patch (2.7.6-7build2) ...
Setting up git-man (1:2.34.1-1ubuntu1.17) ...
Setting up netbase (6.3) ...
Setting up libfido2-1:amd64 (1.10.0-1) ...
Setting up libxmuu1:amd64 (2:1.1.3-3) ...
Setting up libgdbm6:amd64 (1.23-1) ...
Setting up openssh-client (1:8.9p1-3ubuntu0.15) ...
update-alternatives: using /usr/bin/ssh to provide /usr/bin/rsh (rsh) in auto mode
update-alternatives: warning: skip creation of /usr/share/man/man1/rsh.1.gz because associated file /usr/share/man/man1/ssh.1.gz (of link group rsh) doesn't exist
update-alternatives: using /usr/bin/slogin to provide /usr/bin/rlogin (rlogin) in auto mode
update-alternatives: warning: skip creation of /usr/share/man/man1/rlogin.1.gz because associated file /usr/share/man/man1/slogin.1.gz (of link group rlogin) doesn't exist
update-alternatives: using /usr/bin/scp to provide /usr/bin/rcp (rcp) in auto mode
update-alternatives: warning: skip creation of /usr/share/man/man1/rcp.1.gz because associated file /usr/share/man/man1/scp.1.gz (of link group rcp) doesn't exist
Setting up libgdbm-compat4:amd64 (1.23-1) ...
Setting up xauth (1:1.1-1build2) ...
Setting up libperl5.34:amd64 (5.34.0-3ubuntu1.5) ...
Setting up perl (5.34.0-3ubuntu1.5) ...
Setting up liberror-perl (0.17029-1) ...
Setting up git (1:2.34.1-1ubuntu1.17) ...
Processing triggers for libc-bin (2.35-0ubuntu3.13) ...
==> transformers: cloning https://github.com/huggingface/transformers.git at v4.55.4
Cloning into 'vendor/transformers'...
remote: Enumerating objects: 5524, done.
remote: Counting objects: 100% (5524/5524), done.
remote: Compressing objects: 100% (4211/4211), done.
Receiving objects: 100% (5524/5524), 17.76 MiB | 14.45 MiB/s, done.
remote: Total 5524 (delta 1756), reused 2421 (delta 1269), pack-reused 0 (from 0)
Resolving deltas: 100% (1756/1756), done.
Note: switching to 'd79b2d981f28b2730d402244ac3c2e9a8c054eee'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

==> transformers: applying patches/transformers-qwen-rope.patch
../../patches/transformers-qwen-rope.patch:118: trailing whitespace.
    
../../patches/transformers-qwen-rope.patch:274: trailing whitespace.
    return q_embed, k_embed 
warning: 2 lines add whitespace errors.
==> transformers: pip install -e vendor/transformers
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Obtaining file:///quadric/sdk-cli/examples/models/qwen/qwen2.5-0.5b/vendor/transformers
  Installing build dependencies ... [?25l- \ | done
[?25h  Checking if build backend supports build_editable ... [?25ldone
[?25h  Getting requirements to build editable ... [?25l- \ done
[?25h  Preparing editable metadata (pyproject.toml) ... [?25l- \ done
[?25hRequirement already satisfied: filelock in ./qwen_quant_venv/lib/python3.10/site-packages (from transformers==4.55.4) (3.29.4)
Collecting huggingface-hub<1.0,>=0.34.0 (from transformers==4.55.4)
  Downloading huggingface_hub-0.36.2-py3-none-any.whl.metadata (15 kB)
Requirement already satisfied: numpy>=1.17 in ./qwen_quant_venv/lib/python3.10/site-packages (from transformers==4.55.4) (1.26.4)
Requirement already satisfied: packaging>=20.0 in ./qwen_quant_venv/lib/python3.10/site-packages (from transformers==4.55.4) (26.2)
Requirement already satisfied: pyyaml>=5.1 in ./qwen_quant_venv/lib/python3.10/site-packages (from transformers==4.55.4) (6.0.3)
Collecting regex!=2019.12.17 (from transformers==4.55.4)
  Downloading regex-2026.5.9-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (40 kB)
Requirement already satisfied: requests in ./qwen_quant_venv/lib/python3.10/site-packages (from transformers==4.55.4) (2.34.2)
Collecting tokenizers<0.22,>=0.21 (from transformers==4.55.4)
  Downloading tokenizers-0.21.4-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.7 kB)
Collecting safetensors>=0.4.3 (from transformers==4.55.4)
  Downloading safetensors-0.8.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.2 kB)
Requirement already satisfied: tqdm>=4.27 in ./qwen_quant_venv/lib/python3.10/site-packages (from transformers==4.55.4) (4.68.3)
Requirement already satisfied: fsspec>=2023.5.0 in ./qwen_quant_venv/lib/python3.10/site-packages (from huggingface-hub<1.0,>=0.34.0->transformers==4.55.4) (2026.4.0)
Requirement already satisfied: hf-xet<2.0.0,>=1.1.3 in ./qwen_quant_venv/lib/python3.10/site-packages (from huggingface-hub<1.0,>=0.34.0->transformers==4.55.4) (1.5.1)
Requirement already satisfied: typing-extensions>=3.7.4.3 in ./qwen_quant_venv/lib/python3.10/site-packages (from huggingface-hub<1.0,>=0.34.0->transformers==4.55.4) (4.15.0)
Requirement already satisfied: charset_normalizer<4,>=2 in ./qwen_quant_venv/lib/python3.10/site-packages (from requests->transformers==4.55.4) (3.4.7)
Requirement already satisfied: idna<4,>=2.5 in ./qwen_quant_venv/lib/python3.10/site-packages (from requests->transformers==4.55.4) (3.18)
Requirement already satisfied: urllib3<3,>=1.26 in ./qwen_quant_venv/lib/python3.10/site-packages (from requests->transformers==4.55.4) (2.7.0)
Requirement already satisfied: certifi>=2023.5.7 in ./qwen_quant_venv/lib/python3.10/site-packages (from requests->transformers==4.55.4) (2026.6.17)
Downloading huggingface_hub-0.36.2-py3-none-any.whl (566 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 566.4/566.4 kB 21.4 MB/s  0:00:00
[?25hDownloading tokenizers-0.21.4-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.1 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.1/3.1 MB 37.4 MB/s  0:00:00
[?25hDownloading regex-2026.5.9-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (794 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 794.1/794.1 kB 64.2 MB/s  0:00:00
[?25hDownloading safetensors-0.8.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (516 kB)
Building wheels for collected packages: transformers
  Building editable for transformers (pyproject.toml) ... [?25l- \ | / done
[?25h  Created wheel for transformers: filename=transformers-4.55.4-0.editable-py3-none-any.whl size=15528 sha256=baab65c82782c230a4d2bc5af07359c632f1724e8a4dee4e728f8d8d992702b1
  Stored in directory: /tmp/pip-ephem-wheel-cache-azlp3g0r/wheels/95/c8/4e/8bd5b64205269d80194dd843c8d12ebb487cd842498b3c8c7a
Successfully built transformers
Installing collected packages: safetensors, regex, huggingface-hub, tokenizers, transformers
  Attempting uninstall: huggingface-hub
    Found existing installation: huggingface_hub 1.20.1
    Uninstalling huggingface_hub-1.20.1:
      Successfully uninstalled huggingface_hub-1.20.1
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5/5 [transformers]
Successfully installed huggingface-hub-0.36.2 regex-2026.5.9 safetensors-0.8.0 tokenizers-0.21.4 transformers-4.55.4
==> optimum: cloning https://github.com/huggingface/optimum.git at v1.27.0
Cloning into 'vendor/optimum'...
remote: Enumerating objects: 420, done.
remote: Counting objects: 100% (420/420), done.
remote: Compressing objects: 100% (366/366), done.
remote: Total 420 (delta 101), reused 165 (delta 37), pack-reused 0 (from 0)
Receiving objects: 100% (420/420), 933.05 KiB | 3.52 MiB/s, done.
Resolving deltas: 100% (101/101), done.
Note: switching to '51184933521f8c1ef40ce4f75e7972d53cdc59be'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

==> optimum: applying patches/optimum-qwen-rope.patch
==> optimum: pip install -e vendor/optimum
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Obtaining file:///quadric/sdk-cli/examples/models/qwen/qwen2.5-0.5b/vendor/optimum
  Installing build dependencies ... [?25l- \ | / done
[?25h  Checking if build backend supports build_editable ... [?25ldone
[?25h  Getting requirements to build editable ... [?25l- done
[?25h  Preparing editable metadata (pyproject.toml) ... [?25l- done
[?25hRequirement already satisfied: transformers>=4.29 in ./qwen_quant_venv/lib/python3.10/site-packages (from optimum==1.27.0) (4.55.4)
Requirement already satisfied: torch>=1.11 in ./qwen_quant_venv/lib/python3.10/site-packages (from optimum==1.27.0) (2.5.1)
Requirement already satisfied: packaging in ./qwen_quant_venv/lib/python3.10/site-packages (from optimum==1.27.0) (26.2)
Requirement already satisfied: numpy in ./qwen_quant_venv/lib/python3.10/site-packages (from optimum==1.27.0) (1.26.4)
Requirement already satisfied: huggingface_hub>=0.8.0 in ./qwen_quant_venv/lib/python3.10/site-packages (from optimum==1.27.0) (0.36.2)
Requirement already satisfied: filelock in ./qwen_quant_venv/lib/python3.10/site-packages (from huggingface_hub>=0.8.0->optimum==1.27.0) (3.29.4)
Requirement already satisfied: fsspec>=2023.5.0 in ./qwen_quant_venv/lib/python3.10/site-packages (from huggingface_hub>=0.8.0->optimum==1.27.0) (2026.4.0)
Requirement already satisfied: hf-xet<2.0.0,>=1.1.3 in ./qwen_quant_venv/lib/python3.10/site-packages (from huggingface_hub>=0.8.0->optimum==1.27.0) (1.5.1)
Requirement already satisfied: pyyaml>=5.1 in ./qwen_quant_venv/lib/python3.10/site-packages (from huggingface_hub>=0.8.0->optimum==1.27.0) (6.0.3)
Requirement already satisfied: requests in ./qwen_quant_venv/lib/python3.10/site-packages (from huggingface_hub>=0.8.0->optimum==1.27.0) (2.34.2)
Requirement already satisfied: tqdm>=4.42.1 in ./qwen_quant_venv/lib/python3.10/site-packages (from huggingface_hub>=0.8.0->optimum==1.27.0) (4.68.3)
Requirement already satisfied: typing-extensions>=3.7.4.3 in ./qwen_quant_venv/lib/python3.10/site-packages (from huggingface_hub>=0.8.0->optimum==1.27.0) (4.15.0)
Requirement already satisfied: networkx in ./qwen_quant_venv/lib/python3.10/site-packages (from torch>=1.11->optimum==1.27.0) (3.4.2)
Requirement already satisfied: jinja2 in ./qwen_quant_venv/lib/python3.10/site-packages (from torch>=1.11->optimum==1.27.0) (3.1.6)
Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.4.127 in ./qwen_quant_venv/lib/python3.10/site-packages (from torch>=1.11->optimum==1.27.0) (12.4.127)
Requirement already satisfied: nvidia-cuda-runtime-cu12==12.4.127 in ./qwen_quant_venv/lib/python3.10/site-packages (from torch>=1.11->optimum==1.27.0) (12.4.127)
Requirement already satisfied: nvidia-cuda-cupti-cu12==12.4.127 in ./qwen_quant_venv/lib/python3.10/site-packages (from torch>=1.11->optimum==1.27.0) (12.4.127)
Requirement already satisfied: nvidia-cudnn-cu12==9.1.0.70 in ./qwen_quant_venv/lib/python3.10/site-packages (from torch>=1.11->optimum==1.27.0) (9.1.0.70)
Requirement already satisfied: nvidia-cublas-cu12==12.4.5.8 in ./qwen_quant_venv/lib/python3.10/site-packages (from torch>=1.11->optimum==1.27.0) (12.4.5.8)
Requirement already satisfied: nvidia-cufft-cu12==11.2.1.3 in ./qwen_quant_venv/lib/python3.10/site-packages (from torch>=1.11->optimum==1.27.0) (11.2.1.3)
Requirement already satisfied: nvidia-curand-cu12==10.3.5.147 in ./qwen_quant_venv/lib/python3.10/site-packages (from torch>=1.11->optimum==1.27.0) (10.3.5.147)
Requirement already satisfied: nvidia-cusolver-cu12==11.6.1.9 in ./qwen_quant_venv/lib/python3.10/site-packages (from torch>=1.11->optimum==1.27.0) (11.6.1.9)
Requirement already satisfied: nvidia-cusparse-cu12==12.3.1.170 in ./qwen_quant_venv/lib/python3.10/site-packages (from torch>=1.11->optimum==1.27.0) (12.3.1.170)
Requirement already satisfied: nvidia-nccl-cu12==2.21.5 in ./qwen_quant_venv/lib/python3.10/site-packages (from torch>=1.11->optimum==1.27.0) (2.21.5)
Requirement already satisfied: nvidia-nvtx-cu12==12.4.127 in ./qwen_quant_venv/lib/python3.10/site-packages (from torch>=1.11->optimum==1.27.0) (12.4.127)
Requirement already satisfied: nvidia-nvjitlink-cu12==12.4.127 in ./qwen_quant_venv/lib/python3.10/site-packages (from torch>=1.11->optimum==1.27.0) (12.4.127)
Requirement already satisfied: triton==3.1.0 in ./qwen_quant_venv/lib/python3.10/site-packages (from torch>=1.11->optimum==1.27.0) (3.1.0)
Requirement already satisfied: sympy==1.13.1 in ./qwen_quant_venv/lib/python3.10/site-packages (from torch>=1.11->optimum==1.27.0) (1.13.1)
Requirement already satisfied: mpmath<1.4,>=1.1.0 in ./qwen_quant_venv/lib/python3.10/site-packages (from sympy==1.13.1->torch>=1.11->optimum==1.27.0) (1.3.0)
Requirement already satisfied: regex!=2019.12.17 in ./qwen_quant_venv/lib/python3.10/site-packages (from transformers>=4.29->optimum==1.27.0) (2026.5.9)
Requirement already satisfied: tokenizers<0.22,>=0.21 in ./qwen_quant_venv/lib/python3.10/site-packages (from transformers>=4.29->optimum==1.27.0) (0.21.4)
Requirement already satisfied: safetensors>=0.4.3 in ./qwen_quant_venv/lib/python3.10/site-packages (from transformers>=4.29->optimum==1.27.0) (0.8.0)
Requirement already satisfied: MarkupSafe>=2.0 in ./qwen_quant_venv/lib/python3.10/site-packages (from jinja2->torch>=1.11->optimum==1.27.0) (3.0.3)
Requirement already satisfied: charset_normalizer<4,>=2 in ./qwen_quant_venv/lib/python3.10/site-packages (from requests->huggingface_hub>=0.8.0->optimum==1.27.0) (3.4.7)
Requirement already satisfied: idna<4,>=2.5 in ./qwen_quant_venv/lib/python3.10/site-packages (from requests->huggingface_hub>=0.8.0->optimum==1.27.0) (3.18)
Requirement already satisfied: urllib3<3,>=1.26 in ./qwen_quant_venv/lib/python3.10/site-packages (from requests->huggingface_hub>=0.8.0->optimum==1.27.0) (2.7.0)
Requirement already satisfied: certifi>=2023.5.7 in ./qwen_quant_venv/lib/python3.10/site-packages (from requests->huggingface_hub>=0.8.0->optimum==1.27.0) (2026.6.17)
Building wheels for collected packages: optimum
  Building editable for optimum (pyproject.toml) ... [?25l- done
[?25h  Created wheel for optimum: filename=optimum-1.27.0-0.editable-py3-none-any.whl size=11152 sha256=e50d2e4165cbe441ad5f882e84e79e9e3f938228456a15e853c569e0ae009c1f
  Stored in directory: /tmp/pip-ephem-wheel-cache-zd_w3nwp/wheels/4a/5f/f7/de484d307c8973efabc32809ec5c2ae3047aa67226722d856e
Successfully built optimum
Installing collected packages: optimum
Successfully installed optimum-1.27.0

Done. transformers (v4.55.4) and optimum (v1.27.0) installed with quadric patches.

import sys
from pathlib import Path

PIPELINE_DIR = Path.cwd()
SRC_DIR = PIPELINE_DIR / "src"
VENV = PIPELINE_DIR / "qwen_quant_venv"

print(f"Kernel Python: {sys.version.split()[0]}")
print(f"venv:          {VENV}")
print(f"pipeline dir:  {PIPELINE_DIR.name}/  (src/ has {len(list(SRC_DIR.glob('*.py')))} scripts)")

## Confirm the dedicated venv (not the kernel) sees the patched libraries.
!{VENV}/bin/python -c "from importlib.metadata import version; print('transformers:', version('transformers')); print('optimum:     ', version('optimum'))"

Kernel Python: 3.10.12
venv:          /quadric/sdk-cli/examples/models/qwen/qwen2.5-0.5b/qwen_quant_venv
pipeline dir:  qwen2.5-0.5b/  (src/ has 15 scripts)
transformers: 4.55.4
optimum:      1.27.0

2. Pipeline Overview

run_quantization_pipeline.sh orchestrates five stages in order. Each stage is a standalone script in src/ and can also be run by hand for debugging or one-off experiments.

Reference perplexity (Qwen2.5-0.5B-Instruct, SmoothQuant α=0.5, MinMax calibration, 5000 tokens of UltraChat): FP32 7.39 → INT8 9.77.

3. Stage 1 — ONNX Export

get_onnx.py calls optimum.exporters.onnx.export against the patched Qwen3OnnxConfig and writes a single-token autoregressive graph (batch=1, sequence_length=1, past_sequence_length=0). The patched transformers removes the inline RoPE computation, replacing it with two new graph inputs (sin_cache, cos_cache) that the caller fills per position. That's what lets the downstream stages quantize and reshape the model without touching attention math.

The core of the script is the export() call:

import re

src = (SRC_DIR / "get_onnx.py").read_text()
snippet = re.search(r"def generate_autoregressive_onnx.*?(?=\n\nif __name__)", src, re.DOTALL)
print(snippet.group(0))

def generate_autoregressive_onnx(
    model_id="Qwen/Qwen2.5-0.5B-Instruct", onnx_path=DEFAULT_ONNX_PATH, num_decoders=None
):
    """Export Qwen model to ONNX format with KV cache support."""
    print(f"Loading model from: {model_id}")
    base_model = AutoModelForCausalLM.from_pretrained(model_id)
    base_model.config._attn_implementation_autoset = False
    base_model.config._attn_implementation = "eager"
    # base_model.config._attn_implementation = "sdpa"
    if num_decoders is not None:
        base_model.model.layers = base_model.model.layers[:num_decoders]
        base_model.config.num_hidden_layers = num_decoders
        base_model.config.max_window_layers = num_decoders

    print("Base config:", base_model.config)

    onnx_config = Qwen3OnnxConfig(
        config=base_model.config,
        task="text-generation",  # Pass task, might influence other settings
        use_past=True,  # Explicitly enable past mechanism (outputs)
        use_past_in_inputs=True,  # *** Explicitly enable past mechanism (inputs) ***
        # float_dtype = "fp16"
    )

    # seq length is 1 because we are creating the autoregressive graph
    custom_input_shapes = {
        "batch_size": 1,
        "sequence_length": 1,
        "past_sequence_length": 0,
    }

    export(
        base_model,
        onnx_config,
        onnx_path,
        onnx_config.DEFAULT_ONNX_OPSET,
        input_shapes=custom_input_shapes,
        disable_dynamic_axes_fix=True,
        #        dtype="fp16"
        # no_dynamic_axes=True,
    )

4. Stage 2 — SmoothQuant INT8 Quantization

qwen_quant.py is where the quantization itself happens. It uses ONNX Runtime's static quantization with a SmoothQuant pre-pass, which shifts activation magnitude into weights so that per-tensor scales stay tight. The script exposes three families of knobs:

• SmoothQuant strength — --smoothquant-alpha (0.5 by default). Higher α pushes more magnitude into weights; useful when activation outliers dominate.
• Calibration — --calibration-dataset (ultrachat or wikitext), --calibration-samples (128 for production, 10–20 for quick iteration), --calibration-method (MinMax, Entropy, Percentile).
• Op exclusion — down_proj, the Q·K MatMul, Softmax/Sigmoid, and LayerNorm internals are held out by default. --quantize-down-proj, --quantize-up-proj, and --quantize-gate-proj flip individual MLP projections back on if you want to study the cost.

The op exclusion table from README.md:

The exclusion list is built up by name pattern. Take a look at qwen_quant.py for details.

5. Stage 3 — Q/DQ on the Q·K MatMul

Stage 2 leaves the Q·K attention MatMul and its inputs in FP32. In this stage we add a QuantizeLinear → DequantizeLinear (Q/DQ) pair on each of its two inputs (Q and K after RoPE). Quantizing the inputs is a performance win — it lets those tensors travel and compute as INT8, cutting bandwidth and MAC cost on the GPNPU. The MatMul itself and its output stay FP32, which is what matters for accuracy.

6. Stage 4 — Shape Fixing & onnxsim Cleanup

Up through Stage 3 the graph still carries the dynamic batch_size, sequence_length, and past_sequence_length symbolic dimensions — great for perplexity evaluation (which sweeps past_sequence_length) but not deployable. fix_shapes_for_qgemm.py pins those dimensions to concrete values for autoregressive single-token decoding (batch=1, sequence_length=1, past_sequence_length=seq_len-1, default seq_len=1024) and then runs onnxsim.simplify to fold the now-constant shape arithmetic away. The output is cleaned/autoregressive.onnx, the file that is fed to CGC.

7. Stage 5 — Perplexity Evaluation

perplexity_onnx.py measures perplexity by running the dynamic-shape ONNX model (the output of Stage 3, not Stage 4) token-by-token while accumulating the KV cache and the negative log-likelihood. It auto-selects the dataset and chat template based on --model-type: instruct models get UltraChat with the chat template applied; base models get raw WikiText-2.

Two knobs control the speed/accuracy trade-off: --limit (total tokens evaluated; default 5000) and --stride (sliding-window stride; 256 is accurate, 512 is 2× faster). --quick-perplexity is the convenience shortcut that sets --limit 1000 --stride 512 — useful while iterating on calibration knobs.

8. Run the Quick Pipeline

The cell below kicks off all five stages in quick mode: 10 calibration samples and the abbreviated perplexity evaluation. End-to-end wall time is roughly 8 minutes on a recent laptop; full-quality settings (--calib-samples 128, default perplexity limit) take ~30 minutes.

Per-stage logs land in Qwen2.5-0.5B-quick/logs/; the master log is pipeline_<timestamp>.log. The exit status of the cell reflects the script's exit status — a non-zero return means one of the stages failed and the matching 0N_*.log is the place to look.

!./run_quantization_pipeline.sh \
    --output-dir Qwen2.5-0.5B-quick \
    --calib-samples 10 \
    --quick-perplexity

Logging to:
  Master log: Qwen2.5-0.5B-quick/logs/pipeline_20260619_035629.log
  Global log: logs/Qwen2.5-0.5B-quick_20260619_035629.log
  Step logs: Qwen2.5-0.5B-quick/logs/01_*.log through 05_*.log

============================================================================
Qwen Quantization Pipeline - Fri Jun 19 03:56:29 UTC 2026
============================================================================
Command line:
  ./run_quantization_pipeline.sh 

Working directory:
  /quadric/sdk-cli/examples/models/qwen/qwen2.5-0.5b

Environment:
  USER: 
  HOSTNAME: 6b09cca59190
  Python: 

Configuration:
  Output directory: Qwen2.5-0.5B-quick
  Model source: Qwen/Qwen2.5-0.5B-Instruct
  Model type: instruct
  Calibration dataset: ultrachat
  Calibration samples: 10
  Calibration method: MinMax
  SmoothQuant alpha: 0.5
  Fuse Gemm: true
  Add QK Q/DQ: true
  Quantize down_proj: false
  Quantize up_proj: true
  Quantize gate_proj: true
  Run perplexity: true
  Perplexity dataset: ultrachat
  Perplexity limit: 1000 tokens
  Perplexity stride: 512
  Skip ONNX export: false
============================================================================

[1/5] Exporting PyTorch model to ONNX...
/quadric/sdk-cli/examples/models/qwen/qwen2.5-0.5b/qwen_quant_venv/lib/python3.10/site-packages/torch/onnx/_internal/registration.py:168: OnnxExporterWarning: Symbolic function 'aten::scaled_dot_product_attention' already registered for opset 14. Replacing the existing function with new function. This is unexpected. Please report it on https://github.com/pytorch/pytorch/issues.
  warnings.warn(
/quadric/sdk-cli/examples/models/qwen/qwen2.5-0.5b/vendor/transformers/src/transformers/cache_utils.py:108: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if self.keys is None or self.keys.numel() == 0:
Loading model from: Qwen/Qwen2.5-0.5B-Instruct
🚨 `sin` is part of Qwen2Model.forward's signature, but not documented. Make sure to add it to the docstring of the function in /quadric/sdk-cli/examples/models/qwen/qwen2.5-0.5b/vendor/transformers/src/transformers/models/qwen2/modeling_qwen2.py.
🚨 `cos` is part of Qwen2Model.forward's signature, but not documented. Make sure to add it to the docstring of the function in /quadric/sdk-cli/examples/models/qwen/qwen2.5-0.5b/vendor/transformers/src/transformers/models/qwen2/modeling_qwen2.py.
🚨 `sin` is part of Qwen2ForCausalLM.forward's signature, but not documented. Make sure to add it to the docstring of the function in /quadric/sdk-cli/examples/models/qwen/qwen2.5-0.5b/vendor/transformers/src/transformers/models/qwen2/modeling_qwen2.py.
🚨 `cos` is part of Qwen2ForCausalLM.forward's signature, but not documented. Make sure to add it to the docstring of the function in /quadric/sdk-cli/examples/models/qwen/qwen2.5-0.5b/vendor/transformers/src/transformers/models/qwen2/modeling_qwen2.py.
Base config: Qwen2Config {
  "_attn_implementation_autoset": false,
  "architectures": [
    "Qwen2ForCausalLM"
  ],
  "attention_dropout": 0.0,
  "bos_token_id": 151643,
  "eos_token_id": 151645,
  "hidden_act": "silu",
  "hidden_size": 896,
  "initializer_range": 0.02,
  "intermediate_size": 4864,
  "layer_types": [
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention"
  ],
  "max_position_embeddings": 32768,
  "max_window_layers": 21,
  "model_type": "qwen2",
  "num_attention_heads": 14,
  "num_hidden_layers": 24,
  "num_key_value_heads": 2,
  "rms_norm_eps": 1e-06,
  "rope_scaling": null,
  "rope_theta": 1000000.0,
  "sliding_window": null,
  "tie_word_embeddings": true,
  "torch_dtype": "float32",
  "transformers_version": "4.55.4",
  "use_cache": true,
  "use_sliding_window": false,
  "vocab_size": 151936
}

✓ ONNX export complete

[2/5] Quantizing with SmoothQuant (alpha=0.5, method=MinMax)...
INFO:patch_ort_calibration:Applied robust histogram collector patch
INFO:patch_ort_calibration:Applied memory-efficient patch to HistogramCalibrater
INFO:patch_ort_calibration:Applied memory-efficient patch to SmoothQuant calibration
/quadric/sdk-cli/examples/models/qwen/qwen2.5-0.5b/src/qwen_quant.py:180: DeprecationWarning: Call to deprecated class AbstractModelHelper. ('AbstractModelHelper' class is being deprecated. Quadric APIs have been updated to use PyTorch datasets and transforms instead.) -- Deprecated since version 24.01.
  mh_quant = model_helpers.TextModelHelper(tokenize_quant, 500)
INFO:onnxruntime.quantization.shape_inference:Performing symbolic shape inference...
INFO:onnxruntime.quantization.shape_inference:Performing symbolic shape inference...
Model: Qwen/Qwen2.5-0.5B-Instruct (24 layers)
Downloading UltraChat calibration dataset...

=== Fusing MatMul+Add patterns into Gemm ===
Loading model: Qwen2.5-0.5B-quick/onnx_original_export/model.onnx
Found 72 MatMul+Add fusion candidates
  Fusing: /model/layers.0/self_attn/q_proj/MatMul + /model/layers.0/self_attn/q_proj/Add
  Fusing: /model/layers.0/self_attn/k_proj/MatMul + /model/layers.0/self_attn/k_proj/Add
  Fusing: /model/layers.0/self_attn/v_proj/MatMul + /model/layers.0/self_attn/v_proj/Add
  Fusing: /model/layers.1/self_attn/q_proj/MatMul + /model/layers.1/self_attn/q_proj/Add
  Fusing: /model/layers.1/self_attn/k_proj/MatMul + /model/layers.1/self_attn/k_proj/Add
  Fusing: /model/layers.1/self_attn/v_proj/MatMul + /model/layers.1/self_attn/v_proj/Add
  Fusing: /model/layers.2/self_attn/q_proj/MatMul + /model/layers.2/self_attn/q_proj/Add
  Fusing: /model/layers.2/self_attn/k_proj/MatMul + /model/layers.2/self_attn/k_proj/Add
  Fusing: /model/layers.2/self_attn/v_proj/MatMul + /model/layers.2/self_attn/v_proj/Add
  Fusing: /model/layers.3/self_attn/q_proj/MatMul + /model/layers.3/self_attn/q_proj/Add
  Fusing: /model/layers.3/self_attn/k_proj/MatMul + /model/layers.3/self_attn/k_proj/Add
  Fusing: /model/layers.3/self_attn/v_proj/MatMul + /model/layers.3/self_attn/v_proj/Add
  Fusing: /model/layers.4/self_attn/q_proj/MatMul + /model/layers.4/self_attn/q_proj/Add
  Fusing: /model/layers.4/self_attn/k_proj/MatMul + /model/layers.4/self_attn/k_proj/Add
  Fusing: /model/layers.4/self_attn/v_proj/MatMul + /model/layers.4/self_attn/v_proj/Add
  Fusing: /model/layers.5/self_attn/q_proj/MatMul + /model/layers.5/self_attn/q_proj/Add
  Fusing: /model/layers.5/self_attn/k_proj/MatMul + /model/layers.5/self_attn/k_proj/Add
  Fusing: /model/layers.5/self_attn/v_proj/MatMul + /model/layers.5/self_attn/v_proj/Add
  Fusing: /model/layers.6/self_attn/q_proj/MatMul + /model/layers.6/self_attn/q_proj/Add
  Fusing: /model/layers.6/self_attn/k_proj/MatMul + /model/layers.6/self_attn/k_proj/Add
  Fusing: /model/layers.6/self_attn/v_proj/MatMul + /model/layers.6/self_attn/v_proj/Add
  Fusing: /model/layers.7/self_attn/q_proj/MatMul + /model/layers.7/self_attn/q_proj/Add
  Fusing: /model/layers.7/self_attn/k_proj/MatMul + /model/layers.7/self_attn/k_proj/Add
  Fusing: /model/layers.7/self_attn/v_proj/MatMul + /model/layers.7/self_attn/v_proj/Add
  Fusing: /model/layers.8/self_attn/q_proj/MatMul + /model/layers.8/self_attn/q_proj/Add
  Fusing: /model/layers.8/self_attn/k_proj/MatMul + /model/layers.8/self_attn/k_proj/Add
  Fusing: /model/layers.8/self_attn/v_proj/MatMul + /model/layers.8/self_attn/v_proj/Add
  Fusing: /model/layers.9/self_attn/q_proj/MatMul + /model/layers.9/self_attn/q_proj/Add
  Fusing: /model/layers.9/self_attn/k_proj/MatMul + /model/layers.9/self_attn/k_proj/Add
  Fusing: /model/layers.9/self_attn/v_proj/MatMul + /model/layers.9/self_attn/v_proj/Add
  Fusing: /model/layers.10/self_attn/q_proj/MatMul + /model/layers.10/self_attn/q_proj/Add
  Fusing: /model/layers.10/self_attn/k_proj/MatMul + /model/layers.10/self_attn/k_proj/Add
  Fusing: /model/layers.10/self_attn/v_proj/MatMul + /model/layers.10/self_attn/v_proj/Add
  Fusing: /model/layers.11/self_attn/q_proj/MatMul + /model/layers.11/self_attn/q_proj/Add
  Fusing: /model/layers.11/self_attn/k_proj/MatMul + /model/layers.11/self_attn/k_proj/Add
  Fusing: /model/layers.11/self_attn/v_proj/MatMul + /model/layers.11/self_attn/v_proj/Add
  Fusing: /model/layers.12/self_attn/q_proj/MatMul + /model/layers.12/self_attn/q_proj/Add
  Fusing: /model/layers.12/self_attn/k_proj/MatMul + /model/layers.12/self_attn/k_proj/Add
  Fusing: /model/layers.12/self_attn/v_proj/MatMul + /model/layers.12/self_attn/v_proj/Add
  Fusing: /model/layers.13/self_attn/q_proj/MatMul + /model/layers.13/self_attn/q_proj/Add
  Fusing: /model/layers.13/self_attn/k_proj/MatMul + /model/layers.13/self_attn/k_proj/Add
  Fusing: /model/layers.13/self_attn/v_proj/MatMul + /model/layers.13/self_attn/v_proj/Add
  Fusing: /model/layers.14/self_attn/q_proj/MatMul + /model/layers.14/self_attn/q_proj/Add
  Fusing: /model/layers.14/self_attn/k_proj/MatMul + /model/layers.14/self_attn/k_proj/Add
  Fusing: /model/layers.14/self_attn/v_proj/MatMul + /model/layers.14/self_attn/v_proj/Add
  Fusing: /model/layers.15/self_attn/q_proj/MatMul + /model/layers.15/self_attn/q_proj/Add
  Fusing: /model/layers.15/self_attn/k_proj/MatMul + /model/layers.15/self_attn/k_proj/Add
  Fusing: /model/layers.15/self_attn/v_proj/MatMul + /model/layers.15/self_attn/v_proj/Add
  Fusing: /model/layers.16/self_attn/q_proj/MatMul + /model/layers.16/self_attn/q_proj/Add
  Fusing: /model/layers.16/self_attn/k_proj/MatMul + /model/layers.16/self_attn/k_proj/Add
  Fusing: /model/layers.16/self_attn/v_proj/MatMul + /model/layers.16/self_attn/v_proj/Add
  Fusing: /model/layers.17/self_attn/q_proj/MatMul + /model/layers.17/self_attn/q_proj/Add
  Fusing: /model/layers.17/self_attn/k_proj/MatMul + /model/layers.17/self_attn/k_proj/Add
  Fusing: /model/layers.17/self_attn/v_proj/MatMul + /model/layers.17/self_attn/v_proj/Add
  Fusing: /model/layers.18/self_attn/q_proj/MatMul + /model/layers.18/self_attn/q_proj/Add
  Fusing: /model/layers.18/self_attn/k_proj/MatMul + /model/layers.18/self_attn/k_proj/Add
  Fusing: /model/layers.18/self_attn/v_proj/MatMul + /model/layers.18/self_attn/v_proj/Add
  Fusing: /model/layers.19/self_attn/q_proj/MatMul + /model/layers.19/self_attn/q_proj/Add
  Fusing: /model/layers.19/self_attn/k_proj/MatMul + /model/layers.19/self_attn/k_proj/Add
  Fusing: /model/layers.19/self_attn/v_proj/MatMul + /model/layers.19/self_attn/v_proj/Add
  Fusing: /model/layers.20/self_attn/q_proj/MatMul + /model/layers.20/self_attn/q_proj/Add
  Fusing: /model/layers.20/self_attn/k_proj/MatMul + /model/layers.20/self_attn/k_proj/Add
  Fusing: /model/layers.20/self_attn/v_proj/MatMul + /model/layers.20/self_attn/v_proj/Add
  Fusing: /model/layers.21/self_attn/q_proj/MatMul + /model/layers.21/self_attn/q_proj/Add
  Fusing: /model/layers.21/self_attn/k_proj/MatMul + /model/layers.21/self_attn/k_proj/Add
  Fusing: /model/layers.21/self_attn/v_proj/MatMul + /model/layers.21/self_attn/v_proj/Add
  Fusing: /model/layers.22/self_attn/q_proj/MatMul + /model/layers.22/self_attn/q_proj/Add
  Fusing: /model/layers.22/self_attn/k_proj/MatMul + /model/layers.22/self_attn/k_proj/Add
  Fusing: /model/layers.22/self_attn/v_proj/MatMul + /model/layers.22/self_attn/v_proj/Add
  Fusing: /model/layers.23/self_attn/q_proj/MatMul + /model/layers.23/self_attn/q_proj/Add
  Fusing: /model/layers.23/self_attn/k_proj/MatMul + /model/layers.23/self_attn/k_proj/Add
  Fusing: /model/layers.23/self_attn/v_proj/MatMul + /model/layers.23/self_attn/v_proj/Add

Sorting nodes topologically...
  Successfully sorted 6932 nodes
Saving fused model to: Qwen2.5-0.5B-quick/quantized/preprocessed_float/fused_gemm.onnx

Summary:
  Gemm nodes created: 72
  Remaining MatMul nodes: 145
  Remaining Add nodes: 169
  Reshape nodes added: 384
=== Fusion complete ===

Collecting calibration data
Processed
Generating train split: 10000 examples [00:00, 49007.64 examples/s]
2026-06-19 03:58:06 [INFO] Start smooth model calibration.
INFO:patch_ort_calibration:SmoothQuant calibrated 10 samples...
INFO:patch_ort_calibration:SmoothQuant calibration complete: 10 samples
2026-06-19 03:58:24 [INFO] Start smooth scales collection.
WARNING:root:Please use QuantFormat.QDQ for activation type QInt8 and weight type QInt8. Or it will lead to bad performance on x64.
Generating RoPE embeddings for Qwen/Qwen2.5-0.5B-Instruct...
🚨 `sin` is part of Qwen2Model.forward's signature, but not documented. Make sure to add it to the docstring of the function in /quadric/sdk-cli/examples/models/qwen/qwen2.5-0.5b/vendor/transformers/src/transformers/models/qwen2/modeling_qwen2.py.
🚨 `cos` is part of Qwen2Model.forward's signature, but not documented. Make sure to add it to the docstring of the function in /quadric/sdk-cli/examples/models/qwen/qwen2.5-0.5b/vendor/transformers/src/transformers/models/qwen2/modeling_qwen2.py.
🚨 `sin` is part of Qwen2ForCausalLM.forward's signature, but not documented. Make sure to add it to the docstring of the function in /quadric/sdk-cli/examples/models/qwen/qwen2.5-0.5b/vendor/transformers/src/transformers/models/qwen2/modeling_qwen2.py.
🚨 `cos` is part of Qwen2ForCausalLM.forward's signature, but not documented. Make sure to add it to the docstring of the function in /quadric/sdk-cli/examples/models/qwen/qwen2.5-0.5b/vendor/transformers/src/transformers/models/qwen2/modeling_qwen2.py.
Generated sin/cos shape: (1, 1024, 64)
[1, 500]
[1, 500, 500]
[1, 500, 64]
[1, 500, 64]
[1, 2, 0, 64]
[1, 2, 0, 64]
[1, 2, 0, 64]
[1, 2, 0, 64]
[1, 2, 0, 64]
[1, 2, 0, 64]
[1, 2, 0, 64]
[1, 2, 0, 64]
[1, 2, 0, 64]
[1, 2, 0, 64]
[1, 2, 0, 64]
[1, 2, 0, 64]
[1, 2, 0, 64]
[1, 2, 0, 64]
[1, 2, 0, 64]
[1, 2, 0, 64]
[1, 2, 0, 64]
[1, 2, 0, 64]
[1, 2, 0, 64]
[1, 2, 0, 64]
[1, 2, 0, 64]
[1, 2, 0, 64]
[1, 2, 0, 64]
[1, 2, 0, 64]
[1, 2, 0, 64]
[1, 2, 0, 64]
[1, 2, 0, 64]
[1, 2, 0, 64]
[1, 2, 0, 64]
[1, 2, 0, 64]
[1, 2, 0, 64]
[1, 2, 0, 64]
[1, 2, 0, 64]
[1, 2, 0, 64]
[1, 2, 0, 64]
[1, 2, 0, 64]
[1, 2, 0, 64]
[1, 2, 0, 64]
[1, 2, 0, 64]
[1, 2, 0, 64]
[1, 2, 0, 64]
[1, 2, 0, 64]
[1, 2, 0, 64]
[1, 2, 0, 64]
[1, 2, 0, 64]
[1, 2, 0, 64]
[1, 2, 0, 64]
[1, 2, 0, 64]
QUANTIZING
Progress: [####################] 100.00%Stage 3: Creating tensor ranges with MinMax method...
Creating tensor ranges file...
Collecting calibration data for tensor ranges...
Computing tensor ranges...
Got 2517 tensor ranges
Saved tensor ranges to Qwen2.5-0.5B-quick/quantized/quantized/optimized_opt_sym_int8_q.onnx.tranges
✓ Quantization complete

[3/5] Adding Q/DQ nodes on QK MatMul inputs...
Auto-detected 24 layers from Qwen/Qwen2.5-0.5B-Instruct
Loading model: Qwen2.5-0.5B-quick/quantized/quantized/optimized_opt_sym_int8_q.onnx
Loading tensor ranges: Qwen2.5-0.5B-quick/quantized/quantized/optimized_opt_sym_int8_q.onnx.tranges
Found 362 existing quantized tensors in graph

Processing 24 layers...
  Layer 0:
    Q input: /model/layers.0/self_attn/Concat_5_output_0
    K input: /model/layers.0/self_attn/Transpose_3_output_0
    Q scale: 0.634528, zp: 0
    K scale: 1.027281, zp: 0
  Layer 1:
    Q input: /model/layers.1/self_attn/Concat_5_output_0
    K input: /model/layers.1/self_attn/Transpose_3_output_0
    Q scale: 0.123152, zp: 0
    K scale: 1.174882, zp: 0
  Layer 2:
    Q input: /model/layers.2/self_attn/Concat_5_output_0
    K input: /model/layers.2/self_attn/Transpose_3_output_0
    Q scale: 0.082489, zp: 0
    K scale: 0.494956, zp: 0
  Layer 3:
    Q input: /model/layers.3/self_attn/Concat_5_output_0
    K input: /model/layers.3/self_attn/Transpose_3_output_0
    Q scale: 0.148861, zp: 0
    K scale: 0.207596, zp: 0
  Layer 4:
    Q input: /model/layers.4/self_attn/Concat_5_output_0
    K input: /model/layers.4/self_attn/Transpose_3_output_0
    Q scale: 0.176336, zp: 0
    K scale: 0.155155, zp: 0
  Layer 5:
    Q input: /model/layers.5/self_attn/Concat_5_output_0
    K input: /model/layers.5/self_attn/Transpose_3_output_0
    Q scale: 0.179164, zp: 0
    K scale: 0.126087, zp: 0
  Layer 6:
    Q input: /model/layers.6/self_attn/Concat_5_output_0
    K input: /model/layers.6/self_attn/Transpose_3_output_0
    Q scale: 0.144655, zp: 0
    K scale: 0.087886, zp: 0
  Layer 7:
    Q input: /model/layers.7/self_attn/Concat_5_output_0
    K input: /model/layers.7/self_attn/Transpose_3_output_0
    Q scale: 0.223060, zp: 0
    K scale: 0.108387, zp: 0
  Layer 8:
    Q input: /model/layers.8/self_attn/Concat_5_output_0
    K input: /model/layers.8/self_attn/Transpose_3_output_0
    Q scale: 0.259096, zp: 0
    K scale: 1.742281, zp: 0
  Layer 9:
    Q input: /model/layers.9/self_attn/Concat_5_output_0
    K input: /model/layers.9/self_attn/Transpose_3_output_0
    Q scale: 0.268283, zp: 0
    K scale: 0.137111, zp: 0
  Layer 10:
    Q input: /model/layers.10/self_attn/Concat_5_output_0
    K input: /model/layers.10/self_attn/Transpose_3_output_0
    Q scale: 0.185854, zp: 0
    K scale: 0.125851, zp: 0
  Layer 11:
    Q input: /model/layers.11/self_attn/Concat_5_output_0
    K input: /model/layers.11/self_attn/Transpose_3_output_0
    Q scale: 0.294344, zp: 0
    K scale: 0.160704, zp: 0
  Layer 12:
    Q input: /model/layers.12/self_attn/Concat_5_output_0
    K input: /model/layers.12/self_attn/Transpose_3_output_0
    Q scale: 0.199709, zp: 0
    K scale: 0.101523, zp: 0
  Layer 13:
    Q input: /model/layers.13/self_attn/Concat_5_output_0
    K input: /model/layers.13/self_attn/Transpose_3_output_0
    Q scale: 0.195554, zp: 0
    K scale: 0.093981, zp: 0
  Layer 14:
    Q input: /model/layers.14/self_attn/Concat_5_output_0
    K input: /model/layers.14/self_attn/Transpose_3_output_0
    Q scale: 0.178354, zp: 0
    K scale: 0.142587, zp: 0
  Layer 15:
    Q input: /model/layers.15/self_attn/Concat_5_output_0
    K input: /model/layers.15/self_attn/Transpose_3_output_0
    Q scale: 0.190585, zp: 0
    K scale: 0.118886, zp: 0
  Layer 16:
    Q input: /model/layers.16/self_attn/Concat_5_output_0
    K input: /model/layers.16/self_attn/Transpose_3_output_0
    Q scale: 0.226330, zp: 0
    K scale: 0.114348, zp: 0
  Layer 17:
    Q input: /model/layers.17/self_attn/Concat_5_output_0
    K input: /model/layers.17/self_attn/Transpose_3_output_0
    Q scale: 0.194748, zp: 0
    K scale: 0.116256, zp: 0
  Layer 18:
    Q input: /model/layers.18/self_attn/Concat_5_output_0
    K input: /model/layers.18/self_attn/Transpose_3_output_0
    Q scale: 0.227994, zp: 0
    K scale: 0.128154, zp: 0
  Layer 19:
    Q input: /model/layers.19/self_attn/Concat_5_output_0
    K input: /model/layers.19/self_attn/Transpose_3_output_0
    Q scale: 0.193930, zp: 0
    K scale: 0.146158, zp: 0
  Layer 20:
    Q input: /model/layers.20/self_attn/Concat_5_output_0
    K input: /model/layers.20/self_attn/Transpose_3_output_0
    Q scale: 0.149875, zp: 0
    K scale: 0.178503, zp: 0
  Layer 21:
    Q input: /model/layers.21/self_attn/Concat_5_output_0
    K input: /model/layers.21/self_attn/Transpose_3_output_0
    Q scale: 0.138174, zp: 0
    K scale: 0.126289, zp: 0
  Layer 22:
    Q input: /model/layers.22/self_attn/Concat_5_output_0
    K input: /model/layers.22/self_attn/Transpose_3_output_0
    Q scale: 0.094135, zp: 0
    K scale: 0.083985, zp: 0
  Layer 23:
    Q input: /model/layers.23/self_attn/Concat_5_output_0
    K input: /model/layers.23/self_attn/Transpose_3_output_0
    Q scale: 0.164128, zp: 0
    K scale: 0.125224, zp: 0

Adding 96 new nodes to graph
Adding 96 new initializers to graph
Adding 96 new value_infos to graph

Sorting graph nodes topologically...
Topological sort complete

Saving modified model to: Qwen2.5-0.5B-quick/quantized/quantized/model_with_qk_qdq.onnx
Done!

=== Summary ===
Total nodes added: 96
Total initializers added: 96
Total value_infos added: 96
Expected: 96 nodes (24 layers × 2 inputs × 2 Q/DQ nodes)
Expected: 96 initializers (24 layers × 2 inputs × 2 params)
Expected: 96 value_infos (24 layers × 2 inputs × 2 tensors)

Model remains dynamic for autoregressive inference.
Run fix_shapes.py later to make it static for deployment.
✓ Q/DQ nodes added

[4/5] Fixing shapes and optimizing with fix_shapes_for_qgemm.py...
Fixing shapes for QGemm model: Qwen2.5-0.5B-quick/quantized/quantized/model_with_qk_qdq.onnx -> Qwen2.5-0.5B-quick/cleaned/autoregressive.onnx
Loading model: Qwen2.5-0.5B-quick/quantized/quantized/model_with_qk_qdq.onnx
  Nodes before fixing shapes: 7559
  Fixed dimensions: batch=1, seq=1, past_seq=1023
  Current ir_version: 7
  Saving intermediate model before simplification...
  Running onnxsim simplification...
  Nodes after simplification: 2439
  Reduction: 5120 nodes (67.7%)
  Saved to: Qwen2.5-0.5B-quick/cleaned/autoregressive.onnx
  External data: Qwen2.5-0.5B-quick/cleaned/autoregressive.onnx.data
Done!
✓ Shapes fixed and graph optimized (uses onnxsim internally)

[5/5] Evaluating perplexity on ultrachat...
/quadric/sdk-cli/examples/models/qwen/qwen2.5-0.5b/qwen_quant_venv/lib/python3.10/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py:123: UserWarning: Specified provider 'CUDAExecutionProvider' is not in available provider names.Available providers: 'AzureExecutionProvider, CPUExecutionProvider'
  warnings.warn(
Token indices sequence length is longer than the specified maximum sequence length for this model (27753826 > 131072). Running this sequence through the model will result in indexing errors
🚨 `sin` is part of Qwen2Model.forward's signature, but not documented. Make sure to add it to the docstring of the function in /quadric/sdk-cli/examples/models/qwen/qwen2.5-0.5b/vendor/transformers/src/transformers/models/qwen2/modeling_qwen2.py.
🚨 `cos` is part of Qwen2Model.forward's signature, but not documented. Make sure to add it to the docstring of the function in /quadric/sdk-cli/examples/models/qwen/qwen2.5-0.5b/vendor/transformers/src/transformers/models/qwen2/modeling_qwen2.py.
🚨 `sin` is part of Qwen2ForCausalLM.forward's signature, but not documented. Make sure to add it to the docstring of the function in /quadric/sdk-cli/examples/models/qwen/qwen2.5-0.5b/vendor/transformers/src/transformers/models/qwen2/modeling_qwen2.py.
🚨 `cos` is part of Qwen2ForCausalLM.forward's signature, but not documented. Make sure to add it to the docstring of the function in /quadric/sdk-cli/examples/models/qwen/qwen2.5-0.5b/vendor/transformers/src/transformers/models/qwen2/modeling_qwen2.py.
============================================================
ONNX Model Perplexity Calculator (Autoregressive)
============================================================
Model: Qwen2.5-0.5B-quick/quantized/quantized/model_with_qk_qdq.onnx
Model ID: Qwen/Qwen2.5-0.5B-Instruct
Dataset: HuggingFaceH4/ultrachat_200k/default (test_sft)
Max length: 512
Stride: 512
Chat template: enabled
============================================================

Model config: 24 layers, 2 KV heads, 64 head dim

Loading ONNX model: Qwen2.5-0.5B-quick/quantized/quantized/model_with_qk_qdq.onnx
Active providers: ['CPUExecutionProvider']
Preparing RoPE embeddings...
Loading tokenizer...
Loading dataset...
Applying chat template to dataset...
Using ultrachat 'messages' format
Tokenizing...
Limiting to 1000 tokens
Total tokens: 1,000

Calculating perplexity...
Total tokens: 1000
Windows:  50%|█████     | 1/2 [03:23<03:23, 203.83s/it]

============================================================
RESULTS
============================================================
Total tokens scored: 998
Average NLL: 3.4587
Perplexity: 31.7753
============================================================
✓ Perplexity evaluation complete

============================================================================
Pipeline Complete!
============================================================================
Output directory: Qwen2.5-0.5B-quick

Key models:
  - Dynamic (for testing):  Qwen2.5-0.5B-quick/quantized/quantized/model_with_qk_qdq.onnx
  - Static (for deployment): Qwen2.5-0.5B-quick/cleaned/autoregressive.onnx

Logs saved to:
  - Master log: Qwen2.5-0.5B-quick/logs/pipeline_20260619_035629.log
  - Step logs: Qwen2.5-0.5B-quick/logs/01_*.log through 05_*.log
============================================================================

9. Inspect Outputs

After the run, the output directory layout is:

Qwen2.5-0.5B-quick/
├── onnx_original_export/model.onnx                              # Stage 1 (float32)
├── quantized/quantized/optimized_opt_sym_int8_q.onnx{,.data}     # Stage 2 (INT8, dynamic)
├── quantized/quantized/optimized_opt_sym_int8_q.onnx.tranges     # Stage 2 (tensor ranges)
├── quantized/quantized/model_with_qk_qdq.onnx                    # Stage 3
├── cleaned/autoregressive.onnx                                   # Stage 4 (deployment model)
└── logs/                                                         # per-stage + master logs

The cell below lists the key artifacts with their sizes — the dynamic-shape INT8 model is the one used for perplexity, the static-shape cleaned/autoregressive.onnx is what you'd hand to CGC for compilation.

OUTPUT_DIR = PIPELINE_DIR / "Qwen2.5-0.5B-quick"

artifacts = [
    ("Float32 export (Stage 1)", OUTPUT_DIR / "onnx_original_export" / "model.onnx"),
    (
        "INT8 dynamic (Stage 2)",
        OUTPUT_DIR / "quantized" / "quantized" / "optimized_opt_sym_int8_q.onnx",
    ),
    ("With QK Q/DQ (Stage 3)", OUTPUT_DIR / "quantized" / "quantized" / "model_with_qk_qdq.onnx"),
    ("Deployment (Stage 4)", OUTPUT_DIR / "cleaned" / "autoregressive.onnx"),
]

for label, path in artifacts:
    if path.exists():
        size_mb = path.stat().st_size / 1024 / 1024
        print(f"{label:<28} {size_mb:>7.1f} MB  {path.relative_to(PIPELINE_DIR)}")
    else:
        print(f"{label:<28} {'missing':>11}  {path.relative_to(PIPELINE_DIR)}")

Float32 export (Stage 1)         1.2 MB  Qwen2.5-0.5B-quick/onnx_original_export/model.onnx
INT8 dynamic (Stage 2)           3.0 MB  Qwen2.5-0.5B-quick/quantized/quantized/optimized_opt_sym_int8_q.onnx
With QK Q/DQ (Stage 3)           3.0 MB  Qwen2.5-0.5B-quick/quantized/quantized/model_with_qk_qdq.onnx
Deployment (Stage 4)             1.8 MB  Qwen2.5-0.5B-quick/cleaned/autoregressive.onnx

10. Inference Demo

src/autoregressive_runner.py drives the model token-by-token through ONNX Runtime, supplying the patched sin/cos inputs from transformers.Qwen2RotaryEmbedding. It's the same loop CGC's runtime would execute on a GPNPU — useful as a sanity check that the quantized weights still produce a coherent reply before committing to a compile run.

It runs against the dynamic-shape model from Stage 3 (quantized/quantized/model_with_qk_qdq.onnx), because the runner does a multi-token prefill (all prompt tokens in one pass) before single-token decode. The static cleaned/autoregressive.onnx is fixed to a single-token step (input_ids [1, 1], 1024-token context) for hardware deployment and isn't directly drivable by this runner.

The default prompt is "What is a quadric?". Pass --temperature 0.0 for deterministic greedy decoding, or --no-template to skip the chat template and run a raw completion.

!qwen_quant_venv/bin/python src/autoregressive_runner.py \
    --model Qwen2.5-0.5B-quick/quantized/quantized/model_with_qk_qdq.onnx \
    --prompt "What is a quadric?" \
    --max-tokens 80 \
    --temperature 0.0 \
    --quiet

Prompt:   What is a quadric?

Response: A quadric is a type of conic section in geometry, specifically in the plane. It is a three-dimensional figure that is the intersection of a plane and a double cone. It is a type of conic section, which is a curve that is formed by the intersection of a plane and a double cone.

Next step

Hand cleaned/autoregressive.onnx to ChimeraJob along with its .tranges file to compile the model onto a Chimera GPNPU. The examples/models/qwen/qwen2.5-1.5b/qwen2.5.ipynb notebook shows the compile-and-run flow for the 1.5 B sibling and is the natural next read.