NOTE: The Jupyter Notebook below is included in the Chimera SDK and can be run interactively by running the following CLI command:
$ quadric sdk notebook
From the Jupyter Notebook window in your browser, select the notebook named /quadric/sdk-cli/examples/models/sarashina2.2-3b/sarashina_pipeline.ipynb.
Sarashina2.2-3B Text Generation on Chimera GPNPU
This notebook runs the whole path from the Hugging Face model card to a token decoded on Quadric silicon, one stage per cell, and explains what each stage produces. Every stage function lives in sarashina_helpers.py; this notebook only calls them in order.
Pipeline
Each stage also owns one directory of the staged workdir:
fp32 ONNX ─► SmoothQuant ranges ─► int4 weights + static shapes graphs/
─► attention & projections as Chimera custom ops custom_op/<knobs>/
─► CGC: generated C++ + 2.7 GB constant blob cgc/<knobs>/<hw>/
─► driver assembled + compiled to an EPU kernel kernel/<knobs>/<hw>_nt<N>/
─► archsim run (ISS) ─or─ board handoff for the FPGA runner runs/ ─or─ output/<tag>/
The workdir is staged. Each stage writes into its own directory under iss/work_w4a16_fused/ and leaves a STAGE.json recording the inputs, knobs and toolchain it used. A stage is rebuilt only when that record no longer matches what is on disk now, so re-running this notebook after a knob change rebuilds exactly the stages the knob reaches and reuses the rest. (graphs/, hours to regenerate, is the exception: it is reused whenever present and a warning names what moved.) The README's Layout section describes the directories and the manifests.
Two independent choices. TARGET is the silicon (QC-N / QC-P / QC-U): the only knob the compiler sees. DEVICE is where the finished kernel runs: "iss" executes it on the archsim here, "haps" publishes a self-contained handoff directory for the FPGA board runner.
Environment. The sdk-cli README's from-source setup is assumed: SDK_PATH, SDK_INSTALL_PATH, TVM_PATH and QLLVM_ROOT_PATH exported, and this notebook running in the Python that has tvm and sdk_cli installed. Nothing here depends on a particular home directory.
0. Knobs
Everything configurable is in this cell. The defaults are QC-U on the archsim with the shipped graph variant (packed per-member projections, frac-bit headroom 2).
## ---- what to build, where to run it -------------------------------------------------------------
## TARGET: "qcn" (array 8) | "qcp" (array 16, the HAPS-200 bitstream) | "qcu" (array 32)
## DEVICE: "iss" executes on the archsim in this notebook; "haps" publishes output/<tag>/ for the
## FPGA board runner
TARGET = "qcu"
DEVICE = "iss"
## ---- what to say --------------------------------------------------------------------------------
PROMPT = "What is AI?" # rendered through the chat template: <|user|>{PROMPT}</s><|assistant|>
NEW_TOKENS = None # None -> 1 on the ISS (one forward pass), 128 for the board; baked in
## ---- quantization front end: where the int4 graphs come from ------------------------------------
## "fetch": download the published graphs (iss/graphs_manifest.json, 5.1 GB, sha256-verified);
## the quantization stage never runs. The default.
## "run": produce them here with run_all.sh Stages 1-2 (hours; needs `bash run_all.sh setup`).
QUANTIZE = "fetch"
## ---- graph variant (match_att knobs; they name the custom_op/, cgc/ and kernel/ directories) ----
## PROJ_MODE: "packed" = q|k|v and gate|up each one op, per-member smoothing (no int4 refit);
## "fused" = one shared smoothing vector, weights re-fitted to a fresh int4 grid.
## IMF_HEADROOM: range headroom bits on every w4a16 output. 2 = best measured (ISS 0.0554 vs ORT);
## 3 = the more conservative shipped-first build (0.0590).
PROJ_MODE = "packed"
IMF_HEADROOM = 2
FUSE_V = True # keep v in the q|k|v group (v still gets its own output edge)
KV_PERSIST_MB = 64 # persistent int8 KV cache: 32 layers x 512 positions needs ~41 MB
## ---- housekeeping -------------------------------------------------------------------------------
## VARIANT: with DEVICE="haps", publish into output/<tag>_<VARIANT>/ instead of output/<tag>/.
## FORCE: stages to rebuild although their STAGE.json is current,
## e.g. {"quantize", "match_att", "cgc", "kernel"}.
VARIANT = ""
FORCE = set()
import sarashina_helpers as sara
cfg = sara.Config.for_target(TARGET, DEVICE, prompt=PROMPT, new_tokens=NEW_TOKENS)
cfg.proj_mode, cfg.imf_headroom, cfg.fuse_v = PROJ_MODE, IMF_HEADROOM, FUSE_V
cfg.kv_persist_mb, cfg.variant = KV_PERSIST_MB, VARIANT
print(f"Silicon: {cfg.product} (array_size {cfg.array_size})")
where = "FPGA board handoff" if cfg.device == "haps" else "Chimera archsim (ISS)"
print(f"Device: {cfg.device} -> {where}")
print(f"Hardware: {cfg.hw_tag}")
print(f"Variant: {cfg.knob_tag}")
print(f"Decode: {cfg.new_tokens} new token(s) for prompt {cfg.prompt!r}")
print()
sara.stage_status(cfg)
Silicon: QC-U (array_size 32)
Device: iss -> Chimera archsim (ISS)
Hardware: QC-U_1d5_8MB_4kB_128GBps_128GBps_8_OFF_x1_x1
Variant: packed_hr2
Decode: 1 new token(s) for prompt 'What is AI?'
workdir /quadric/sdk-cli/examples/models/sarashina2.2-3b/iss/work_w4a16_fused
variant packed_hr2 hardware QC-U_1d5_8MB_4kB_128GBps_128GBps_8_OFF_x1_x1 new_tokens 1
graphs graphs -- not built
custom_op custom_op/packed_hr2 -- not built
cgc cgc/packed_hr2/QC-U_1d5_8MB_4kB_128GBps_128GBps_8_OFF_x1_x1 -- not built
kernel kernel/packed_hr2/QC-U_1d5_8MB_4kB_128GBps_128GBps_8_OFF_x1_x1_nt1 -- not built
inputs inputs/what_is_ai_p128_n1 -- not built
runs none yet
1. Toolchain
The compile steps need the TVM checkout (TVM_PATH, for the tvm.contrib.epu compiler and the C++ headers), the Quadric clang (QLLVM_ROOT_PATH), and an SDK install built for this target's array size (NUM_CORES is baked into the SDK). setup_env() fills in what the environment left unset from the interpreter running this notebook; ensure_sdk() uses SDK_INSTALL_PATH when its core count matches, else an install this example built earlier, else the SDK_PATH checkout's own build/install, else builds one from that checkout (minutes, once per array size).
sara.setup_env()
sdk = sara.ensure_sdk(cfg)
PY_COMPILE /usr/bin/python3
TVM_PATH /quadric/tvm/
QLLVM_ROOT_PATH /quadric/llvm (clang only -- not the SDK the kernel links against)
[sdk] reusing /quadric/sdk_install (NUM_CORES not recorded; archsim target)
2. Quantization front end → graphs/
Hardware-independent, so one run serves every target. run_all.sh Stages 1–2:
- Export: the model card's safetensors → one fp32 ONNX decoder with KV-cache inputs/outputs.
- SmoothQuant: per-channel smoothing of every projection, with all of them (q/k/v included) held out of int8 so they stay fp for the int4 step. The by-product that matters most is
optimized_opt_sym_int8_q.onnx.tranges, the calibrated range of every edge — every FixedPoint32 fractional-bit choice downstream derives from it. - int4 weights: every projection and the lm_head become
MatMulNBits(4-bit, block 128, asymmetric) on the dynamic-shape graph, then a copy is pinned to sequence length 512 for CGC. - Static Q/DQ on post-RoPE Q and K, on both twins: the skeleton the attention custom op is matched on, and the reason the int8 KV cache is already in the reference graph. Result:
seq_int4_qkqdq.onnx(static, what CGC ingests) andseq_int4_qkqdq_dyn.onnx(what ONNX Runtime scores against), both reading their weights fromseq_int4_everywhere*.onnx.data.
With QUANTIZE = "fetch" (the default) quantize() reuses the graphs when they exist and downloads the published copy (iss/graphs_manifest.json, 5.1 GB, checksums verified) when they do not; the quantization stage itself never runs, and an unreachable download is an error rather than an hours-long fallback. QUANTIZE = "run" produces the graphs on this machine with run_all.sh Stages 1-2 (hours; needs the front-end venv from bash run_all.sh setup), replacing a fetched copy. Add "quantize" to FORCE to re-download or re-run regardless of what is on disk.
graph = sara.quantize(cfg, mode=QUANTIZE, force="quantize" in FORCE)
print("CGC ingests:", graph.relative_to(cfg.workdir))
print("ORT golden graph:", (cfg.graphs_dir / "seq_int4_qkqdq_dyn.onnx").relative_to(cfg.workdir))
[quantize] fetching the published graphs from https://sdk-cli-models.s3.amazonaws.com/sarashina2.2-3b/graphs/
seq_int4_qkqdq.onnx (1.1 MB, sha256 verified)
seq_int4_qkqdq_dyn.onnx (5.7 MB, sha256 verified)
seq_int4_everywhere.onnx (1.1 MB, sha256 verified)
seq_int4_everywhere.onnx.data (2,707.8 MB, sha256 verified)
seq_int4_everywhere_dyn.onnx (5.6 MB, sha256 verified)
seq_int4_everywhere_dyn.onnx.data (2,704.6 MB, sha256 verified)
optimized_opt_sym_int8_q.onnx.tranges (0.4 MB, sha256 verified)
[quantize] graphs/ was fetched from https://sdk-cli-models.s3.amazonaws.com/sarashina2.2-3b/graphs/
[quantize] reusing seq_int4_qkqdq.onnx (1.1 MB, 2026-09-22)
[quantize] and optimized_opt_sym_int8_q.onnx.tranges -- hardware-independent, shared by every target
CGC ingests: graphs/seq_int4_qkqdq.onnx
ORT golden graph: graphs/seq_int4_qkqdq_dyn.onnx
3. Custom ops → custom_op/<knobs>/
match_att.py walks the 32 decoders and rewrites the graph for the kernel library:
- attention →
qwenMultiheadAttentionFx32RoPE(RoPE from a sin/cos table, static int8 Q/K, online softmax, int8 K/V cache in persistent memory); - q|k|v and gate|up → one
w4a16PackedProjop per group (orw4a16FusedProjforPROJ_MODE="fused"); - o, down, lm_head →
w4a16NBitsMatMul<im_frac>.
Every output's fractional bits come from the calibrated range minus IMF_HEADROOM bits of margin. The knobs are recorded beside the graph in build_knobs.json and travel into the board handoff's manifest. Weights stay external: the new constants in sarashina.onnx.data, the untouched Stage-2 weights referenced from graphs/.
custom_op = sara.match_att(cfg, force="match_att" in FORCE)
print("custom-op graph:", custom_op.relative_to(cfg.workdir))
[match_att] building custom_op/packed_hr2 (no STAGE.json)
im_frac: using true fp output ranges for 129 projections (/quadric/sdk-cli/examples/models/sarashina2.2-3b/iss/fp_proj_imfrac.json)
inserted 32 v_proj int8 QuantizeLinear (int4 v_proj: V is quantized post-cache in this graph; the custom op needs it at the projection output)
attention: kvDim=1280 numGroups=2 reserved_ext=1310720 B/layer (40.0 MB persistent for 32 layers)
down_proj: int4 for 32 layers (asymmetric DoubleInt8 w4a16NBitsMatMul)
down_proj.0 -> nn_custom::w4a16NBitsMatMul<24, 31>
down_proj.1 -> nn_custom::w4a16NBitsMatMul<18, 29>
down_proj.2 -> nn_custom::w4a16NBitsMatMul<21, 31>
down_proj.3 -> nn_custom::w4a16NBitsMatMul<15, 28>
down_proj.4 -> nn_custom::w4a16NBitsMatMul<19, 30>
down_proj.5 -> nn_custom::w4a16NBitsMatMul<25, 31>
down_proj.6 -> nn_custom::w4a16NBitsMatMul<20, 30>
down_proj.7 -> nn_custom::w4a16NBitsMatMul<25, 31>
down_proj.8 -> nn_custom::w4a16NBitsMatMul<25, 31>
down_proj.9 -> nn_custom::w4a16NBitsMatMul<25, 31>
down_proj.10 -> nn_custom::w4a16NBitsMatMul<24, 31>
down_proj.11 -> nn_custom::w4a16NBitsMatMul<21, 31>
down_proj.12 -> nn_custom::w4a16NBitsMatMul<25, 31>
down_proj.13 -> nn_custom::w4a16NBitsMatMul<25, 31>
down_proj.14 -> nn_custom::w4a16NBitsMatMul<25, 31>
down_proj.15 -> nn_custom::w4a16NBitsMatMul<24, 31>
down_proj.16 -> nn_custom::w4a16NBitsMatMul<24, 31>
down_proj.17 -> nn_custom::w4a16NBitsMatMul<25, 31>
down_proj.18 -> nn_custom::w4a16NBitsMatMul<25, 31>
down_proj.19 -> nn_custom::w4a16NBitsMatMul<25, 31>
down_proj.20 -> nn_custom::w4a16NBitsMatMul<25, 31>
down_proj.21 -> nn_custom::w4a16NBitsMatMul<25, 31>
down_proj.22 -> nn_custom::w4a16NBitsMatMul<25, 31>
down_proj.23 -> nn_custom::w4a16NBitsMatMul<24, 31>
down_proj.24 -> nn_custom::w4a16NBitsMatMul<25, 31>
down_proj.25 -> nn_custom::w4a16NBitsMatMul<24, 31>
down_proj.26 -> nn_custom::w4a16NBitsMatMul<24, 31>
down_proj.27 -> nn_custom::w4a16NBitsMatMul<24, 31>
down_proj.28 -> nn_custom::w4a16NBitsMatMul<23, 31>
down_proj.29 -> nn_custom::w4a16NBitsMatMul<23, 31>
down_proj.30 -> nn_custom::w4a16NBitsMatMul<22, 31>
down_proj.31 -> nn_custom::w4a16NBitsMatMul<17, 30>
not packable: qkv has a different smoothing vector per projection; needs a shared vector from sarashina_quant.py
not packable: gate|up has a different smoothing vector per projection; needs a shared vector from sarashina_quant.py
PACK_FUSED_PROJ: emitted 64 packed group ops (smooth muls + matmuls absorbed; layernorm now single-consumer)
o_proj.0 -> nn_custom::w4a16NBitsMatMul<25, 31>
o_proj.1 -> nn_custom::w4a16NBitsMatMul<25, 31>
o_proj.2 -> nn_custom::w4a16NBitsMatMul<25, 31>
o_proj.3 -> nn_custom::w4a16NBitsMatMul<26, 31>
o_proj.4 -> nn_custom::w4a16NBitsMatMul<26, 31>
o_proj.5 -> nn_custom::w4a16NBitsMatMul<26, 31>
o_proj.6 -> nn_custom::w4a16NBitsMatMul<26, 31>
o_proj.7 -> nn_custom::w4a16NBitsMatMul<26, 31>
o_proj.8 -> nn_custom::w4a16NBitsMatMul<26, 31>
o_proj.9 -> nn_custom::w4a16NBitsMatMul<25, 31>
o_proj.10 -> nn_custom::w4a16NBitsMatMul<25, 31>
o_proj.11 -> nn_custom::w4a16NBitsMatMul<25, 31>
o_proj.12 -> nn_custom::w4a16NBitsMatMul<26, 31>
o_proj.13 -> nn_custom::w4a16NBitsMatMul<26, 31>
o_proj.14 -> nn_custom::w4a16NBitsMatMul<25, 31>
o_proj.15 -> nn_custom::w4a16NBitsMatMul<24, 31>
o_proj.16 -> nn_custom::w4a16NBitsMatMul<25, 31>
o_proj.17 -> nn_custom::w4a16NBitsMatMul<25, 31>
o_proj.18 -> nn_custom::w4a16NBitsMatMul<25, 31>
o_proj.19 -> nn_custom::w4a16NBitsMatMul<25, 31>
o_proj.20 -> nn_custom::w4a16NBitsMatMul<26, 31>
o_proj.21 -> nn_custom::w4a16NBitsMatMul<26, 31>
o_proj.22 -> nn_custom::w4a16NBitsMatMul<25, 31>
o_proj.23 -> nn_custom::w4a16NBitsMatMul<25, 31>
o_proj.24 -> nn_custom::w4a16NBitsMatMul<25, 31>
o_proj.25 -> nn_custom::w4a16NBitsMatMul<25, 31>
o_proj.26 -> nn_custom::w4a16NBitsMatMul<25, 31>
o_proj.27 -> nn_custom::w4a16NBitsMatMul<24, 31>
o_proj.28 -> nn_custom::w4a16NBitsMatMul<24, 31>
o_proj.29 -> nn_custom::w4a16NBitsMatMul<23, 31>
o_proj.30 -> nn_custom::w4a16NBitsMatMul<22, 31>
o_proj.31 -> nn_custom::w4a16NBitsMatMul<18, 31>
projections (single smoothing source): replaced 32 int4 w4a16NBitsMatMul from the graph
lm_head[logits] -> nn_custom::w4a16NBitsMatMul<23, 31>
lm_head: int4 w4a16NBitsMatMul x1 (split=False)
custom-op graph: custom_op/packed_hr2/sarashina.onnx
4. Chimera Graph Compiler → cgc/<knobs>/<hw>/
ChimeraJob(model, hw_config).compile() lowers the graph to one C++ entry function whose signature is the kernel's contract (input ids, sin/cos, the constant blob, external temps, persistent KV, logits out, sequence out) plus the 2.7 GB constant blob. CGC picks the remaining FixedPoint32 formats from its own range analysis — including the logits', which is why the driver reads that value from the generated source instead of assuming it. About 35 minutes; reused while the graph, ranges, hardware config and TVM checkout are unchanged.
job = sara.compile_cgc(cfg, custom_op, force="cgc" in FORCE)
print(job)
[cgc] compiling into cgc/packed_hr2/QC-U_1d5_8MB_4kB_128GBps_128GBps_8_OFF_x1_x1 (no STAGE.json)
2026-09-22 02:57 - INFO - epu - chimera_job - START==================================onnx_ingest
2026-09-22 02:57 - INFO - epu - chimera_job - Numerical ranges provided
/usr/local/lib/python3.10/dist-packages/tvm/relay/frontend/onnx.py:6272: UserWarning: This protobuf of onnx model is too large (>2GB). Call check_model with model path instead.
warnings.warn(str(e))
2026-09-22 02:58 - INFO - epu - codegen - START===============================optimize_relay
2026-09-22 02:58 - INFO - epu - codegen - START====================quantize_to_cpu_runnable_fx
2026-09-22 02:58 - INFO - epu - fx - Clamped annotated range on /model/layers.4/self_attn/MatMul_1_output_0_DequantizeLinear to static range: (-4.747185707092285, 4.982583522796631) -> (-4.747185707092285, 4.982583314180374) (within tol 0.0392329)
2026-09-22 02:58 - INFO - epu - fx - Clamped annotated range on /model/layers.4/self_attn/Reshape_9 to static range: (-4.747185707092285, 4.982583522796631) -> (-4.747185707092285, 4.982583314180374) (within tol 0.038156)
2026-09-22 02:58 - INFO - epu - fx - Clamped annotated range on /model/layers.4/Add_1 to static range: (-135.37454223632812, 11347.30078125) -> (-135.3745346069336, 11347.30078125) (within tol 45.0301)
2026-09-22 02:58 - INFO - epu - fx - Clamped annotated range on /model/layers.4/Add_1 to static range: (-135.37454223632812, 11347.30078125) -> (-135.3745346069336, 11347.30078125) (within tol 45.0301)
2026-09-22 02:58 - INFO - epu - fx - Clamped annotated range on /model/layers.6/Add to static range: (-146.45631408691406, 11345.2880859375) -> (-146.45630931854248, 11345.2880859375) (within tol 45.0909)
2026-09-22 02:58 - INFO - epu - fx - Clamped annotated range on /model/layers.6/Add to static range: (-146.45631408691406, 11345.2880859375) -> (-146.45630931854248, 11345.2880859375) (within tol 45.0909)
2026-09-22 02:58 - INFO - epu - fx - Clamped annotated range on /model/layers.7/self_attn/MatMul_1_output_0_DequantizeLinear to static range: (-3.7836551666259766, 4.57642126083374) -> (-3.7836551666259766, 4.576421033591032) (within tol 0.0360348)
2026-09-22 02:58 - INFO - epu - fx - Clamped annotated range on /model/layers.7/self_attn/Reshape_9 to static range: (-3.7836551666259766, 4.57642126083374) -> (-3.7836551666259766, 4.576421033591032) (within tol 0.0327846)
2026-09-22 02:58 - INFO - epu - fx - Clamped annotated range on /model/layers.9/self_attn/MatMul_1_output_0_DequantizeLinear to static range: (-5.8798699378967285, 6.222862243652344) -> (-5.8798699378967285, 6.222862161695957) (within tol 0.0489989)
2026-09-22 02:58 - INFO - epu - fx - Clamped annotated range on /model/layers.9/self_attn/Reshape_9 to static range: (-5.8798699378967285, 6.222862243652344) -> (-5.8798699378967285, 6.222862161695957) (within tol 0.0474617)
2026-09-22 02:58 - INFO - epu - fx - Clamped annotated range on /model/layers.9/Add to static range: (-162.0902862548828, 11705.775390625) -> (-162.0902795791626, 11705.775390625) (within tol 46.5741)
2026-09-22 02:58 - INFO - epu - fx - Clamped annotated range on /model/layers.9/Add to static range: (-162.0902862548828, 11705.775390625) -> (-162.0902795791626, 11705.775390625) (within tol 46.5741)
2026-09-22 02:58 - INFO - epu - fx - Clamped annotated range on /model/layers.10/Add_1 to static range: (-166.00888061523438, 11718.998046875) -> (-166.00888061523438, 11718.997871398926) (within tol 46.6466)
2026-09-22 02:58 - INFO - epu - fx - Clamped annotated range on /model/layers.10/Add_1 to static range: (-166.00888061523438, 11718.998046875) -> (-166.00888061523438, 11718.997871398926) (within tol 46.6466)
2026-09-22 02:58 - INFO - epu - fx - Clamped annotated range on /model/layers.11/Add_1 to static range: (-178.7839813232422, 11916.3720703125) -> (-178.7839813232422, 11916.371795654297) (within tol 47.6006)
2026-09-22 02:58 - INFO - epu - fx - Clamped annotated range on /model/layers.11/Add_1 to static range: (-178.7839813232422, 11916.3720703125) -> (-178.7839813232422, 11916.371795654297) (within tol 47.6006)
2026-09-22 02:58 - INFO - epu - fx - Clamped annotated range on /model/layers.24/self_attn/MatMul_1_output_0_DequantizeLinear to static range: (-6.803412914276123, 6.912267684936523) -> (-6.803412914276123, 6.912267528474331) (within tol 0.0544273)
2026-09-22 02:58 - INFO - epu - fx - Clamped annotated range on /model/layers.24/self_attn/Reshape_9 to static range: (-6.803412914276123, 6.912267684936523) -> (-6.803412914276123, 6.912267528474331) (within tol 0.053787)
2026-09-22 02:58 - INFO - epu - fx - Clamped annotated range on /model/layers.30/self_attn/MatMul_1_output_0_DequantizeLinear to static range: (-9.23527717590332, 13.328184127807617) -> (-9.23527717590332, 13.328183934092522) (within tol 0.104946)
2026-09-22 02:58 - INFO - epu - fx - Clamped annotated range on /model/layers.30/self_attn/Reshape_9 to static range: (-9.23527717590332, 13.328184127807617) -> (-9.23527717590332, 13.328183934092522) (within tol 0.0884842)
2026-09-22 02:58 - INFO - epu - fx -
Source name Op Output 0 Range Output 0 Frac Bits Output 1 Range Output 1 Frac Bits
------------------------------------------------------------- ----------------------------- ----------------------------- -------------------- ---------------- --------------------
/model/embed_tokens/Gather contrib.epu.embedding [-8.1875f, 6.34375f] 27 N/A N/A
/model/layers.0/input_layernorm/Mul_1 contrib.epu.rms_norm [-18.9287f, 28.7483f] 26 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<3, 21, 31, 31>2000 contrib.epu.quadric_custom_op (-1024.0, 1023.9999995231628) 21 N/A 21
/model/layers.0/self_attn/q_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.0/self_attn/k_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.0/self_attn/MatMul_1_output_0_DequantizeLinear contrib.epu.dequantize [-10.6596f, 11.3762f] 27 N/A N/A
/model/layers.0/self_attn/Reshape_9 reshape [-10.6596f, 11.3762f] 27 N/A N/A
/model/layers.0/self_attn/o_proj/MatMul_smooth_mul multiply [-1.70659f, 1.93791f] 28 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<25, 31>2064 contrib.epu.quadric_custom_op [-11.0497f, 9.87976f] 25 N/A N/A
/model/layers.0/Add add [-10.8817f, 11.1882f] 25 N/A N/A
/model/layers.0/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-19.4549f, 12.6022f] 26 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<2, 24, 31, 31>2032 contrib.epu.quadric_custom_op (-128.0, 127.99999994039536) 24 N/A N/A
/model/layers.0/mlp/gate_proj/MatMul_output_0_slice strided_slice (-128.0, 127.99999994039536) 24 N/A N/A
/model/layers.0/mlp/act_fn/Sigmoid sigmoid [1.40071e-05f, 0.999586f] 31 N/A N/A
/model/layers.0/mlp/act_fn/Mul multiply [-0.278465f, 7.78503f] 24 N/A N/A
/model/layers.0/mlp/up_proj/MatMul_output_0_slice strided_slice (-128.0, 127.99999994039536) 24 N/A N/A
/model/layers.0/mlp/Mul multiply [-40.6006f, 27.927f] 21 N/A N/A
/model/layers.0/mlp/down_proj/MatMul_smooth_mul multiply [-4.60692f, 2.82909f] 25 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<24, 31>32 contrib.epu.quadric_custom_op [-22.287f, 22.9399f] 24 N/A N/A
/model/layers.0/Add_1 add [-28.9546f, 28.4438f] 24 N/A N/A
/model/layers.1/input_layernorm/Mul_1 contrib.epu.rms_norm [-26.6093f, 23.89f] 26 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<3, 21, 31, 31>2001 contrib.epu.quadric_custom_op (-1024.0, 1023.9999995231628) 21 N/A 21
/model/layers.1/self_attn/q_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.1/self_attn/k_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.1/self_attn/MatMul_1_output_0_DequantizeLinear contrib.epu.dequantize [-5.54615f, 5.24285f] 28 N/A N/A
/model/layers.1/self_attn/Reshape_9 reshape [-5.54615f, 5.24285f] 28 N/A N/A
/model/layers.1/self_attn/o_proj/MatMul_smooth_mul multiply [-1.0892f, 1.12525f] 29 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<25, 31>2065 contrib.epu.quadric_custom_op [-7.09223f, 8.08643f] 25 N/A N/A
/model/layers.1/Add add [-24.8562f, 24.5854f] 24 N/A N/A
/model/layers.1/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-34.4119f, 30.277f] 25 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<2, 24, 31, 31>2033 contrib.epu.quadric_custom_op (-128.0, 127.99999994039536) 24 N/A N/A
/model/layers.1/mlp/gate_proj/MatMul_output_0_slice strided_slice (-128.0, 127.99999994039536) 24 N/A N/A
/model/layers.1/mlp/act_fn/Sigmoid sigmoid [4.47035e-07f, 1f] 31 N/A N/A
/model/layers.1/mlp/act_fn/Mul multiply [-0.278465f, 25.7029f] 24 N/A N/A
/model/layers.1/mlp/up_proj/MatMul_output_0_slice strided_slice (-128.0, 127.99999994039536) 24 N/A N/A
/model/layers.1/mlp/Mul multiply [-37.9871f, 558.652f] 19 N/A N/A
/model/layers.1/mlp/down_proj/MatMul_smooth_mul multiply [-3.2304f, 41.3513f] 22 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<18, 29>33 contrib.epu.quadric_custom_op [-78.9917f, 1718.65f] 18 N/A N/A
/model/layers.1/Add_1 add [-102.197f, 1738.34f] 18 N/A N/A
/model/layers.2/input_layernorm/Mul_1 contrib.epu.rms_norm [-36.7319f, 42.7586f] 25 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<3, 21, 31, 31>2002 contrib.epu.quadric_custom_op (-1024.0, 1023.9999995231628) 21 N/A 21
/model/layers.2/self_attn/q_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.2/self_attn/k_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.2/self_attn/MatMul_1_output_0_DequantizeLinear contrib.epu.dequantize [-6.016f, 5.25808f] 28 N/A N/A
/model/layers.2/self_attn/Reshape_9 reshape [-6.016f, 5.25808f] 28 N/A N/A
/model/layers.2/self_attn/o_proj/MatMul_smooth_mul multiply [-1.23116f, 0.949909f] 28 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<25, 31>2066 contrib.epu.quadric_custom_op [-7.75351f, 8.7895f] 25 N/A N/A
/model/layers.2/Add add [-101.797f, 1737.94f] 18 N/A N/A
/model/layers.2/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-18.4131f, 45.7424f] 25 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<2, 23, 31, 31>2034 contrib.epu.quadric_custom_op (-256.0, 255.9999998807907) 23 N/A N/A
/model/layers.2/mlp/gate_proj/MatMul_output_0_slice strided_slice (-256.0, 255.9999998807907) 23 N/A N/A
/model/layers.2/mlp/act_fn/Sigmoid sigmoid [2.98023e-08f, 0.999669f] 31 N/A N/A
/model/layers.2/mlp/act_fn/Mul multiply [-0.278465f, 8.01146f] 23 N/A N/A
/model/layers.2/mlp/up_proj/MatMul_output_0_slice strided_slice (-256.0, 255.9999998807907) 23 N/A N/A
/model/layers.2/mlp/Mul multiply [-252.035f, 55.1893f] 19 N/A N/A
/model/layers.2/mlp/down_proj/MatMul_smooth_mul multiply [-13.5495f, 3.82104f] 23 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<21, 31>34 contrib.epu.quadric_custom_op [-83.4408f, 221.53f] 21 N/A N/A
/model/layers.2/Add_1 add [-104.126f, 1837.56f] 18 N/A N/A
/model/layers.3/input_layernorm/Mul_1 contrib.epu.rms_norm [-40.6396f, 50.3933f] 25 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<3, 21, 31, 31>2003 contrib.epu.quadric_custom_op (-1024.0, 1023.9999995231628) 21 N/A 21
/model/layers.3/self_attn/q_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.3/self_attn/k_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.3/self_attn/MatMul_1_output_0_DequantizeLinear contrib.epu.dequantize [-4.9364f, 4.502f] 28 N/A N/A
/model/layers.3/self_attn/Reshape_9 reshape [-4.9364f, 4.502f] 28 N/A N/A
/model/layers.3/self_attn/o_proj/MatMul_smooth_mul multiply [-0.887145f, 0.86594f] 29 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<26, 31>2067 contrib.epu.quadric_custom_op [-4.44471f, 5.00306f] 26 N/A N/A
/model/layers.3/Add add [-103.936f, 1837.18f] 18 N/A N/A
/model/layers.3/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-17.0245f, 95.751f] 24 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<2, 22, 31, 31>2035 contrib.epu.quadric_custom_op (-512.0, 511.9999997615814) 22 N/A N/A
/model/layers.3/mlp/gate_proj/MatMul_output_0_slice strided_slice (-512.0, 511.9999997615814) 22 N/A N/A
/model/layers.3/mlp/act_fn/Sigmoid sigmoid [0f, 1f] 31 N/A N/A
/model/layers.3/mlp/act_fn/Mul multiply [-0.278465f, 49.1432f] 22 N/A N/A
/model/layers.3/mlp/up_proj/MatMul_output_0_slice strided_slice (-512.0, 511.9999997615814) 22 N/A N/A
/model/layers.3/mlp/Mul multiply [-3755.83f, 36.6731f] 16 N/A N/A
/model/layers.3/mlp/down_proj/MatMul_smooth_mul multiply [-102.729f, 4.16084f] 19 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<15, 28>35 contrib.epu.quadric_custom_op [-54.1731f, 10572.1f] 15 N/A N/A
/model/layers.3/Add_1 add [-126.07f, 10805.1f] 15 N/A N/A
/model/layers.4/input_layernorm/Mul_1 contrib.epu.rms_norm [-52.779f, 65.563f] 24 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<3, 21, 31, 31>2004 contrib.epu.quadric_custom_op (-1024.0, 1023.9999995231628) 21 N/A 21
/model/layers.4/self_attn/q_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.4/self_attn/k_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.4/self_attn/MatMul_1_output_0_DequantizeLinear contrib.epu.dequantize [-4.74719f, 4.98258f] 28 N/A N/A
/model/layers.4/self_attn/Reshape_9 reshape [-4.74719f, 4.98258f] 28 N/A N/A
/model/layers.4/self_attn/o_proj/MatMul_smooth_mul multiply [-1.30319f, 1.35598f] 29 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<26, 31>2068 contrib.epu.quadric_custom_op [-5.45043f, 4.98739f] 26 N/A N/A
/model/layers.4/Add add [-126.505f, 10805f] 15 N/A N/A
/model/layers.4/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-16.6444f, 116.046f] 24 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<2, 24, 31, 31>2036 contrib.epu.quadric_custom_op (-128.0, 127.99999994039536) 24 N/A N/A
/model/layers.4/mlp/gate_proj/MatMul_output_0_slice strided_slice (-128.0, 127.99999994039536) 24 N/A N/A
/model/layers.4/mlp/act_fn/Sigmoid sigmoid [0f, 1f] 31 N/A N/A
/model/layers.4/mlp/act_fn/Mul multiply [-0.278465f, 18.2721f] 24 N/A N/A
/model/layers.4/mlp/up_proj/MatMul_output_0_slice strided_slice (-128.0, 127.99999994039536) 24 N/A N/A
/model/layers.4/mlp/Mul multiply [-172.255f, 29.2797f] 19 N/A N/A
/model/layers.4/mlp/down_proj/MatMul_smooth_mul multiply [-23.3508f, 7.55311f] 24 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<19, 30>36 contrib.epu.quadric_custom_op [-8.8693f, 542.29f] 19 N/A N/A
/model/layers.4/Add_1 add [-135.375f, 11347.3f] 15 N/A N/A
/model/layers.5/input_layernorm/Mul_1 contrib.epu.rms_norm [-52.4452f, 70.1606f] 24 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<3, 21, 31, 31>2005 contrib.epu.quadric_custom_op (-1024.0, 1023.9999995231628) 21 N/A 21
/model/layers.5/self_attn/q_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.5/self_attn/k_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.5/self_attn/MatMul_1_output_0_DequantizeLinear contrib.epu.dequantize [-5.89011f, 4.68426f] 28 N/A N/A
/model/layers.5/self_attn/Reshape_9 reshape [-5.89011f, 4.68426f] 28 N/A N/A
/model/layers.5/self_attn/o_proj/MatMul_smooth_mul multiply [-1.22272f, 1.1645f] 29 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<26, 31>2069 contrib.epu.quadric_custom_op [-6.39057f, 6.02962f] 26 N/A N/A
/model/layers.5/Add add [-136.316f, 11347.2f] 15 N/A N/A
/model/layers.5/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-15.5537f, 119.213f] 24 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<2, 25, 31, 31>2037 contrib.epu.quadric_custom_op (-64.0, 63.99999997019768) 25 N/A N/A
/model/layers.5/mlp/gate_proj/MatMul_output_0_slice strided_slice (-64.0, 63.99999997019768) 25 N/A N/A
/model/layers.5/mlp/act_fn/Sigmoid sigmoid [0.000208676f, 0.999989f] 31 N/A N/A
/model/layers.5/mlp/act_fn/Mul multiply [-0.278465f, 11.4588f] 25 N/A N/A
/model/layers.5/mlp/up_proj/MatMul_output_0_slice strided_slice (-64.0, 63.99999997019768) 25 N/A N/A
/model/layers.5/mlp/Mul multiply [-41.0238f, 26.5058f] 21 N/A N/A
/model/layers.5/mlp/down_proj/MatMul_smooth_mul multiply [-2.47005f, 2.32877f] 25 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<25, 31>37 contrib.epu.quadric_custom_op [-9.77686f, 8.74657f] 25 N/A N/A
/model/layers.5/Add_1 add [-140.247f, 11345.8f] 15 N/A N/A
/model/layers.6/input_layernorm/Mul_1 contrib.epu.rms_norm [-46.4727f, 66.2337f] 24 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<3, 21, 31, 31>2006 contrib.epu.quadric_custom_op (-1024.0, 1023.9999995231628) 21 N/A 21
/model/layers.6/self_attn/q_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.6/self_attn/k_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.6/self_attn/MatMul_1_output_0_DequantizeLinear contrib.epu.dequantize [-4.91116f, 4.99037f] 28 N/A N/A
/model/layers.6/self_attn/Reshape_9 reshape [-4.91116f, 4.99037f] 28 N/A N/A
/model/layers.6/self_attn/o_proj/MatMul_smooth_mul multiply [-1.28599f, 1.02958f] 29 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<26, 31>2070 contrib.epu.quadric_custom_op [-6.20925f, 5.95986f] 26 N/A N/A
/model/layers.6/Add add [-146.456f, 11345.3f] 15 N/A N/A
/model/layers.6/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-17.3043f, 123.159f] 24 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<2, 24, 31, 31>2038 contrib.epu.quadric_custom_op (-128.0, 127.99999994039536) 24 N/A N/A
/model/layers.6/mlp/gate_proj/MatMul_output_0_slice strided_slice (-128.0, 127.99999994039536) 24 N/A N/A
/model/layers.6/mlp/act_fn/Sigmoid sigmoid [1.19209e-07f, 0.999995f] 31 N/A N/A
/model/layers.6/mlp/act_fn/Mul multiply [-0.278465f, 12.1884f] 24 N/A N/A
/model/layers.6/mlp/up_proj/MatMul_output_0_slice strided_slice (-128.0, 127.99999994039536) 24 N/A N/A
/model/layers.6/mlp/Mul multiply [-31.2444f, 171.596f] 20 N/A N/A
/model/layers.6/mlp/down_proj/MatMul_smooth_mul multiply [-2.62008f, 19.2874f] 23 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<20, 30>38 contrib.epu.quadric_custom_op [-7.42428f, 368.51f] 20 N/A N/A
/model/layers.6/Add_1 add [-149.099f, 11713.8f] 15 N/A N/A
/model/layers.7/input_layernorm/Mul_1 contrib.epu.rms_norm [-48.8074f, 70.4718f] 24 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<3, 21, 31, 31>2007 contrib.epu.quadric_custom_op (-1024.0, 1023.9999995231628) 21 N/A 21
/model/layers.7/self_attn/q_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.7/self_attn/k_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.7/self_attn/MatMul_1_output_0_DequantizeLinear contrib.epu.dequantize [-3.78366f, 4.57642f] 28 N/A N/A
/model/layers.7/self_attn/Reshape_9 reshape [-3.78366f, 4.57642f] 28 N/A N/A
/model/layers.7/self_attn/o_proj/MatMul_smooth_mul multiply [-1.32305f, 1.10551f] 28 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<26, 31>2071 contrib.epu.quadric_custom_op [-3.93443f, 3.82065f] 26 N/A N/A
/model/layers.7/Add add [-150.886f, 11713.7f] 15 N/A N/A
/model/layers.7/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-20.9677f, 114.487f] 24 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<2, 25, 31, 31>2039 contrib.epu.quadric_custom_op (-64.0, 63.99999997019768) 25 N/A N/A
/model/layers.7/mlp/gate_proj/MatMul_output_0_slice strided_slice (-64.0, 63.99999997019768) 25 N/A N/A
/model/layers.7/mlp/act_fn/Sigmoid sigmoid [1.76132e-05f, 0.999975f] 31 N/A N/A
/model/layers.7/mlp/act_fn/Mul multiply [-0.278465f, 10.579f] 25 N/A N/A
/model/layers.7/mlp/up_proj/MatMul_output_0_slice strided_slice (-64.0, 63.99999997019768) 25 N/A N/A
/model/layers.7/mlp/Mul multiply [-29.2401f, 37.9283f] 21 N/A N/A
/model/layers.7/mlp/down_proj/MatMul_smooth_mul multiply [-2.61986f, 2.84772f] 26 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<25, 31>39 contrib.epu.quadric_custom_op [-11.0535f, 11.7687f] 25 N/A N/A
/model/layers.7/Add_1 add [-151.285f, 11709.8f] 15 N/A N/A
/model/layers.8/input_layernorm/Mul_1 contrib.epu.rms_norm [-43.2311f, 68.474f] 24 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<3, 21, 31, 31>2008 contrib.epu.quadric_custom_op (-1024.0, 1023.9999995231628) 21 N/A 21
/model/layers.8/self_attn/q_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.8/self_attn/k_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.8/self_attn/MatMul_1_output_0_DequantizeLinear contrib.epu.dequantize [-5.85194f, 4.82785f] 28 N/A N/A
/model/layers.8/self_attn/Reshape_9 reshape [-5.85194f, 4.82785f] 28 N/A N/A
/model/layers.8/self_attn/o_proj/MatMul_smooth_mul multiply [-1.60141f, 1.24081f] 29 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<26, 31>2072 contrib.epu.quadric_custom_op [-7.77506f, 7.98162f] 26 N/A N/A
/model/layers.8/Add add [-152.928f, 11709.8f] 15 N/A N/A
/model/layers.8/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-22.2379f, 102.644f] 24 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<2, 24, 31, 31>2040 contrib.epu.quadric_custom_op (-128.0, 127.99999994039536) 24 N/A N/A
/model/layers.8/mlp/gate_proj/MatMul_output_0_slice strided_slice (-128.0, 127.99999994039536) 24 N/A N/A
/model/layers.8/mlp/act_fn/Sigmoid sigmoid [9.28044e-05f, 1f] 31 N/A N/A
/model/layers.8/mlp/act_fn/Mul multiply [-0.278465f, 15.313f] 24 N/A N/A
/model/layers.8/mlp/up_proj/MatMul_output_0_slice strided_slice (-128.0, 127.99999994039536) 24 N/A N/A
/model/layers.8/mlp/Mul multiply [-30.9985f, 31.5176f] 20 N/A N/A
/model/layers.8/mlp/down_proj/MatMul_smooth_mul multiply [-5.46518f, 3.74797f] 26 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<25, 31>40 contrib.epu.quadric_custom_op [-12.8567f, 13.1663f] 25 N/A N/A
/model/layers.8/Add_1 add [-154.226f, 11705.8f] 15 N/A N/A
/model/layers.9/input_layernorm/Mul_1 contrib.epu.rms_norm [-46.2538f, 69.3696f] 24 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<3, 21, 31, 31>2009 contrib.epu.quadric_custom_op (-1024.0, 1023.9999995231628) 21 N/A 21
/model/layers.9/self_attn/q_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.9/self_attn/k_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.9/self_attn/MatMul_1_output_0_DequantizeLinear contrib.epu.dequantize [-5.87987f, 6.22286f] 28 N/A N/A
/model/layers.9/self_attn/Reshape_9 reshape [-5.87987f, 6.22286f] 28 N/A N/A
/model/layers.9/self_attn/o_proj/MatMul_smooth_mul multiply [-1.17674f, 1.14783f] 28 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<25, 31>2073 contrib.epu.quadric_custom_op [-7.86445f, 8.5267f] 25 N/A N/A
/model/layers.9/Add add [-162.09f, 11705.8f] 15 N/A N/A
/model/layers.9/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-19.0429f, 99.4843f] 24 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<2, 23, 31, 31>2041 contrib.epu.quadric_custom_op (-256.0, 255.9999998807907) 23 N/A N/A
/model/layers.9/mlp/gate_proj/MatMul_output_0_slice strided_slice (-256.0, 255.9999998807907) 23 N/A N/A
/model/layers.9/mlp/act_fn/Sigmoid sigmoid [6.61612e-06f, 0.999975f] 31 N/A N/A
/model/layers.9/mlp/act_fn/Mul multiply [-0.278465f, 10.5781f] 23 N/A N/A
/model/layers.9/mlp/up_proj/MatMul_output_0_slice strided_slice (-256.0, 255.9999998807907) 23 N/A N/A
/model/layers.9/mlp/Mul multiply [-33.437f, 34.3111f] 19 N/A N/A
/model/layers.9/mlp/down_proj/MatMul_smooth_mul multiply [-4.69104f, 3.36842f] 26 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<25, 31>41 contrib.epu.quadric_custom_op [-9.07723f, 9.37795f] 25 N/A N/A
/model/layers.9/Add_1 add [-162.908f, 11702.3f] 15 N/A N/A
/model/layers.10/input_layernorm/Mul_1 contrib.epu.rms_norm [-46.7832f, 76.2424f] 24 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<3, 21, 31, 31>2010 contrib.epu.quadric_custom_op (-1024.0, 1023.9999995231628) 21 N/A 21
/model/layers.10/self_attn/q_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.10/self_attn/k_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.10/self_attn/MatMul_1_output_0_DequantizeLinear contrib.epu.dequantize [-4.888f, 4.77162f] 28 N/A N/A
/model/layers.10/self_attn/Reshape_9 reshape [-4.888f, 4.77162f] 28 N/A N/A
/model/layers.10/self_attn/o_proj/MatMul_smooth_mul multiply [-1.20264f, 1.14082f] 29 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<25, 31>2074 contrib.epu.quadric_custom_op [-9.42305f, 8.37773f] 25 N/A N/A
/model/layers.10/Add add [-166.135f, 11702.2f] 15 N/A N/A
/model/layers.10/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-19.4933f, 93.5621f] 24 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<2, 24, 31, 31>2042 contrib.epu.quadric_custom_op (-128.0, 127.99999994039536) 24 N/A N/A
/model/layers.10/mlp/gate_proj/MatMul_output_0_slice strided_slice (-128.0, 127.99999994039536) 24 N/A N/A
/model/layers.10/mlp/act_fn/Sigmoid sigmoid [5.58794e-05f, 0.999923f] 31 N/A N/A
/model/layers.10/mlp/act_fn/Mul multiply [-0.278465f, 9.47006f] 24 N/A N/A
/model/layers.10/mlp/up_proj/MatMul_output_0_slice strided_slice (-128.0, 127.99999994039536) 24 N/A N/A
/model/layers.10/mlp/Mul multiply [-36.2754f, 44.1129f] 20 N/A N/A
/model/layers.10/mlp/down_proj/MatMul_smooth_mul multiply [-4.39494f, 4.11812f] 26 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<24, 31>42 contrib.epu.quadric_custom_op [-9.74571f, 16.8416f] 24 N/A N/A
/model/layers.10/Add_1 add [-166.009f, 11719f] 15 N/A N/A
/model/layers.11/input_layernorm/Mul_1 contrib.epu.rms_norm [-56.941f, 79.1904f] 24 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<3, 21, 31, 31>2011 contrib.epu.quadric_custom_op (-1024.0, 1023.9999995231628) 21 N/A 21
/model/layers.11/self_attn/q_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.11/self_attn/k_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.11/self_attn/MatMul_1_output_0_DequantizeLinear contrib.epu.dequantize [-5.77098f, 5.5285f] 28 N/A N/A
/model/layers.11/self_attn/Reshape_9 reshape [-5.77098f, 5.5285f] 28 N/A N/A
/model/layers.11/self_attn/o_proj/MatMul_smooth_mul multiply [-1.0321f, 1.1338f] 29 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<25, 31>2075 contrib.epu.quadric_custom_op [-9.77645f, 8.36185f] 25 N/A N/A
/model/layers.11/Add add [-175.112f, 11718.9f] 15 N/A N/A
/model/layers.11/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-18.6846f, 88.4316f] 24 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<2, 24, 31, 31>2043 contrib.epu.quadric_custom_op (-128.0, 127.99999994039536) 24 N/A N/A
/model/layers.11/mlp/gate_proj/MatMul_output_0_slice strided_slice (-128.0, 127.99999994039536) 24 N/A N/A
/model/layers.11/mlp/act_fn/Sigmoid sigmoid [0f, 1f] 31 N/A N/A
/model/layers.11/mlp/act_fn/Mul multiply [-0.278465f, 24.625f] 24 N/A N/A
/model/layers.11/mlp/up_proj/MatMul_output_0_slice strided_slice (-128.0, 127.99999994039536) 24 N/A N/A
/model/layers.11/mlp/Mul multiply [-32.2227f, 137.698f] 19 N/A N/A
/model/layers.11/mlp/down_proj/MatMul_smooth_mul multiply [-3.32926f, 14.1737f] 24 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<21, 31>43 contrib.epu.quadric_custom_op [-46.6751f, 197.446f] 21 N/A N/A
/model/layers.11/Add_1 add [-178.784f, 11916.4f] 15 N/A N/A
/model/layers.12/input_layernorm/Mul_1 contrib.epu.rms_norm [-50.7619f, 75.4228f] 24 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<3, 21, 31, 31>2012 contrib.epu.quadric_custom_op (-1024.0, 1023.9999995231628) 21 N/A 21
/model/layers.12/self_attn/q_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.12/self_attn/k_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.12/self_attn/MatMul_1_output_0_DequantizeLinear contrib.epu.dequantize [-5.34228f, 5.08788f] 28 N/A N/A
/model/layers.12/self_attn/Reshape_9 reshape [-5.34228f, 5.08788f] 28 N/A N/A
/model/layers.12/self_attn/o_proj/MatMul_smooth_mul multiply [-0.894391f, 1.17356f] 29 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<26, 31>2076 contrib.epu.quadric_custom_op [-7.04131f, 6.32305f] 26 N/A N/A
/model/layers.12/Add add [-179.883f, 11916.1f] 15 N/A N/A
/model/layers.12/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-19.1241f, 95.923f] 24 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<2, 24, 31, 31>2044 contrib.epu.quadric_custom_op (-128.0, 127.99999994039536) 24 N/A N/A
/model/layers.12/mlp/gate_proj/MatMul_output_0_slice strided_slice (-128.0, 127.99999994039536) 24 N/A N/A
/model/layers.12/mlp/act_fn/Sigmoid sigmoid [0.000120848f, 1f] 31 N/A N/A
/model/layers.12/mlp/act_fn/Mul multiply [-0.278465f, 28.8895f] 24 N/A N/A
/model/layers.12/mlp/up_proj/MatMul_output_0_slice strided_slice (-128.0, 127.99999994039536) 24 N/A N/A
/model/layers.12/mlp/Mul multiply [-25.428f, 28.0639f] 19 N/A N/A
/model/layers.12/mlp/down_proj/MatMul_smooth_mul multiply [-2.05318f, 1.96882f] 26 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<25, 31>44 contrib.epu.quadric_custom_op [-12.9149f, 13.2328f] 25 N/A N/A
/model/layers.12/Add_1 add [-177.519f, 11913.3f] 15 N/A N/A
/model/layers.13/input_layernorm/Mul_1 contrib.epu.rms_norm [-43.7769f, 83.2984f] 24 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<3, 21, 31, 31>2013 contrib.epu.quadric_custom_op (-1024.0, 1023.9999995231628) 21 N/A 21
/model/layers.13/self_attn/q_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.13/self_attn/k_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.13/self_attn/MatMul_1_output_0_DequantizeLinear contrib.epu.dequantize [-6.52061f, 6.7923f] 28 N/A N/A
/model/layers.13/self_attn/Reshape_9 reshape [-6.52061f, 6.7923f] 28 N/A N/A
/model/layers.13/self_attn/o_proj/MatMul_smooth_mul multiply [-1.58962f, 1.15127f] 28 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<26, 31>2077 contrib.epu.quadric_custom_op [-7.42975f, 7.80089f] 26 N/A N/A
/model/layers.13/Add add [-180.026f, 11912.9f] 15 N/A N/A
/model/layers.13/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-17.7134f, 91.5865f] 24 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<2, 25, 31, 31>2045 contrib.epu.quadric_custom_op (-64.0, 63.99999997019768) 25 N/A N/A
/model/layers.13/mlp/gate_proj/MatMul_output_0_slice strided_slice (-64.0, 63.99999997019768) 25 N/A N/A
/model/layers.13/mlp/act_fn/Sigmoid sigmoid [0.000205547f, 0.999985f] 31 N/A N/A
/model/layers.13/mlp/act_fn/Mul multiply [-0.278465f, 11.1382f] 25 N/A N/A
/model/layers.13/mlp/up_proj/MatMul_output_0_slice strided_slice (-64.0, 63.99999997019768) 25 N/A N/A
/model/layers.13/mlp/Mul multiply [-43.0674f, 23.3873f] 21 N/A N/A
/model/layers.13/mlp/down_proj/MatMul_smooth_mul multiply [-2.65824f, 3.22264f] 26 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<25, 31>45 contrib.epu.quadric_custom_op [-15.3465f, 14.4307f] 25 N/A N/A
/model/layers.13/Add_1 add [-178.422f, 11911f] 15 N/A N/A
/model/layers.14/input_layernorm/Mul_1 contrib.epu.rms_norm [-44.088f, 71.8568f] 24 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<3, 21, 31, 31>2014 contrib.epu.quadric_custom_op (-1024.0, 1023.9999995231628) 21 N/A 21
/model/layers.14/self_attn/q_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.14/self_attn/k_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.14/self_attn/MatMul_1_output_0_DequantizeLinear contrib.epu.dequantize [-5.18911f, 5.51635f] 28 N/A N/A
/model/layers.14/self_attn/Reshape_9 reshape [-5.18911f, 5.51635f] 28 N/A N/A
/model/layers.14/self_attn/o_proj/MatMul_smooth_mul multiply [-1.3136f, 1.43462f] 28 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<25, 31>2078 contrib.epu.quadric_custom_op [-13.355f, 10.8465f] 25 N/A N/A
/model/layers.14/Add add [-180.162f, 11911f] 15 N/A N/A
/model/layers.14/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-19.8879f, 82.1189f] 24 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<2, 25, 31, 31>2046 contrib.epu.quadric_custom_op (-64.0, 63.99999997019768) 25 N/A N/A
/model/layers.14/mlp/gate_proj/MatMul_output_0_slice strided_slice (-64.0, 63.99999997019768) 25 N/A N/A
/model/layers.14/mlp/act_fn/Sigmoid sigmoid [3.17693e-05f, 1f] 31 N/A N/A
/model/layers.14/mlp/act_fn/Mul multiply [-0.278465f, 14.7442f] 25 N/A N/A
/model/layers.14/mlp/up_proj/MatMul_output_0_slice strided_slice (-64.0, 63.99999997019768) 25 N/A N/A
/model/layers.14/mlp/Mul multiply [-22.0931f, 25.1863f] 21 N/A N/A
/model/layers.14/mlp/down_proj/MatMul_smooth_mul multiply [-4.78741f, 5.43421f] 26 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<25, 31>46 contrib.epu.quadric_custom_op [-10.6479f, 11.3865f] 25 N/A N/A
/model/layers.14/Add_1 add [-176.499f, 11908f] 15 N/A N/A
/model/layers.15/input_layernorm/Mul_1 contrib.epu.rms_norm [-37.6198f, 80.9442f] 24 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<3, 21, 31, 31>2015 contrib.epu.quadric_custom_op (-1024.0, 1023.9999995231628) 21 N/A 21
/model/layers.15/self_attn/q_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.15/self_attn/k_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.15/self_attn/MatMul_1_output_0_DequantizeLinear contrib.epu.dequantize [-5.63087f, 5.00521f] 28 N/A N/A
/model/layers.15/self_attn/Reshape_9 reshape [-5.63087f, 5.00521f] 28 N/A N/A
/model/layers.15/self_attn/o_proj/MatMul_smooth_mul multiply [-1.04014f, 1.15987f] 29 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<24, 31>2079 contrib.epu.quadric_custom_op [-21.6362f, 20.1737f] 24 N/A N/A
/model/layers.15/Add add [-198.135f, 11907.3f] 15 N/A N/A
/model/layers.15/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-21.1332f, 74.232f] 24 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<2, 24, 31, 31>2047 contrib.epu.quadric_custom_op (-128.0, 127.99999994039536) 24 N/A N/A
/model/layers.15/mlp/gate_proj/MatMul_output_0_slice strided_slice (-128.0, 127.99999994039536) 24 N/A N/A
/model/layers.15/mlp/act_fn/Sigmoid sigmoid [4.51803e-05f, 1f] 31 N/A N/A
/model/layers.15/mlp/act_fn/Mul multiply [-0.278465f, 16.9958f] 24 N/A N/A
/model/layers.15/mlp/up_proj/MatMul_output_0_slice strided_slice (-128.0, 127.99999994039536) 24 N/A N/A
/model/layers.15/mlp/Mul multiply [-29.1666f, 23.2331f] 19 N/A N/A
/model/layers.15/mlp/down_proj/MatMul_smooth_mul multiply [-4.47987f, 2.84912f] 26 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<24, 31>47 contrib.epu.quadric_custom_op [-21.5122f, 17.8665f] 24 N/A N/A
/model/layers.15/Add_1 add [-195.721f, 11905.8f] 15 N/A N/A
/model/layers.16/input_layernorm/Mul_1 contrib.epu.rms_norm [-36.2461f, 84.112f] 24 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<3, 21, 31, 31>2016 contrib.epu.quadric_custom_op (-1024.0, 1023.9999995231628) 21 N/A 21
/model/layers.16/self_attn/q_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.16/self_attn/k_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.16/self_attn/MatMul_1_output_0_DequantizeLinear contrib.epu.dequantize [-5.53058f, 5.95967f] 28 N/A N/A
/model/layers.16/self_attn/Reshape_9 reshape [-5.53058f, 5.95967f] 28 N/A N/A
/model/layers.16/self_attn/o_proj/MatMul_smooth_mul multiply [-2.63635f, 1.66058f] 28 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<25, 31>2080 contrib.epu.quadric_custom_op [-10.3541f, 13.0028f] 25 N/A N/A
/model/layers.16/Add add [-203.728f, 11905.5f] 15 N/A N/A
/model/layers.16/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-22.703f, 99.516f] 24 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<2, 23, 31, 31>2048 contrib.epu.quadric_custom_op (-256.0, 255.9999998807907) 23 N/A N/A
/model/layers.16/mlp/gate_proj/MatMul_output_0_slice strided_slice (-256.0, 255.9999998807907) 23 N/A N/A
/model/layers.16/mlp/act_fn/Sigmoid sigmoid [2.5928e-06f, 0.999999f] 31 N/A N/A
/model/layers.16/mlp/act_fn/Mul multiply [-0.278465f, 13.5776f] 23 N/A N/A
/model/layers.16/mlp/up_proj/MatMul_output_0_slice strided_slice (-256.0, 255.9999998807907) 23 N/A N/A
/model/layers.16/mlp/Mul multiply [-32.668f, 29.8404f] 19 N/A N/A
/model/layers.16/mlp/down_proj/MatMul_smooth_mul multiply [-3.73503f, 4.7745f] 26 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<24, 31>48 contrib.epu.quadric_custom_op [-16.2742f, 13.5607f] 24 N/A N/A
/model/layers.16/Add_1 add [-200.106f, 11903.6f] 15 N/A N/A
/model/layers.17/input_layernorm/Mul_1 contrib.epu.rms_norm [-42.6755f, 115.323f] 24 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<3, 21, 31, 31>2017 contrib.epu.quadric_custom_op (-1024.0, 1023.9999995231628) 21 N/A 21
/model/layers.17/self_attn/q_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.17/self_attn/k_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.17/self_attn/MatMul_1_output_0_DequantizeLinear contrib.epu.dequantize [-5.64975f, 6.51895f] 28 N/A N/A
/model/layers.17/self_attn/Reshape_9 reshape [-5.64975f, 6.51895f] 28 N/A N/A
/model/layers.17/self_attn/o_proj/MatMul_smooth_mul multiply [-1.49948f, 1.46266f] 28 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<25, 31>2081 contrib.epu.quadric_custom_op [-9.39301f, 11.3241f] 25 N/A N/A
/model/layers.17/Add add [-201.989f, 11903.5f] 15 N/A N/A
/model/layers.17/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-22.1857f, 80.9608f] 24 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<2, 25, 31, 31>2049 contrib.epu.quadric_custom_op (-64.0, 63.99999997019768) 25 N/A N/A
/model/layers.17/mlp/gate_proj/MatMul_output_0_slice strided_slice (-64.0, 63.99999997019768) 25 N/A N/A
/model/layers.17/mlp/act_fn/Sigmoid sigmoid [0.00023514f, 0.999998f] 31 N/A N/A
/model/layers.17/mlp/act_fn/Mul multiply [-0.278465f, 13.3774f] 25 N/A N/A
/model/layers.17/mlp/up_proj/MatMul_output_0_slice strided_slice (-64.0, 63.99999997019768) 25 N/A N/A
/model/layers.17/mlp/Mul multiply [-25.2184f, 24.2935f] 21 N/A N/A
/model/layers.17/mlp/down_proj/MatMul_smooth_mul multiply [-2.36083f, 2.28943f] 26 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<25, 31>49 contrib.epu.quadric_custom_op [-12.9577f, 12.6768f] 25 N/A N/A
/model/layers.17/Add_1 add [-200.304f, 11902.2f] 15 N/A N/A
/model/layers.18/input_layernorm/Mul_1 contrib.epu.rms_norm [-40.1976f, 84.1268f] 24 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<3, 21, 31, 31>2018 contrib.epu.quadric_custom_op (-1024.0, 1023.9999995231628) 21 N/A 21
/model/layers.18/self_attn/q_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.18/self_attn/k_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.18/self_attn/MatMul_1_output_0_DequantizeLinear contrib.epu.dequantize [-6.1547f, 6.95999f] 28 N/A N/A
/model/layers.18/self_attn/Reshape_9 reshape [-6.1547f, 6.95999f] 28 N/A N/A
/model/layers.18/self_attn/o_proj/MatMul_smooth_mul multiply [-1.18576f, 1.37133f] 28 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<25, 31>2082 contrib.epu.quadric_custom_op [-9.43028f, 12.0179f] 25 N/A N/A
/model/layers.18/Add add [-208.957f, 11902.1f] 15 N/A N/A
/model/layers.18/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-21.6636f, 93.6084f] 24 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<2, 24, 31, 31>2050 contrib.epu.quadric_custom_op (-128.0, 127.99999994039536) 24 N/A N/A
/model/layers.18/mlp/gate_proj/MatMul_output_0_slice strided_slice (-128.0, 127.99999994039536) 24 N/A N/A
/model/layers.18/mlp/act_fn/Sigmoid sigmoid [0f, 0.999977f] 31 N/A N/A
/model/layers.18/mlp/act_fn/Mul multiply [-0.278465f, 10.6897f] 24 N/A N/A
/model/layers.18/mlp/up_proj/MatMul_output_0_slice strided_slice (-128.0, 127.99999994039536) 24 N/A N/A
/model/layers.18/mlp/Mul multiply [-31.2847f, 29.0324f] 20 N/A N/A
/model/layers.18/mlp/down_proj/MatMul_smooth_mul multiply [-1.84823f, 3.69489f] 26 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<25, 31>50 contrib.epu.quadric_custom_op [-11.4183f, 10.5006f] 25 N/A N/A
/model/layers.18/Add_1 add [-206.793f, 11900.5f] 15 N/A N/A
/model/layers.19/input_layernorm/Mul_1 contrib.epu.rms_norm [-46.8404f, 99.9351f] 24 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<3, 21, 31, 31>2019 contrib.epu.quadric_custom_op (-1024.0, 1023.9999995231628) 21 N/A 21
/model/layers.19/self_attn/q_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.19/self_attn/k_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.19/self_attn/MatMul_1_output_0_DequantizeLinear contrib.epu.dequantize [-6.32537f, 5.91059f] 28 N/A N/A
/model/layers.19/self_attn/Reshape_9 reshape [-6.32537f, 5.91059f] 28 N/A N/A
/model/layers.19/self_attn/o_proj/MatMul_smooth_mul multiply [-1.1201f, 1.11377f] 29 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<25, 31>2083 contrib.epu.quadric_custom_op [-8.3925f, 8.18429f] 25 N/A N/A
/model/layers.19/Add add [-207.072f, 11900.5f] 15 N/A N/A
/model/layers.19/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-21.9142f, 109.809f] 24 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<2, 25, 31, 31>2051 contrib.epu.quadric_custom_op (-64.0, 63.99999997019768) 25 N/A N/A
/model/layers.19/mlp/gate_proj/MatMul_output_0_slice strided_slice (-64.0, 63.99999997019768) 25 N/A N/A
/model/layers.19/mlp/act_fn/Sigmoid sigmoid [5.96046e-08f, 1f] 31 N/A N/A
/model/layers.19/mlp/act_fn/Mul multiply [-0.278465f, 14.7918f] 25 N/A N/A
/model/layers.19/mlp/up_proj/MatMul_output_0_slice strided_slice (-64.0, 63.99999997019768) 25 N/A N/A
/model/layers.19/mlp/Mul multiply [-20.5575f, 42.4342f] 21 N/A N/A
/model/layers.19/mlp/down_proj/MatMul_smooth_mul multiply [-2.63107f, 2.60942f] 26 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<25, 31>51 contrib.epu.quadric_custom_op [-9.97335f, 12.2379f] 25 N/A N/A
/model/layers.19/Add_1 add [-205.579f, 11898.9f] 15 N/A N/A
/model/layers.20/input_layernorm/Mul_1 contrib.epu.rms_norm [-35.0801f, 87.3005f] 24 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<3, 21, 31, 31>2020 contrib.epu.quadric_custom_op (-1024.0, 1023.9999995231628) 21 N/A 21
/model/layers.20/self_attn/q_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.20/self_attn/k_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.20/self_attn/MatMul_1_output_0_DequantizeLinear contrib.epu.dequantize [-6.82097f, 5.96164f] 28 N/A N/A
/model/layers.20/self_attn/Reshape_9 reshape [-6.82097f, 5.96164f] 28 N/A N/A
/model/layers.20/self_attn/o_proj/MatMul_smooth_mul multiply [-1.04382f, 1.25627f] 29 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<26, 31>2084 contrib.epu.quadric_custom_op [-6.40514f, 6.22769f] 26 N/A N/A
/model/layers.20/Add add [-208.441f, 11898.7f] 15 N/A N/A
/model/layers.20/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-23.79f, 108.237f] 24 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<2, 23, 31, 31>2052 contrib.epu.quadric_custom_op (-256.0, 255.9999998807907) 23 N/A N/A
/model/layers.20/mlp/gate_proj/MatMul_output_0_slice strided_slice (-256.0, 255.9999998807907) 23 N/A N/A
/model/layers.20/mlp/act_fn/Sigmoid sigmoid [4.98295e-05f, 1f] 31 N/A N/A
/model/layers.20/mlp/act_fn/Mul multiply [-0.278465f, 36.3986f] 23 N/A N/A
/model/layers.20/mlp/up_proj/MatMul_output_0_slice strided_slice (-256.0, 255.9999998807907) 23 N/A N/A
/model/layers.20/mlp/Mul multiply [-35.0667f, 26.8339f] 17 N/A N/A
/model/layers.20/mlp/down_proj/MatMul_smooth_mul multiply [-2.56975f, 2.50132f] 26 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<25, 31>52 contrib.epu.quadric_custom_op [-11.1419f, 10.412f] 25 N/A N/A
/model/layers.20/Add_1 add [-208.339f, 11897.8f] 15 N/A N/A
/model/layers.21/input_layernorm/Mul_1 contrib.epu.rms_norm [-46.6267f, 129.576f] 23 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<3, 21, 31, 31>2021 contrib.epu.quadric_custom_op (-1024.0, 1023.9999995231628) 21 N/A 21
/model/layers.21/self_attn/q_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.21/self_attn/k_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.21/self_attn/MatMul_1_output_0_DequantizeLinear contrib.epu.dequantize [-7.99768f, 10.497f] 27 N/A N/A
/model/layers.21/self_attn/Reshape_9 reshape [-7.99768f, 10.497f] 27 N/A N/A
/model/layers.21/self_attn/o_proj/MatMul_smooth_mul multiply [-1.35199f, 1.39296f] 28 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<26, 31>2085 contrib.epu.quadric_custom_op [-7.79889f, 7.82725f] 26 N/A N/A
/model/layers.21/Add add [-208.633f, 11897.6f] 15 N/A N/A
/model/layers.21/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-24.781f, 101.921f] 24 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<2, 25, 31, 31>2053 contrib.epu.quadric_custom_op (-64.0, 63.99999997019768) 25 N/A N/A
/model/layers.21/mlp/gate_proj/MatMul_output_0_slice strided_slice (-64.0, 63.99999997019768) 25 N/A N/A
/model/layers.21/mlp/act_fn/Sigmoid sigmoid [2.37823e-05f, 0.999961f] 31 N/A N/A
/model/layers.21/mlp/act_fn/Mul multiply [-0.278465f, 10.1493f] 25 N/A N/A
/model/layers.21/mlp/up_proj/MatMul_output_0_slice strided_slice (-64.0, 63.99999997019768) 25 N/A N/A
/model/layers.21/mlp/Mul multiply [-24.557f, 32.4977f] 21 N/A N/A
/model/layers.21/mlp/down_proj/MatMul_smooth_mul multiply [-2.16436f, 3.55633f] 26 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<25, 31>53 contrib.epu.quadric_custom_op [-10.8824f, 10.7202f] 25 N/A N/A
/model/layers.21/Add_1 add [-209.695f, 11896.4f] 15 N/A N/A
/model/layers.22/input_layernorm/Mul_1 contrib.epu.rms_norm [-40.5426f, 80.1986f] 24 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<3, 21, 31, 31>2022 contrib.epu.quadric_custom_op (-1024.0, 1023.9999995231628) 21 N/A 21
/model/layers.22/self_attn/q_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.22/self_attn/k_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.22/self_attn/MatMul_1_output_0_DequantizeLinear contrib.epu.dequantize [-8.42352f, 6.83168f] 27 N/A N/A
/model/layers.22/self_attn/Reshape_9 reshape [-8.42352f, 6.83168f] 27 N/A N/A
/model/layers.22/self_attn/o_proj/MatMul_smooth_mul multiply [-1.22943f, 1.34143f] 29 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<25, 31>2086 contrib.epu.quadric_custom_op [-9.87298f, 10.0269f] 25 N/A N/A
/model/layers.22/Add add [-213.08f, 11895.2f] 15 N/A N/A
/model/layers.22/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-27.4618f, 94.8141f] 24 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<2, 24, 31, 31>2054 contrib.epu.quadric_custom_op (-128.0, 127.99999994039536) 24 N/A N/A
/model/layers.22/mlp/gate_proj/MatMul_output_0_slice strided_slice (-128.0, 127.99999994039536) 24 N/A N/A
/model/layers.22/mlp/act_fn/Sigmoid sigmoid [1.78814e-07f, 1f] 31 N/A N/A
/model/layers.22/mlp/act_fn/Mul multiply [-0.278465f, 14.9643f] 24 N/A N/A
/model/layers.22/mlp/up_proj/MatMul_output_0_slice strided_slice (-128.0, 127.99999994039536) 24 N/A N/A
/model/layers.22/mlp/Mul multiply [-35.1554f, 34.8362f] 20 N/A N/A
/model/layers.22/mlp/down_proj/MatMul_smooth_mul multiply [-2.63244f, 2.88847f] 26 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<25, 31>54 contrib.epu.quadric_custom_op [-13.4469f, 14.1453f] 25 N/A N/A
/model/layers.22/Add_1 add [-213.382f, 11894.5f] 15 N/A N/A
/model/layers.23/input_layernorm/Mul_1 contrib.epu.rms_norm [-44.0543f, 68.0886f] 24 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<3, 21, 31, 31>2023 contrib.epu.quadric_custom_op (-1024.0, 1023.9999995231628) 21 N/A 21
/model/layers.23/self_attn/q_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.23/self_attn/k_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.23/self_attn/MatMul_1_output_0_DequantizeLinear contrib.epu.dequantize [-7.58594f, 6.7776f] 28 N/A N/A
/model/layers.23/self_attn/Reshape_9 reshape [-7.58594f, 6.7776f] 28 N/A N/A
/model/layers.23/self_attn/o_proj/MatMul_smooth_mul multiply [-1.2456f, 1.48805f] 28 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<25, 31>2087 contrib.epu.quadric_custom_op [-12.3359f, 10.3974f] 25 N/A N/A
/model/layers.23/Add add [-211.914f, 11894.8f] 15 N/A N/A
/model/layers.23/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-28.2503f, 85.3382f] 24 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<2, 24, 31, 31>2055 contrib.epu.quadric_custom_op (-128.0, 127.99999994039536) 24 N/A N/A
/model/layers.23/mlp/gate_proj/MatMul_output_0_slice strided_slice (-128.0, 127.99999994039536) 24 N/A N/A
/model/layers.23/mlp/act_fn/Sigmoid sigmoid [1.01328e-06f, 1f] 31 N/A N/A
/model/layers.23/mlp/act_fn/Mul multiply [-0.278465f, 15.283f] 24 N/A N/A
/model/layers.23/mlp/up_proj/MatMul_output_0_slice strided_slice (-128.0, 127.99999994039536) 24 N/A N/A
/model/layers.23/mlp/Mul multiply [-44.2191f, 67.2178f] 20 N/A N/A
/model/layers.23/mlp/down_proj/MatMul_smooth_mul multiply [-3.83869f, 2.74354f] 26 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<24, 31>55 contrib.epu.quadric_custom_op [-13.1122f, 16.7432f] 24 N/A N/A
/model/layers.23/Add_1 add [-214.17f, 11891.4f] 15 N/A N/A
/model/layers.24/input_layernorm/Mul_1 contrib.epu.rms_norm [-48.4429f, 82.1183f] 24 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<3, 21, 31, 31>2024 contrib.epu.quadric_custom_op (-1024.0, 1023.9999995231628) 21 N/A 21
/model/layers.24/self_attn/q_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.24/self_attn/k_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.24/self_attn/MatMul_1_output_0_DequantizeLinear contrib.epu.dequantize [-6.80341f, 6.91227f] 28 N/A N/A
/model/layers.24/self_attn/Reshape_9 reshape [-6.80341f, 6.91227f] 28 N/A N/A
/model/layers.24/self_attn/o_proj/MatMul_smooth_mul multiply [-2.04986f, 1.35927f] 29 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<25, 31>2088 contrib.epu.quadric_custom_op [-11.0673f, 11.0469f] 25 N/A N/A
/model/layers.24/Add add [-212.156f, 11891.5f] 15 N/A N/A
/model/layers.24/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-30.2424f, 90.4779f] 24 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<2, 24, 31, 31>2056 contrib.epu.quadric_custom_op (-128.0, 127.99999994039536) 24 N/A N/A
/model/layers.24/mlp/gate_proj/MatMul_output_0_slice strided_slice (-128.0, 127.99999994039536) 24 N/A N/A
/model/layers.24/mlp/act_fn/Sigmoid sigmoid [1.3113e-06f, 1f] 31 N/A N/A
/model/layers.24/mlp/act_fn/Mul multiply [-0.278465f, 16.0448f] 24 N/A N/A
/model/layers.24/mlp/up_proj/MatMul_output_0_slice strided_slice (-128.0, 127.99999994039536) 24 N/A N/A
/model/layers.24/mlp/Mul multiply [-53.8111f, 48.3586f] 19 N/A N/A
/model/layers.24/mlp/down_proj/MatMul_smooth_mul multiply [-4.47011f, 3.50675f] 25 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<25, 31>56 contrib.epu.quadric_custom_op [-11.3723f, 11.5038f] 25 N/A N/A
/model/layers.24/Add_1 add [-208.892f, 11888.2f] 15 N/A N/A
/model/layers.25/input_layernorm/Mul_1 contrib.epu.rms_norm [-62.9176f, 96.3882f] 24 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<3, 21, 31, 31>2025 contrib.epu.quadric_custom_op (-1024.0, 1023.9999995231628) 21 N/A 21
/model/layers.25/self_attn/q_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.25/self_attn/k_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.25/self_attn/MatMul_1_output_0_DequantizeLinear contrib.epu.dequantize [-8.93482f, 7.30386f] 27 N/A N/A
/model/layers.25/self_attn/Reshape_9 reshape [-8.93482f, 7.30386f] 27 N/A N/A
/model/layers.25/self_attn/o_proj/MatMul_smooth_mul multiply [-1.40608f, 1.30193f] 29 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<25, 31>2089 contrib.epu.quadric_custom_op [-11.3195f, 15.912f] 25 N/A N/A
/model/layers.25/Add add [-208.439f, 11888.3f] 15 N/A N/A
/model/layers.25/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-31.3073f, 91.6664f] 24 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<2, 24, 31, 31>2057 contrib.epu.quadric_custom_op (-128.0, 127.99999994039536) 24 N/A N/A
/model/layers.25/mlp/gate_proj/MatMul_output_0_slice strided_slice (-128.0, 127.99999994039536) 24 N/A N/A
/model/layers.25/mlp/act_fn/Sigmoid sigmoid [5.96046e-08f, 1f] 31 N/A N/A
/model/layers.25/mlp/act_fn/Mul multiply [-0.278465f, 21.5028f] 24 N/A N/A
/model/layers.25/mlp/up_proj/MatMul_output_0_slice strided_slice (-128.0, 127.99999994039536) 24 N/A N/A
/model/layers.25/mlp/Mul multiply [-81.7404f, 63.9284f] 19 N/A N/A
/model/layers.25/mlp/down_proj/MatMul_smooth_mul multiply [-5.17006f, 8.23903f] 25 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<24, 31>57 contrib.epu.quadric_custom_op [-22.2006f, 28.2485f] 24 N/A N/A
/model/layers.25/Add_1 add [-206.494f, 11886.1f] 15 N/A N/A
/model/layers.26/input_layernorm/Mul_1 contrib.epu.rms_norm [-61.4985f, 89.9974f] 24 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<3, 21, 31, 31>2026 contrib.epu.quadric_custom_op (-1024.0, 1023.9999995231628) 21 N/A 21
/model/layers.26/self_attn/q_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.26/self_attn/k_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.26/self_attn/MatMul_1_output_0_DequantizeLinear contrib.epu.dequantize [-7.74915f, 8.00115f] 27 N/A N/A
/model/layers.26/self_attn/Reshape_9 reshape [-7.74915f, 8.00115f] 27 N/A N/A
/model/layers.26/self_attn/o_proj/MatMul_smooth_mul multiply [-1.59953f, 1.65345f] 28 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<25, 31>2090 contrib.epu.quadric_custom_op [-15.053f, 15.2435f] 25 N/A N/A
/model/layers.26/Add add [-205.78f, 11886.2f] 15 N/A N/A
/model/layers.26/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-33.2342f, 99.1755f] 24 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<2, 24, 31, 31>2058 contrib.epu.quadric_custom_op (-128.0, 127.99999994039536) 24 N/A N/A
/model/layers.26/mlp/gate_proj/MatMul_output_0_slice strided_slice (-128.0, 127.99999994039536) 24 N/A N/A
/model/layers.26/mlp/act_fn/Sigmoid sigmoid [0f, 1f] 31 N/A N/A
/model/layers.26/mlp/act_fn/Mul multiply [-0.278465f, 18.3359f] 24 N/A N/A
/model/layers.26/mlp/up_proj/MatMul_output_0_slice strided_slice (-128.0, 127.99999994039536) 24 N/A N/A
/model/layers.26/mlp/Mul multiply [-84.2759f, 82.2671f] 19 N/A N/A
/model/layers.26/mlp/down_proj/MatMul_smooth_mul multiply [-4.35105f, 3.87934f] 25 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<24, 31>58 contrib.epu.quadric_custom_op [-16.4064f, 22.7769f] 24 N/A N/A
/model/layers.26/Add_1 add [-195.304f, 11879.7f] 15 N/A N/A
/model/layers.27/input_layernorm/Mul_1 contrib.epu.rms_norm [-66.1633f, 100.225f] 24 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<3, 21, 31, 31>2027 contrib.epu.quadric_custom_op (-1024.0, 1023.9999995231628) 21 N/A 21
/model/layers.27/self_attn/q_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.27/self_attn/k_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.27/self_attn/MatMul_1_output_0_DequantizeLinear contrib.epu.dequantize [-10.516f, 9.69442f] 27 N/A N/A
/model/layers.27/self_attn/Reshape_9 reshape [-10.516f, 9.69442f] 27 N/A N/A
/model/layers.27/self_attn/o_proj/MatMul_smooth_mul multiply [-1.90749f, 2.45369f] 28 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<24, 31>2091 contrib.epu.quadric_custom_op [-15.4283f, 20.433f] 24 N/A N/A
/model/layers.27/Add add [-193.383f, 11879.7f] 15 N/A N/A
/model/layers.27/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-38.4993f, 124.068f] 24 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<2, 24, 31, 31>2059 contrib.epu.quadric_custom_op (-128.0, 127.99999994039536) 24 N/A N/A
/model/layers.27/mlp/gate_proj/MatMul_output_0_slice strided_slice (-128.0, 127.99999994039536) 24 N/A N/A
/model/layers.27/mlp/act_fn/Sigmoid sigmoid [0f, 1f] 31 N/A N/A
/model/layers.27/mlp/act_fn/Mul multiply [-0.278465f, 17.7156f] 24 N/A N/A
/model/layers.27/mlp/up_proj/MatMul_output_0_slice strided_slice (-128.0, 127.99999994039536) 24 N/A N/A
/model/layers.27/mlp/Mul multiply [-108.582f, 109.192f] 19 N/A N/A
/model/layers.27/mlp/down_proj/MatMul_smooth_mul multiply [-6.33056f, 5.03525f] 25 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<24, 31>59 contrib.epu.quadric_custom_op [-23.22f, 26.6939f] 24 N/A N/A
/model/layers.27/Add_1 add [-172.051f, 11869.6f] 15 N/A N/A
/model/layers.28/input_layernorm/Mul_1 contrib.epu.rms_norm [-59.978f, 80.5393f] 24 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<3, 21, 31, 31>2028 contrib.epu.quadric_custom_op (-1024.0, 1023.9999995231628) 21 N/A 21
/model/layers.28/self_attn/q_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.28/self_attn/k_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.28/self_attn/MatMul_1_output_0_DequantizeLinear contrib.epu.dequantize [-10.4083f, 9.99531f] 27 N/A N/A
/model/layers.28/self_attn/Reshape_9 reshape [-10.4083f, 9.99531f] 27 N/A N/A
/model/layers.28/self_attn/o_proj/MatMul_smooth_mul multiply [-1.76316f, 1.77728f] 28 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<24, 31>2092 contrib.epu.quadric_custom_op [-16.6094f, 21.3718f] 24 N/A N/A
/model/layers.28/Add add [-170.639f, 11869.6f] 15 N/A N/A
/model/layers.28/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-39.2268f, 161.993f] 23 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<2, 24, 31, 30>2060 contrib.epu.quadric_custom_op (-128.0, 127.99999994039536) 24 N/A N/A
/model/layers.28/mlp/gate_proj/MatMul_output_0_slice strided_slice (-128.0, 127.99999994039536) 24 N/A N/A
/model/layers.28/mlp/act_fn/Sigmoid sigmoid [0f, 1f] 31 N/A N/A
/model/layers.28/mlp/act_fn/Mul multiply [-0.278465f, 19.8816f] 24 N/A N/A
/model/layers.28/mlp/up_proj/MatMul_output_0_slice strided_slice (-128.0, 127.99999994039536) 24 N/A N/A
/model/layers.28/mlp/Mul multiply [-122.736f, 118.794f] 19 N/A N/A
/model/layers.28/mlp/down_proj/MatMul_smooth_mul multiply [-6.35068f, 5.46193f] 25 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<23, 31>60 contrib.epu.quadric_custom_op [-36.8502f, 33.2024f] 23 N/A N/A
/model/layers.28/Add_1 add [-155.349f, 11860.9f] 15 N/A N/A
/model/layers.29/input_layernorm/Mul_1 contrib.epu.rms_norm [-61.4842f, 82.8766f] 24 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<3, 21, 31, 31>2029 contrib.epu.quadric_custom_op (-1024.0, 1023.9999995231628) 21 N/A 21
/model/layers.29/self_attn/q_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.29/self_attn/k_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.29/self_attn/MatMul_1_output_0_DequantizeLinear contrib.epu.dequantize [-10.6212f, 11.6284f] 27 N/A N/A
/model/layers.29/self_attn/Reshape_9 reshape [-10.6212f, 11.6284f] 27 N/A N/A
/model/layers.29/self_attn/o_proj/MatMul_smooth_mul multiply [-2.46839f, 2.49912f] 28 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<23, 31>2093 contrib.epu.quadric_custom_op [-33.6687f, 39.5864f] 23 N/A N/A
/model/layers.29/Add add [-166.203f, 11860.7f] 15 N/A N/A
/model/layers.29/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-45.1833f, 225.968f] 23 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<2, 22, 31, 31>2061 contrib.epu.quadric_custom_op (-512.0, 511.9999997615814) 22 N/A N/A
/model/layers.29/mlp/gate_proj/MatMul_output_0_slice strided_slice (-512.0, 511.9999997615814) 22 N/A N/A
/model/layers.29/mlp/act_fn/Sigmoid sigmoid [0f, 1f] 31 N/A N/A
/model/layers.29/mlp/act_fn/Mul multiply [-0.278465f, 91.9513f] 22 N/A N/A
/model/layers.29/mlp/up_proj/MatMul_output_0_slice strided_slice (-512.0, 511.9999997615814) 22 N/A N/A
/model/layers.29/mlp/Mul multiply [-91.1271f, 95.7995f] 15 N/A N/A
/model/layers.29/mlp/down_proj/MatMul_smooth_mul multiply [-5.87016f, 5.67612f] 25 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<23, 31>61 contrib.epu.quadric_custom_op [-43.9362f, 47.5904f] 23 N/A N/A
/model/layers.29/Add_1 add [-152.491f, 11850.9f] 15 N/A N/A
/model/layers.30/input_layernorm/Mul_1 contrib.epu.rms_norm [-59.4241f, 70.712f] 24 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<3, 21, 31, 31>2030 contrib.epu.quadric_custom_op (-1024.0, 1023.9999995231628) 21 N/A 21
/model/layers.30/self_attn/q_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.30/self_attn/k_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.30/self_attn/MatMul_1_output_0_DequantizeLinear contrib.epu.dequantize [-9.23528f, 13.3282f] 27 N/A N/A
/model/layers.30/self_attn/Reshape_9 reshape [-9.23528f, 13.3282f] 27 N/A N/A
/model/layers.30/self_attn/o_proj/MatMul_smooth_mul multiply [-2.55306f, 2.95738f] 27 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<22, 31>2094 contrib.epu.quadric_custom_op [-114.016f, 51.831f] 22 N/A N/A
/model/layers.30/Add add [-158.057f, 11850.9f] 15 N/A N/A
/model/layers.30/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-73.8593f, 307.994f] 22 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<2, 22, 31, 31>2062 contrib.epu.quadric_custom_op (-512.0, 511.9999997615814) 22 N/A N/A
/model/layers.30/mlp/gate_proj/MatMul_output_0_slice strided_slice (-512.0, 511.9999997615814) 22 N/A N/A
/model/layers.30/mlp/act_fn/Sigmoid sigmoid [0f, 1f] 31 N/A N/A
/model/layers.30/mlp/act_fn/Mul multiply [-0.278465f, 104.112f] 22 N/A N/A
/model/layers.30/mlp/up_proj/MatMul_output_0_slice strided_slice (-512.0, 511.9999997615814) 22 N/A N/A
/model/layers.30/mlp/Mul multiply [-131.596f, 91.2558f] 15 N/A N/A
/model/layers.30/mlp/down_proj/MatMul_smooth_mul multiply [-6.00577f, 7.463f] 25 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<22, 31>62 contrib.epu.quadric_custom_op [-98.0825f, 63.2593f] 22 N/A N/A
/model/layers.30/Add_1 add [-155.469f, 11842.9f] 15 N/A N/A
/model/layers.31/input_layernorm/Mul_1 contrib.epu.rms_norm [-69.0611f, 127.97f] 23 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<3, 21, 31, 31>2031 contrib.epu.quadric_custom_op (-1024.0, 1023.9999995231628) 21 N/A 21
/model/layers.31/self_attn/q_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.31/self_attn/k_proj/MatMul_output_0_slice strided_slice (-1024.0, 1023.9999995231628) 21 N/A N/A
/model/layers.31/self_attn/MatMul_1_output_0_DequantizeLinear contrib.epu.dequantize [-22.1057f, 19.6099f] 26 N/A N/A
/model/layers.31/self_attn/Reshape_9 reshape [-22.1057f, 19.6099f] 26 N/A N/A
/model/layers.31/self_attn/o_proj/MatMul_smooth_mul multiply [-3.5795f, 3.53745f] 28 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<18, 31>2095 contrib.epu.quadric_custom_op [-1840.37f, 87.7969f] 18 N/A N/A
/model/layers.31/Add add [-146.437f, 11843.5f] 15 N/A N/A
/model/layers.31/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-190.178f, 672.768f] 21 N/A N/A
CustomOp/nn_custom::w4a16PackedProj<2, 19, 30, 31>2063 contrib.epu.quadric_custom_op (-4096.0, 4095.9999980926514) 19 N/A N/A
/model/layers.31/mlp/gate_proj/MatMul_output_0_slice strided_slice (-4096.0, 4095.9999980926514) 19 N/A N/A
/model/layers.31/mlp/act_fn/Sigmoid sigmoid [0f, 1f] 31 N/A N/A
/model/layers.31/mlp/act_fn/Mul multiply [-0.278465f, 394.496f] 19 N/A N/A
/model/layers.31/mlp/up_proj/MatMul_output_0_slice strided_slice (-4096.0, 4095.9999980926514) 19 N/A N/A
/model/layers.31/mlp/Mul multiply [-3034.05f, 1733.99f] 10 N/A N/A
/model/layers.31/mlp/down_proj/MatMul_smooth_mul multiply [-27.9686f, 25.8695f] 20 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<17, 30>63 contrib.epu.quadric_custom_op [-780.54f, 616.187f] 17 N/A N/A
/model/layers.31/Add_1 add [-788.549f, 11280.2f] 15 N/A N/A
/model/norm/Mul_1 contrib.epu.rms_norm [-299.409f, 257.785f] 22 N/A N/A
/Gather take [-238.949f, 236.277f] 22 N/A N/A
/lm_head/MatMul_smooth_mul multiply [-4.66097f, 4.65724f] 23 N/A N/A
CustomOp/nn_custom::w4a16NBitsMatMul<23, 31>2096 contrib.epu.quadric_custom_op [-29.1443f, 37.6655f] 23 N/A N/A
2026-09-22 02:58 - INFO - epu - codegen - START====================build_cpu_runnable_fx_relay
2026-09-22 02:58 - INFO - epu - codegen - START=======================quantize_to_chimera_fx
2026-09-22 03:00 - INFO - epu - codegen - START=================================relay_to_tir
2026-09-22 03:00 - INFO - epu - codegen - START===========================relay_to_epu_relay
2026-09-22 03:00 - INFO - epu - codegen - START==============================adapt_and_order
2026-09-22 03:02 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-09-22 03:02 - INFO - epu - codegen - START=============================plan_lrm_virtual
2026-09-22 03:05 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-09-22 03:05 - INFO - epu - codegen - START===============================lrm_alloc_loop
2026-09-22 03:08 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-09-22 03:09 - INFO - epu - codegen - START================================lrm_splitting
2026-09-22 03:15 - INFO - epu - codegen - START==============================ext_split_relay
2026-09-22 03:19 - INFO - epu - codegen - START====================================build_tir
2026-09-22 03:19 - INFO - epu - chimera_job - Compilation of sarashina_QC_U_1d5_8MB_4kB_128GBps_128GBps_8_OFF_x1_x1 successful
[meta] sarashina_QC_U_1d5_8MB_4kB_128GBps_128GBps_8_OFF_x1_x1 logits<23> rope<30> ext_persistent=41943040 -> iss_meta.json
╒═════════════════════╤═══════════════════════════════════════════════════════════════════════════════════════════════════════════╕
│ Module Name │ sarashina_QC_U_1d5_8MB_4kB_128GBps_128GBps_8_OFF_x1_x1 │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ ONNX File │ /quadric/sdk-cli/examples/models/sarashina2.2-3b/iss/work_w4a16_fused/custom_op/packed_hr2/sarashina.onnx │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Product Target │ QC-U │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Number of Cores │ 1 │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ ISS Clock Frequency │ 1.500 │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ L2M Size │ 8MB │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ LRM Size │ 4kB │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ External Read BW │ 128GBps │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ External Write BW │ 128GBps │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ MACS per PE │ 8 │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Max L2M │ 0.361MB │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Max LRM │ 0.180kB │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Max Temp Ext Bytes │ 0.400MB │
├─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Network GMACs │ │
╘═════════════════════╧═══════════════════════════════════════════════════════════════════════════════════════════════════════════╛
╒═════╤════════╤══════════════════════════╤══════════════════╤══════════════════════════╤═══════╕
│ │ Type │ Name │ shape │ type │ mse │
╞═════╪════════╪══════════════════════════╪══════════════════╪══════════════════════════╪═══════╡
│ 0 │ Input │ input_ids │ [1, 1] │ tensor[int32] │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 1 │ Input │ attention_mask │ [1, 1, 512] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 2 │ Input │ sin │ [1, 1, 160] │ tensor[FixedPoint32<30>] │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 3 │ Input │ cos │ [1, 1, 160] │ tensor[FixedPoint32<30>] │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 4 │ Input │ past_key_values.0.key │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 5 │ Input │ past_key_values.0.value │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 6 │ Input │ past_key_values.1.key │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 7 │ Input │ past_key_values.1.value │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 8 │ Input │ past_key_values.2.key │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 9 │ Input │ past_key_values.2.value │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 10 │ Input │ past_key_values.3.key │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 11 │ Input │ past_key_values.3.value │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 12 │ Input │ past_key_values.4.key │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 13 │ Input │ past_key_values.4.value │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 14 │ Input │ past_key_values.5.key │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 15 │ Input │ past_key_values.5.value │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 16 │ Input │ past_key_values.6.key │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 17 │ Input │ past_key_values.6.value │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 18 │ Input │ past_key_values.7.key │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 19 │ Input │ past_key_values.7.value │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 20 │ Input │ past_key_values.8.key │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 21 │ Input │ past_key_values.8.value │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 22 │ Input │ past_key_values.9.key │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 23 │ Input │ past_key_values.9.value │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 24 │ Input │ past_key_values.10.key │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 25 │ Input │ past_key_values.10.value │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 26 │ Input │ past_key_values.11.key │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 27 │ Input │ past_key_values.11.value │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 28 │ Input │ past_key_values.12.key │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 29 │ Input │ past_key_values.12.value │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 30 │ Input │ past_key_values.13.key │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 31 │ Input │ past_key_values.13.value │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 32 │ Input │ past_key_values.14.key │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 33 │ Input │ past_key_values.14.value │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 34 │ Input │ past_key_values.15.key │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 35 │ Input │ past_key_values.15.value │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 36 │ Input │ past_key_values.16.key │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 37 │ Input │ past_key_values.16.value │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 38 │ Input │ past_key_values.17.key │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 39 │ Input │ past_key_values.17.value │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 40 │ Input │ past_key_values.18.key │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 41 │ Input │ past_key_values.18.value │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 42 │ Input │ past_key_values.19.key │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 43 │ Input │ past_key_values.19.value │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 44 │ Input │ past_key_values.20.key │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 45 │ Input │ past_key_values.20.value │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 46 │ Input │ past_key_values.21.key │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 47 │ Input │ past_key_values.21.value │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 48 │ Input │ past_key_values.22.key │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 49 │ Input │ past_key_values.22.value │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 50 │ Input │ past_key_values.23.key │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 51 │ Input │ past_key_values.23.value │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 52 │ Input │ past_key_values.24.key │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 53 │ Input │ past_key_values.24.value │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 54 │ Input │ past_key_values.25.key │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 55 │ Input │ past_key_values.25.value │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 56 │ Input │ past_key_values.26.key │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 57 │ Input │ past_key_values.26.value │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 58 │ Input │ past_key_values.27.key │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 59 │ Input │ past_key_values.27.value │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 60 │ Input │ past_key_values.28.key │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 61 │ Input │ past_key_values.28.value │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 62 │ Input │ past_key_values.29.key │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 63 │ Input │ past_key_values.29.value │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 64 │ Input │ past_key_values.30.key │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 65 │ Input │ past_key_values.30.value │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 66 │ Input │ past_key_values.31.key │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 67 │ Input │ past_key_values.31.value │ [1, 8, 511, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 68 │ Output │ logits │ [1, 102400] │ tensor[FixedPoint32<23>] │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 69 │ Output │ present.0.key │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 70 │ Output │ present.0.value │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 71 │ Output │ present.1.key │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 72 │ Output │ present.1.value │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 73 │ Output │ present.2.key │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 74 │ Output │ present.2.value │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 75 │ Output │ present.3.key │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 76 │ Output │ present.3.value │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 77 │ Output │ present.4.key │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 78 │ Output │ present.4.value │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 79 │ Output │ present.5.key │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 80 │ Output │ present.5.value │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 81 │ Output │ present.6.key │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 82 │ Output │ present.6.value │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 83 │ Output │ present.7.key │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 84 │ Output │ present.7.value │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 85 │ Output │ present.8.key │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 86 │ Output │ present.8.value │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 87 │ Output │ present.9.key │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 88 │ Output │ present.9.value │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 89 │ Output │ present.10.key │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 90 │ Output │ present.10.value │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 91 │ Output │ present.11.key │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 92 │ Output │ present.11.value │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 93 │ Output │ present.12.key │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 94 │ Output │ present.12.value │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 95 │ Output │ present.13.key │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 96 │ Output │ present.13.value │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 97 │ Output │ present.14.key │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 98 │ Output │ present.14.value │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 99 │ Output │ present.15.key │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 100 │ Output │ present.15.value │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 101 │ Output │ present.16.key │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 102 │ Output │ present.16.value │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 103 │ Output │ present.17.key │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 104 │ Output │ present.17.value │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 105 │ Output │ present.18.key │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 106 │ Output │ present.18.value │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 107 │ Output │ present.19.key │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 108 │ Output │ present.19.value │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 109 │ Output │ present.20.key │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 110 │ Output │ present.20.value │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 111 │ Output │ present.21.key │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 112 │ Output │ present.21.value │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 113 │ Output │ present.22.key │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 114 │ Output │ present.22.value │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 115 │ Output │ present.23.key │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 116 │ Output │ present.23.value │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 117 │ Output │ present.24.key │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 118 │ Output │ present.24.value │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 119 │ Output │ present.25.key │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 120 │ Output │ present.25.value │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 121 │ Output │ present.26.key │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 122 │ Output │ present.26.value │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 123 │ Output │ present.27.key │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 124 │ Output │ present.27.value │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 125 │ Output │ present.28.key │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 126 │ Output │ present.28.value │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 127 │ Output │ present.29.key │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 128 │ Output │ present.29.value │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 129 │ Output │ present.30.key │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 130 │ Output │ present.30.value │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 131 │ Output │ present.31.key │ [1, 8, 512, 160] │ n/a │ n/a │
├─────┼────────┼──────────────────────────┼──────────────────┼──────────────────────────┼───────┤
│ 132 │ Output │ present.31.value │ [1, 8, 512, 160] │ n/a │ n/a │
╘═════╧════════╧══════════════════════════╧══════════════════╧══════════════════════════╧═══════╛
5. Prompt, driver, kernel → inputs/<prompt>/, kernel/<knobs>/<hw>_nt<N>/
The prompt is tokenized through the chat template and padded to 128 positions; the RoPE tables cover pad_to + new_tokens positions at the FixedPoint32 format CGC declared for the sin/cos inputs.
Sarashina decodes autoregressively, so it has its own driver: sarashina_driver.cpp runs the graph once per prompt token, then new_tokens times with the on-device argmax fed back, raising a software interrupt after every generated token (the board runner reads the token there). assemble_cpp.py splices the generated entry function into it and records the logits' FixedPoint32<F> in logits_frac_bits.txt. The SDK compiles the result to sarashina_epu.qo (the kernel) and sarashina_host (the archsim host program).
rendered, token_ids = sara.preview_prompt(cfg)
print("Prompt as the kernel sees it:", repr(rendered))
print(f"Token ids ({len(token_ids)}, padded to {cfg.pad_to}):", token_ids)
kernel = sara.build_kernel(cfg, quiet=True, force="kernel" in FORCE)
print(f"\nkernel: {kernel.relative_to(cfg.workdir)} ({kernel.stat().st_size/1e6:.2f} MB)")
tokenizer_config.json: 0.00B [00:00, ?B/s]
tokenizer.model: 0%| | 0.00/1.83M [00:00<?, ?B/s]
special_tokens_map.json: 0%| | 0.00/968 [00:00<?, ?B/s]
Prompt as the kernel sees it: '<|user|>What is AI?</s><|assistant|>'
Token ids (7, padded to 128): [9, 803, 301, 8880, 369, 2, 8]
[inputs] inputs/what_is_ai_p128_n1 (no STAGE.json)
[kernel] building kernel/packed_hr2/QC-U_1d5_8MB_4kB_128GBps_128GBps_8_OFF_x1_x1_nt1 (no STAGE.json)
[kv] ext_persistent_size -> 67108864 (64 MB)
[assemble]
[compile] QC-U (array_size 32) -> sarashina_QC-U_1d5_8MB_4kB_128GBps_128GBps_8_OFF_x1_x1
2026-09-22 03:19 - DEBUG - sdk - cli - Executing command: cmake CMakeLists.txt -B /quadric/sdk-cli/examples/models/sarashina2.2-3b/iss/work_w4a16_fused/kernel/packed_hr2/QC-U_1d5_8MB_4kB_128GBps_128GBps_8_OFF_x1_x1_nt1/sarashina_QC-U_1d5_8MB_4kB_128GBps_128GBps_8_OFF_x1_x1/build -DNUM_GPNPUS=1 -DNUM_CORES=32 -DNUM_BORDERS=2 -DEPU_VERSION=2.0.0 -DQLLVM_ROOT_PATH=/quadric/llvm -DOCM_SIZE_KIBIBYTES=8192 -DNUM_PE_MACS=8 -DASSERT_MLS_WIDTH_LINE_ALIGN=ON -DSTACKOVERFLOW_ERROR=OFF -DHARDWARE_TARGET=OFF
2026-09-22 03:19 - DEBUG - sdk - cli - Executing command: make -j8
kernel: kernel/packed_hr2/QC-U_1d5_8MB_4kB_128GBps_128GBps_8_OFF_x1_x1_nt1/sarashina_QC-U_1d5_8MB_4kB_128GBps_128GBps_8_OFF_x1_x1/output/sarashina_epu.qo (3.65 MB)
6. Execute
ISS: the host program runs the kernel cycle-accurately with the product's real DDR width and OCM size. Each run gets its own runs/<stamp>_…/ directory (inputs linked in, outputs written there, RUN.json with the kernel fingerprint, cycle count, tokens and the logits' frac bits); nothing is overwritten. A 7-token prompt with one generated token is 7 kernel passes (the last prompt token's pass yields the first generated token): about 9 minutes on QC-P, 20 on QC-U.
Board: publish() copies the kernel stage into output/<tag>/ — kernel, packaging files, the constant blob (copied, not linked), inputs, iss_meta.json, logits_frac_bits.txt, build_knobs.json and a MANIFEST.json naming the stage directories it came from. That directory is the complete contract with the board runner (fpga/run_haps.py, submitted by fpga/run_haps.sbatch); nothing else in the workdir is needed to run the kernel on the FPGA, and fpga/score_haps.py scores the board's first generated token against the same ORT golden the ISS is scored on.
if cfg.device == "iss":
run_dir = sara.run_iss(cfg)
ids, text = sara.decode_output(cfg)
print("\nGenerated token ids:", ids)
print("Response:", repr(text))
# One line per metric rather than a dict repr: this output is what the docs pipeline parses.
for name, value in sara.profile_summary(cfg).items():
print(f" {name:16} {value}")
else:
published = sara.publish(cfg)
print(
f"\nPublished {published}. On the board host: "
f"sbatch fpga/run_haps.sbatch {published} "
"(then fpga/score_haps.py <run dir> here; README, 'Running the handoff on the HAPS-200')."
)
[iss] QC-U @ 1.5 GHz, AXI 1024 -> runs/20260922-032125_packed_hr2_QC-U_what_is_ai_p128_n1 (~9 min for a 7-token prompt)
pass 1 complete (585 regions)
pass 2 complete (585 regions)
pass 3 complete (585 regions)
pass 4 complete (585 regions)
pass 5 complete (585 regions)
pass 6 complete (585 regions)
pass 7 complete (585 regions)
[ PC 284819, KGVCID {00:00} ][Cycle Count ] IMD register cyclecount = 32590744
[iss] cyclecount 32,590,744 tokens [3792] -> /quadric/sdk-cli/examples/models/sarashina2.2-3b/iss/work_w4a16_fused/runs/20260922-032125_packed_hr2_QC-U_what_is_ai_p128_n1
Generated token ids: [3792]
Response: 'AI'
cycles 32,590,744
tokens_per_sec 46.0
7. Compare Against ONNX Runtime
A token that reads well is not a correctness check. The reference is the same int4 graph before custom-op replacement, executed by ONNX Runtime in full precision on the same chat-templated prompt: score_run() builds that golden itself (iss/ort_logits.py on the dynamic twin, cached under graphs/ort_golden/) and compares it with the logits the kernel wrote.
The gate, and why these numbers. Two conditions must both hold, and the cell asserts them:
rank_of_ort_argmax == 0— the kernel's most likely token is ONNX Runtime's, so generation starts on the same path.rel_rmse_affine <= 0.08— the residual after the best affine fit between the two logit vectors, over the reference's std. A constant offset is a softmax no-op, which is why the fit is affine.
The ceiling is set from two measurements rather than taste. Every build of this pipeline has landed between 0.0448 (current devel) and 0.0554 (the 2026-09 toolchain), and two ONNX Runtime runs of the same weights under two nearby quantization recipes differ by 0.0383. So 0.08 sits above every kernel measured and below twice the disagreement that recipe choice alone produces: it fails a regression without failing on fixed-point noise.
Fixed-point custom kernels do not reproduce ONNX Runtime's fp32 execution bit for bit, which is why the gate is a bounded divergence rather than equality.
Two heavier cross-checks are available for the same graph, and neither is part of the gate. Greedy decode in ONNX Runtime (sara.ort_reference(cfg)) shows what full precision emits for this prompt, and teacher-forcing perplexity on the built-in EN/JP passages (sara.perplexity(cfg)) puts a number on the quantization itself. Both take minutes and need the front-end venv:
ort_ids, ort_text = sara.ort_reference(cfg) # same prompt, same new_tokens, fp32
print(sara.perplexity(cfg)) # teacher-forced perplexity of the ISS graph
## 0.08: above every kernel measured (0.0448 on current devel, 0.0554 on the 2026-09 toolchain),
## below twice the 0.0383 that two ORT runs of the same weights differ by under nearby recipes.
REL_RMSE_CEILING = 0.08
if cfg.device == "haps":
print(
"DEVICE='haps' publishes a handoff instead of running the kernel here. Score the board "
"run with fpga/score_haps.py, which applies this same gate to the board's first token."
)
elif cfg.new_tokens != 1:
# Not a skip: the golden is the FIRST generated token, and a multi-token ISS run leaves only
# the last step in output_logits.bin, so there is nothing here this gate can legitimately
# compare. Fail rather than print a reassuring line.
raise RuntimeError(
f"cannot score an ISS run with new_tokens={cfg.new_tokens}: the ORT golden is the first "
"generated token's logits, and output_logits.bin holds only the last step. Set "
"NEW_TOKENS = 1 to score here, or score a board run with fpga/score_haps.py."
)
else:
metrics = sara.score_run(cfg)
assert metrics["rank_of_ort_argmax"] == 0, f"kernel top-1 is not ORT's top-1: {metrics}"
assert (
metrics["rel_rmse_affine"] <= REL_RMSE_CEILING
), f"logit divergence {metrics['rel_rmse_affine']:.4f} > {REL_RMSE_CEILING}: {metrics}"
print(
f"PASS: same top-1 token as ONNX Runtime, rel_rmse_affine "
f"{metrics['rel_rmse_affine']:.4f} <= {REL_RMSE_CEILING}"
)
[golden] ORT on seq_int4_qkqdq_dyn.onnx for 'What is AI?' (no STAGE.json)
ids=[9, 803, 301, 8880, 369, 2, 8] argmax=3792 gen=[] saved /quadric/sdk-cli/examples/models/sarashina2.2-3b/iss/work_w4a16_fused/graphs/ort_golden/what_is_ai.npy shape=(1, 102400)
OK 20260922-032125_packed_hr2_QC-U_what_is_ai_p128_n1 (cycles 32,590,744, logits FixedPoint32<23>)
rank_of_ort_argmax 0 topk_overlap 1.00 pearson 0.99900 spearman 0.99873
scale_fit 0.9991 (log2 -0.00) rel_rmse 0.0453 KL 0.0025
affine a=1.0032 b=+0.0220 ** rel_rmse_affine 0.0448 **
PASS: same top-1 token as ONNX Runtime, rel_rmse_affine 0.0448 <= 0.08
8. Where everything is
iss/work_w4a16_fused/
graphs/ int4 graphs + calibration ranges (Stages 1-2), ort_golden/
custom_op/<knobs>/ sarashina.onnx (+ .data), build_knobs.json
cgc/<knobs>/<hw>/ ccl_build/<entry>/ (generated C++, const blob), iss_meta.json
kernel/<knobs>/<hw>_nt<N>/ sarashina.cpp, logits_frac_bits.txt, sarashina_<hw>/output/*.qo
inputs/<prompt>_p<pad>_n<N>/ input_logits.bin, prompt_len.bin, sin.bin, cos.bin
runs/<stamp>_<knobs>_<product>_<prompt>/ output_*.bin, iss_run.log, RUN.json
runs/latest -> newest run
output/<tag>[_<variant>]/ board handoffs
Change a knob above and re-run from the top: only the stages downstream of it rebuild. A different IMF_HEADROOM builds beside the current variant rather than over it; a different TARGET reuses graphs/ and custom_op/ and compiles only cgc/ and kernel/ for the new hardware.
sara.stage_status(cfg)
workdir /quadric/sdk-cli/examples/models/sarashina2.2-3b/iss/work_w4a16_fused
variant packed_hr2 hardware QC-U_1d5_8MB_4kB_128GBps_128GBps_8_OFF_x1_x1 new_tokens 1
graphs graphs 2026-09-22T02:53 mode=w4a16_fused, seq_len=512, calib_samples=128
custom_op custom_op/packed_hr2 2026-09-22T02:57 proj_mode=packed, imf_headroom=2, fuse_v_with_qk=True, pack_fused_proj=True, seq_len=512, num_kv_heads=8, embed_dim=2560, head_dim=160
cgc cgc/packed_hr2/QC-U_1d5_8MB_4kB_128GBps_128GBps_8_OFF_x1_x1 2026-09-22T03:19 target_lang=QIL, seq_len=512, io_to_ignore_layers=32
kernel kernel/packed_hr2/QC-U_1d5_8MB_4kB_128GBps_128GBps_8_OFF_x1_x1_nt1 2026-09-22T03:21 new_tokens=1, max_prompt_length=128, kv_persist_mb=64
inputs inputs/what_is_ai_p128_n1 2026-09-22T03:19 prompt=What is AI?, pad_to=128, new_tokens=1, rope_frac_bits=30
latest run runs/20260922-032125_packed_hr2_QC-U_what_is_ai_p128_n1 cycles 32590744 tokens [3792]
Summary
| Model | Sarashina2.2-3B Instruct — 32 layers, hidden 2560, 16 query / 8 KV |
| heads, head_dim 160, 102,400-token vocabulary, sequence 512 | |
| Quantization | w4a16 — SmoothQuant ranges, then one weight-only int4 pass |
| (block 128, asymmetric) over every projection and the LM head; int8 Q/K and KV cache | |
| Target | QC-U (array 32), 1.5 GHz, 8 MB OCM, 4 kB LRM, 128 GB/s DDR, 1024-bit AXI |
| Custom ops | nn::qwenMultiheadAttentionFx32RoPE, w4a16PackedProj (q|k|v, |
gate|up), nn_custom::w4a16NBitsMatMul (o, down, lm_head) | |
| Decode | one step of a 7-token prompt in 32,600,227 cycles, ~46 tokens/s; first |
token 3792, which is ONNX Runtime's, at rel_rmse_affine 0.0448 | |
| Hardware | the same graph on the HAPS-200 generated 128 tokens in 322 s, first |
| token matching ONNX Runtime |
Key takeaways
- One quantizer is the single source of truth. SmoothQuant smooths every projection but holds all of them out of int8, so the int4 step alone decides every weight, and the calibration ranges it leaves behind size every FixedPoint32 format downstream. Nothing re-fits a second grid.
- The autoregressive loop is the kernel's, not the host's.
sarashina_driver.cppruns the compiled graph once per prompt token and then per generated token, feeding the on-device argmax back and keeping the int8 KV cache in persistent DDR, so the host never round-trips attention state between tokens. - Stages are content-addressed, not ordered. Each writes a
STAGE.jsonnaming the inputs, knobs and toolchain it used, so changingTARGETrecompiles only CGC and the kernel, while a newIMF_HEADROOMbuilds beside the current variant instead of over it. - The gate is a bounded divergence from ONNX Runtime. Fixed-point kernels do not reproduce fp32 bit for bit, so section 7 asserts the same top-1 token plus
rel_rmse_affineunder a ceiling justified by measurement, rather than pinning a string in the test file.
Citation
@misc{sarashina2.2-3b-instruct-v0.1,
title = {Sarashina2.2-3B-Instruct-v0.1},
author = {SB Intuitions},
year = {2025},
howpublished = {\url{https://huggingface.co/sbintuitions/sarashina2.2-3b-instruct-v0.1}}
}
