NOTE: The Jupyter Notebook below is included in the Chimera SDK and can be run interactively by running the following CLI command:
$ quadric sdk notebook
From the Jupyter Notebook window in your browser, select the notebook named /quadric/sdk-cli/examples/models/qwen/components/qwen_decoder.ipynb.
Lowering the QWEN Decoder
1. Overview
In this notebook, we demonstrate the the approach we take ot lower the QWEN decoder block.
The final network runner will utilize two ONNX graphs:
- The first graph is used for the initial run (first token).
- The second graph is used for subsequent autoregressive steps. (attach diagrams of runner + 2 onnx graphs)
This methodology enables efficient network execution while exclusively requiring static shapes in ONNX. Here we show the differences in dimension between the first graph (left) and the second graph (right):
Note: In addition to the decoder, we have also attached the final Language Modeling (LM) head to our second graph to demonstate our ability to lower it.
2. Lowering the First Graph (Initial Token Processing)
This section details the lowering process for the ONNX graph responsible for handling the first input token.
2.1 Quantization
After generating the ONNX graph from our Hugging Face config -> onnx generator script, we apply a quantization step. During this process, nodes associated with RMSNorm are explicitly excluded.
Currently, we are utilizing synthetic inputs for quantization. In the future, we plan to incorporate real datasets for this step.
from tvm.contrib.epu.chimera_job.quantize import quadric_quantize
from tvm.contrib.epu import onnx_util
import onnx
def quantize(onnx_file):
# we want to exclude qwen rms norm
quantize_result = quadric_quantize(
onnx_file,
100,
synthetic_input=True,
asymmetric_activation=False,
extra_options={"AddQDQPairToWeight": True},
# we exclude RMS Norm nodes.
exclude_nodes_by_name=[
"/decoder_layer/self_attn/Add_2",
"/decoder_layer/self_attn/Add",
"/decoder_layer/input_layernorm/Pow",
"/decoder_layer/input_layernorm/ReduceMean",
"/decoder_layer/input_layernorm/Add",
"/decoder_layer/input_layernorm/Sqrt",
"/decoder_layer/input_layernorm/Div",
"/decoder_layer/input_layernorm/Mul",
"/decoder_layer/input_layernorm/Mul_1",
"/decoder_layer/post_attention_layernorm/Pow",
"/decoder_layer/post_attention_layernorm/ReduceMean",
"/decoder_layer/post_attention_layernorm/Add",
"/decoder_layer/post_attention_layernorm/Sqrt",
"/decoder_layer/post_attention_layernorm/Div",
"/decoder_layer/post_attention_layernorm/Mul",
"/decoder_layer/post_attention_layernorm/Mul_1",
"/decoder_layer/Add",
"/decoder_layer/Add_1",
],
)
return quantize_result.qmodel_path
model_path = "../qwen_decoder.onnx"
converted_model_path = "qwen_decoder_converted.onnx"
quantized_model_path = quantize(model_path)
2026-06-19 03:53 - INFO - epu - quantize - Generating synthetic data
2026-06-19 03:53 - INFO - epu - quantize - Optimized model to opset
2026-06-19 03:53 - INFO - epu - quantize - Saved optimized model to qwen_decoder_float32_opt.onnx
2026-06-19 03:53 - INFO - epu - quantize - Input shapes: [1, 16, 1536]. Input names: hidden_states
2026-06-19 03:53 - INFO - epu - quantize - Output shapes: [[1, 16, 1536]]. Output names: ['output_hidden_states']
2026-06-19 03:53 - DEBUG - epu - quantize - Full exclusion set for quantization: ['Softmax', 'Sigmoid', 'QuadricCustomOp']
2026-06-19 03:53 - DEBUG - epu - quantize - excl_nodes ['/decoder_layer/self_attn/Softmax', '/decoder_layer/mlp/act_fn/Sigmoid', '/decoder_layer/self_attn/Add_2', '/decoder_layer/self_attn/Add', '/decoder_layer/input_layernorm/Pow', '/decoder_layer/input_layernorm/ReduceMean', '/decoder_layer/input_layernorm/Add', '/decoder_layer/input_layernorm/Sqrt', '/decoder_layer/input_layernorm/Div', '/decoder_layer/input_layernorm/Mul', '/decoder_layer/input_layernorm/Mul_1', '/decoder_layer/post_attention_layernorm/Pow', '/decoder_layer/post_attention_layernorm/ReduceMean', '/decoder_layer/post_attention_layernorm/Add', '/decoder_layer/post_attention_layernorm/Sqrt', '/decoder_layer/post_attention_layernorm/Div', '/decoder_layer/post_attention_layernorm/Mul', '/decoder_layer/post_attention_layernorm/Mul_1', '/decoder_layer/Add', '/decoder_layer/Add_1', '/decoder_layer/mlp/act_fn/Sigmoid', '/decoder_layer/mlp/act_fn/Mul']
2026-06-19 03:53 - INFO - epu - quantize - Quantization started...
WARNING:root:Please use QuantFormat.QDQ for activation type QInt8 and weight type QInt8. Or it will lead to bad performance on x64.
WARNING:root:Inference failed or unsupported type to quantize for tensor '/decoder_layer/self_attn/ConstantOfShape_output_0', type is tensor_type {
elem_type: 7
shape {
dim {
dim_value: 5
}
}
}
.
2026-06-19 03:53 - INFO - epu - quantize - Quantization done succesfully!
2026-06-19 03:53 - INFO - epu - quantize - ONNX full precision model size: 178.53 MB
2026-06-19 03:53 - INFO - epu - quantize - ONNX quantized model size: 44.67 MB
2026-06-19 03:53 - INFO - epu - quantize - Saved quantized model to /quadric/sdk-cli/examples/models/qwen/components/qwen_decoder_opt_sym_int8_q.onnx
2026-06-19 03:53 - INFO - epu - quantize - Saved shape inferenced model to /quadric/sdk-cli/examples/models/qwen/components/qwen_decoder_opt_sym_int8_q.onnx
2026-06-19 03:53 - INFO - epu - quantize - Checking for remaining FLOAT/FLOAT16 types.
2026-06-19 03:53 - INFO - epu - quantize - Model still has FLOAT/FLOAT16 types. Creating ranges for floating point tensors using calibration data
2026-06-19 03:53 - INFO - epu - quantize - Saved tensor ranges to /quadric/sdk-cli/examples/models/qwen/components/qwen_decoder_opt_sym_int8_q.onnx.tranges
2.2 Custom op Replacement for Grouped Query Attention (GQA)
Following quantization, the next step involves replacing the standard Grouped Query Attention (GQA) nodes within the ONNX graph with a dedicated custom operator. We employ a custom operator matcher designed to identify GQA based on node patterns in the graph.
from IPython.display import Code
from gqa import gqa_pattern
%psource gqa_pattern
from gqa import gqa_replacer
custom_op_model_path = "qwen_decoder_custom_op.onnx"
gqa_replacer(
input_model_path=quantized_model_path,
output_model_path=custom_op_model_path,
num_heads=2,
embed_dim=1536,
)
preproc_const_attention, Pattern Matched!
The source code for the attention custom op can be found in the gqa.hpp file:
template <std::int32_t embedDim, std::int32_t numHeads, std::int32_t seqLength,
bool scaleQOnly = false, typename OcmTokenShape,
typename OcmBufferShape, typename OcmOutShape,
FracRepType matMulQKScaleFracBits, FracRepType mulScaleFracBits,
FracRepType softmaxQKScaleXFracBits,
FracRepType softmaxQKScaleYFracBits,
FracRepType matMulQKVScaleFracBits, typename OcmAllocatorType>
INLINE void multiheadAttention(
OcmTokenShape &ocmQBuffer, OcmBufferShape &ocmKBuffer,
OcmBufferShape &ocmVBuffer,
const FixedPoint32<matMulQKScaleFracBits> &matmul_qk_scale,
const FixedPoint32<mulScaleFracBits> &mul_scale, const std::int8_t &mul_b,
const FixedPoint32<softmaxQKScaleXFracBits> &softmax_qk_scale_x,
const FixedPoint32<softmaxQKScaleYFracBits> &softmax_qk_scale_y,
const FixedPoint32<matMulQKVScaleFracBits> &matmul_qkv_scale,
OcmOutShape &ocmOut, OcmAllocatorType &ocmMemAlloc, DdrPtrTy ext_temp_ptr) {
2.3 Compilation with CGC and Execution via ISS
In the final step for the first graph, the ONNX model (now incorporating custom operators) is compiled using the Chimera Graph Compiler (CGC). We then verify its successful compilation and validate its performance using (ISS).
The following code demonstrates this graph lowering and validation process.
from tvm.contrib.epu.chimera_job.chimera_job import ChimeraJob
from tvm.contrib.epu.chimera_job.hw_config import HWConfig
trange_path = f"{quantized_model_path}.tranges"
hw_config = HWConfig(product="QC-N")
cgc_job = ChimeraJob(
custom_op_model_path,
hw_config=hw_config,
trange_file=trange_path,
custom_ops="gqa.hpp",
)
cgc_job.compile()
cgc_job.run_inference_harness(compare_ort=True)
cgc_job.plot_run_statistics()
print(cgc_job)
2026-06-19 03:53 - INFO - epu - chimera_job - START==================================onnx_ingest
2026-06-19 03:53 - INFO - epu - chimera_job - Numerical ranges provided
2026-06-19 03:53 - INFO - epu - codegen - START===============================optimize_relay
2026-06-19 03:53 - INFO - epu - codegen - START====================quantize_to_cpu_runnable_fx
2026-06-19 03:53 - INFO - epu - fx -
Source name Op Output 0 Range Output 0 Frac Bits
---------------------------------------------------------------- ---------------------- --------------------- --------------------
/decoder_layer/input_layernorm/Mul_1 contrib.epu.rms_norm [-4.50209f, 3.39198f] 28
/decoder_layer/self_attn/o_proj/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-11.0189f, 9.63068f] 27
/decoder_layer/Add add [-10.7892f, 9.34146f] 27
/decoder_layer/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-2.11395f, 2.30017f] 29
/decoder_layer/mlp/down_proj/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-7.94773f, 7.44709f] 27
/decoder_layer/Add_1 add [-15.2186f, 13.4181f] 26
2026-06-19 03:53 - INFO - epu - codegen - START====================build_cpu_runnable_fx_relay
2026-06-19 03:53 - INFO - epu - codegen - START=======================quantize_to_chimera_fx
2026-06-19 03:53 - INFO - epu - codegen - START=================================relay_to_tir
2026-06-19 03:53 - INFO - epu - codegen - START===========================relay_to_epu_relay
2026-06-19 03:53 - INFO - epu - codegen - START==============================adapt_and_order
2026-06-19 03:53 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-06-19 03:53 - INFO - epu - codegen - START=============================plan_lrm_virtual
2026-06-19 03:53 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-06-19 03:53 - INFO - epu - codegen - START===============================lrm_alloc_loop
2026-06-19 03:54 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-06-19 03:54 - INFO - epu - codegen - START================================lrm_splitting
2026-06-19 03:54 - INFO - epu - codegen - START==============================ext_split_relay
2026-06-19 03:54 - INFO - epu - codegen - START====================================build_tir
2026-06-19 03:55 - INFO - epu - chimera_job - Compilation of qwen_decoder_custom_op_QC_N_1d7_16MB_4kB_128GBps_128GBps_16_OFF_x1_x1 successful
2026-06-19 03:55 - INFO - epu - iss_testing - Found tranges for input: <tvm.contrib.epu.interval.Interval object at 0x7d85e05c48e0>
FILM 22/22: 100%|███████████████████████████████████████████████████| 22/22 [01:03<00:00, 2.88s/it]
2026-06-19 03:56 - INFO - epu - iss_testing - Found tranges for input: <tvm.contrib.epu.interval.Interval object at 0x7d85e05c48e0>
2026-06-19 03:56 - INFO - epu - iss_testing - Started Executing Onnxruntime...
2026-06-19 03:56 - INFO - epu - iss_testing - Done 0:00:00.116819
2026-06-19 03:56 - INFO - epu - chimera_job - Combined plots generated and saved to:
/quadric/sdk-cli/examples/models/qwen/components/ccl_build/qwen_decoder_custom_op_QC_N_1d7_16MB_4kB_128GBps_128GBps_16_OFF_x1_x1/run/20260619_035504_0142ad/data/qwen_decoder_custom_op_QC_N_1d7_16MB_4kB_128GBps_128GBps_16_OFF_x1_x1.combined.png
╒═════════════════════╤═══════════════════════════════════════════════════════════════════════╕
│ Module Name │ qwen_decoder_custom_op_QC_N_1d7_16MB_4kB_128GBps_128GBps_16_OFF_x1_x1 │
├─────────────────────┼───────────────────────────────────────────────────────────────────────┤
│ ONNX File │ qwen_decoder_custom_op.onnx │
├─────────────────────┼───────────────────────────────────────────────────────────────────────┤
│ Custom Ops │ /quadric/sdk-cli/examples/models/qwen/components/gqa.hpp │
├─────────────────────┼───────────────────────────────────────────────────────────────────────┤
│ Product Target │ QC-N │
├─────────────────────┼───────────────────────────────────────────────────────────────────────┤
│ Number of Cores │ 1 │
├─────────────────────┼───────────────────────────────────────────────────────────────────────┤
│ ISS Clock Frequency │ 1.700 │
├─────────────────────┼───────────────────────────────────────────────────────────────────────┤
│ L2M Size │ 16MB │
├─────────────────────┼───────────────────────────────────────────────────────────────────────┤
│ LRM Size │ 4kB │
├─────────────────────┼───────────────────────────────────────────────────────────────────────┤
│ External Read BW │ 128GBps │
├─────────────────────┼───────────────────────────────────────────────────────────────────────┤
│ External Write BW │ 128GBps │
├─────────────────────┼───────────────────────────────────────────────────────────────────────┤
│ MACS per PE │ 16 │
├─────────────────────┼───────────────────────────────────────────────────────────────────────┤
│ Max L2M │ 14.293MB │
├─────────────────────┼───────────────────────────────────────────────────────────────────────┤
│ Max LRM │ 0.252kB │
├─────────────────────┼───────────────────────────────────────────────────────────────────────┤
│ Max Temp Ext Bytes │ 0.000MB │
├─────────────────────┼───────────────────────────────────────────────────────────────────────┤
│ Network GMACs │ 0.749 │
╘═════════════════════╧═══════════════════════════════════════════════════════════════════════╛
╒════╤════════╤══════════════════════╤═══════════════╤══════════════════════════╤═══════╕
│ │ Type │ Name │ shape │ type │ mse │
╞════╪════════╪══════════════════════╪═══════════════╪══════════════════════════╪═══════╡
│ 0 │ Input │ hidden_states │ [1, 16, 1536] │ tensor[FixedPoint32<30>] │ n/a │
├────┼────────┼──────────────────────┼───────────────┼──────────────────────────┼───────┤
│ 1 │ Output │ output_hidden_states │ [1, 16, 1536] │ tensor[FixedPoint32<26>] │ 0.283 │
╘════╧════════╧══════════════════════╧═══════════════╧══════════════════════════╧═══════╛
Post-ISS Report 1.7 GHz ***
Fully placed-and-routed gate simulation:
╒══════════════════════════════════╤═════════╕
│ Latency (ms) │ 6.90 │
├──────────────────────────────────┼─────────┤
│ FPS │ 144.99 │
├──────────────────────────────────┼─────────┤
│ Average Power @ 3nm SSGNP (mW) │ 120.08 │
├──────────────────────────────────┼─────────┤
│ FPS per Watt @ 3nm SSGNP (FPS/W) │ 1207.48 │
├──────────────────────────────────┼─────────┤
│ Ext Rd Bytes (MB) │ 44.83 │
├──────────────────────────────────┼─────────┤
│ Ext Wr Bytes (MB) │ 0.10 │
├──────────────────────────────────┼─────────┤
│ Avg Ext Rd BW (GBps) │ 6.35 │
├──────────────────────────────────┼─────────┤
│ Avg Ext Wr BW (GBps) │ 0.01 │
├──────────────────────────────────┼─────────┤
│ MAC Utilization │ 6.24% │
╘══════════════════════════════════╧═════════╛
*** Data generated using 7nm SSGNP gatesim and scaled to 3nm
[SDK-CLI] : TotalCycles: 11,724,620
[SDK-CLI] : Executions/second: 144.99
compute : ▇▇▇▇ 806.588K
data_array : ▇▇ 392.569K
mac : ▇▇▇▇ 793.728K
data_external: ▇▇▇▇▇▇ 1.065M
data_ocm : ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 8.648M
for more information check run directory: /quadric/sdk-cli/examples/models/qwen/components/ccl_build/qwen_decoder_custom_op_QC_N_1d7_16MB_4kB_128GBps_128GBps_16_OFF_x1_x1/run/20260619_035504_0142ad
3 Lowering the Second Graph
The second ONNX graph, designed for the autoregressive generation steps (i.e., for the second token onwards), is obtained using a methodology similar to that of the first graph. This graph which includes one decoder and the final language modeling head of the network is also compiled using the Chimera Graph Compiler (CGC).
We are currently developing the autoregressive runner, which will enable full end-to-end validation of this second graph in the near future. The code below demonstrates its compilation.
single_tokmodel_path = "../qwen_decoder_single_tok.onnx"
cgc_job = ChimeraJob(
single_tokmodel_path,
trange_file=f"{single_tokmodel_path}.tranges",
custom_ops="gqa.hpp",
)
cgc_job.compile()
cgc_job.run_inference_harness()
cgc_job.plot_run_statistics()
print(cgc_job)
2026-06-19 03:56 - INFO - epu - chimera_job - START==================================onnx_ingest
2026-06-19 03:56 - INFO - epu - chimera_job - Numerical ranges provided
2026-06-19 03:56 - INFO - epu - codegen - START===============================optimize_relay
2026-06-19 03:56 - INFO - epu - codegen - START====================quantize_to_cpu_runnable_fx
2026-06-19 03:56 - INFO - epu - fx -
Source name Op Output 0 Range Output 0 Frac Bits
----------------------------------------------------------------- ---------------------- ----------------------- --------------------
/model/layers.0/input_layernorm/Mul_1 contrib.epu.rms_norm [-3.16465f, 2.58947f] 29
/model/layers.0/self_attn/o_proj/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-3.24962f, 3.22423f] 29
/model/layers.0/Add add [-3.29088f, 3.25915f] 29
/model/layers.0/post_attention_layernorm/Mul_1 contrib.epu.rms_norm [-1.66584f, 3.74452f] 28
/model/layers.0/mlp/down_proj/MatMul_output_0_DequantizeLinear contrib.epu.dequantize [-0.982546f, 0.655031f] 29
/model/layers.0/Add_1 add [-3.56668f, 3.15572f] 28
/model/norm/Mul_1 contrib.epu.rms_norm [-26.5547f, 61.243f] 25
logits_DequantizeLinear contrib.epu.dequantize [-17.3244f, 15.4295f] 26
2026-06-19 03:56 - INFO - epu - codegen - START====================build_cpu_runnable_fx_relay
2026-06-19 03:56 - INFO - epu - codegen - START=======================quantize_to_chimera_fx
2026-06-19 03:56 - INFO - epu - codegen - START=================================relay_to_tir
2026-06-19 03:56 - INFO - epu - codegen - START===========================relay_to_epu_relay
2026-06-19 03:56 - INFO - epu - codegen - START==============================adapt_and_order
2026-06-19 03:56 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-06-19 03:56 - INFO - epu - codegen - START=============================plan_lrm_virtual
2026-06-19 03:56 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-06-19 03:56 - INFO - epu - codegen - START===============================lrm_alloc_loop
2026-06-19 03:56 - INFO - epu - codegen - START==============================amend_ctrl_flow
2026-06-19 03:56 - INFO - epu - codegen - START================================lrm_splitting
2026-06-19 03:56 - INFO - epu - codegen - START==============================ext_split_relay
2026-06-19 03:57 - INFO - epu - codegen - START====================================build_tir
2026-06-19 03:57 - INFO - epu - chimera_job - Compilation of qwen_decoder_single_tok_QC_U_1d7_16MB_4kB_128GBps_128GBps_16_OFF_x1_x1 successful
2026-06-19 03:57 - INFO - epu - iss_testing - No tranges found for input, use default float range: <tvm.contrib.epu.interval.Interval object at 0x7d85c56f93f0>
2026-06-19 03:57 - INFO - epu - iss_testing - No tranges found for input, use default float range: <tvm.contrib.epu.interval.Interval object at 0x7d85c56f93f0>
2026-06-19 03:57 - INFO - epu - iss_testing - Found tranges for input: <tvm.contrib.epu.interval.Interval object at 0x7d85c56f8100>
2026-06-19 03:57 - INFO - epu - iss_testing - Found tranges for input: <tvm.contrib.epu.interval.Interval object at 0x7d85c56fa170>
FILM 19/19: 100%|███████████████████████████████████████████████████| 19/19 [00:44<00:00, 2.33s/it]
2026-06-19 03:58 - INFO - epu - iss_testing - No tranges found for input, use default float range: <tvm.contrib.epu.interval.Interval object at 0x7d85c56f8fd0>
2026-06-19 03:58 - INFO - epu - iss_testing - No tranges found for input, use default float range: <tvm.contrib.epu.interval.Interval object at 0x7d85c56f8fd0>
2026-06-19 03:58 - INFO - epu - iss_testing - Found tranges for input: <tvm.contrib.epu.interval.Interval object at 0x7d85c56fbb20>
2026-06-19 03:58 - INFO - epu - iss_testing - Found tranges for input: <tvm.contrib.epu.interval.Interval object at 0x7d85c5672020>
2026-06-19 03:58 - INFO - epu - iss_testing - Started Executing Onnxruntime...
2026-06-19 03:58 - INFO - epu - iss_testing - Done 0:00:00.507911
2026-06-19 03:58 - INFO - epu - chimera_job - Combined plots generated and saved to:
/quadric/sdk-cli/examples/models/qwen/components/ccl_build/qwen_decoder_single_tok_QC_U_1d7_16MB_4kB_128GBps_128GBps_16_OFF_x1_x1/run/20260619_035721_e9802f/data/qwen_decoder_single_tok_QC_U_1d7_16MB_4kB_128GBps_128GBps_16_OFF_x1_x1.combined.png
╒═════════════════════╤════════════════════════════════════════════════════════════════════════╕
│ Module Name │ qwen_decoder_single_tok_QC_U_1d7_16MB_4kB_128GBps_128GBps_16_OFF_x1_x1 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────┤
│ ONNX File │ ../qwen_decoder_single_tok.onnx │
├─────────────────────┼────────────────────────────────────────────────────────────────────────┤
│ Custom Ops │ /quadric/sdk-cli/examples/models/qwen/components/gqa.hpp │
├─────────────────────┼────────────────────────────────────────────────────────────────────────┤
│ Product Target │ QC-U │
├─────────────────────┼────────────────────────────────────────────────────────────────────────┤
│ Number of Cores │ 1 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────┤
│ ISS Clock Frequency │ 1.700 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────┤
│ L2M Size │ 16MB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────┤
│ LRM Size │ 4kB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────┤
│ External Read BW │ 128GBps │
├─────────────────────┼────────────────────────────────────────────────────────────────────────┤
│ External Write BW │ 128GBps │
├─────────────────────┼────────────────────────────────────────────────────────────────────────┤
│ MACS per PE │ 16 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────┤
│ Max L2M │ 13.536MB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────┤
│ Max LRM │ 0.957kB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────┤
│ Max Temp Ext Bytes │ 0.000MB │
├─────────────────────┼────────────────────────────────────────────────────────────────────────┤
│ Network GMACs │ │
╘═════════════════════╧════════════════════════════════════════════════════════════════════════╛
╒════╤════════╤═════════════════════════════════════════╤═════════════════╤══════════════════════════╤═══════╕
│ │ Type │ Name │ shape │ type │ mse │
╞════╪════════╪═════════════════════════════════════════╪═════════════════╪══════════════════════════╪═══════╡
│ 0 │ Input │ past_key_values.0.value_quantized │ [1, 2, 15, 128] │ tensor[int8] │ n/a │
├────┼────────┼─────────────────────────────────────────┼─────────────────┼──────────────────────────┼───────┤
│ 1 │ Input │ past_key_values.0.key_quantized │ [1, 2, 15, 128] │ tensor[int8] │ n/a │
├────┼────────┼─────────────────────────────────────────┼─────────────────┼──────────────────────────┼───────┤
│ 2 │ Input │ /model/layers.0/self_attn/Cast_output_0 │ [1, 16] │ tensor[FixedPoint32<31>] │ n/a │
├────┼────────┼─────────────────────────────────────────┼─────────────────┼──────────────────────────┼───────┤
│ 3 │ Input │ /model/embed_tokens/Gather_output_0 │ [1, 1, 1536] │ tensor[FixedPoint32<31>] │ n/a │
├────┼────────┼─────────────────────────────────────────┼─────────────────┼──────────────────────────┼───────┤
│ 4 │ Output │ logits │ [1, 1, 151936] │ tensor[FixedPoint32<26>] │ 6.913 │
╘════╧════════╧═════════════════════════════════════════╧═════════════════╧══════════════════════════╧═══════╛
Post-ISS Report 1.7 GHz ***
Fully placed-and-routed gate simulation:
╒══════════════════════════════════╤════════╕
│ Latency (ms) │ 2.56 │
├──────────────────────────────────┼────────┤
│ FPS │ 391.26 │
├──────────────────────────────────┼────────┤
│ Average Power @ 3nm SSGNP (mW) │ 712.28 │
├──────────────────────────────────┼────────┤
│ FPS per Watt @ 3nm SSGNP (FPS/W) │ 549.31 │
├──────────────────────────────────┼────────┤
│ Ext Rd Bytes (MB) │ 277.03 │
├──────────────────────────────────┼────────┤
│ Ext Wr Bytes (MB) │ 0.58 │
├──────────────────────────────────┼────────┤
│ Avg Ext Rd BW (GBps) │ 105.85 │
├──────────────────────────────────┼────────┤
│ Avg Ext Wr BW (GBps) │ 0.22 │
╘══════════════════════════════════╧════════╛
*** Data generated using 7nm SSGNP gatesim and scaled to 3nm
[SDK-CLI] : TotalCycles: 4,344,931
[SDK-CLI] : Executions/second: 391.26
compute : ▇ 73.632K
data_array : ▏ 15.116K
mac : ▏ 16.938K
data_external: ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 1.572M
data_ocm : ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 2.667M
for more information check run directory: /quadric/sdk-cli/examples/models/qwen/components/ccl_build/qwen_decoder_single_tok_QC_U_1d7_16MB_4kB_128GBps_128GBps_16_OFF_x1_x1/run/20260619_035721_e9802f
