Unlock this feature

This feature isn’t part of your plan yet

Contact sales to get upgraded to the full DevStudio experience.

Unlock this feature

This feature isn't part of your plan yet.

Model Demo: Tiny Llama (stories15M) Text Generation


NOTE: The Jupyter Notebook below is included in the Chimera SDK and can be run interactively by running the following CLI command:

$ quadric sdk notebook

From the Jupyter Notebook window in your browser, select the notebook named /quadric/sdk-cli/examples/models/tiny_llama/tiny_llama.ipynb.


Tiny Llama (stories15M) Text Generation on Chimera GPNPU

This notebook takes stories15M, a 15M-parameter Llama-2 trained on TinyStories, from its PyTorch checkpoint to autoregressive text generation on the Chimera GPNPU: quantize it with quadric_pyquant, compose a decode step from CCL (Chimera Compute Language) kernels with ChiPy, compile it with CGC (Chimera Graph Compiler), and generate tokens on the ISS (Instruction Set Simulator) with a KV cache.

Why the PyTorch path?

The model never leaves PyTorch. The same torch.nn.Module code is the float reference, the quantization target for quadric_pyquant, and the CPU oracle of every GPNPU kernel: each ChiPy op is a module whose forward() computes exactly what its CCL kernel computes. The recipe carries straight over to larger LLMs, VLMs and VLAs, where an ONNX export is a model of its own to maintain.

Why a KV-cache decode step?

A generated token only needs the new token's projections plus attention over what came before. Caching every layer's keys and values makes each step a single pass of matrix-vector products (GEMVs) over the weights, instead of re-running the whole context. The GEMVs map one output column per PE: each PE holds a weight column in its local memory and the activation vector is broadcast once, so the whole PE array works on a one-row product.

Pipeline

Tiny Llama Pipeline

Model: stories15M (Llama-2 architecture: 6 layers, width 288, 6 heads, SwiGLU MLP 768, vocabulary 32000, 256-token context; 15M parameters; trained on TinyStories)

License Notice

The Llama 2 reference model code and the Llama 2 tokenizer this example uses are Llama Materials under the LLAMA 2 Community License Agreement, Copyright (c) Meta Platforms, Inc. and affiliates: https://github.com/meta-llama/llama/blob/main/LICENSE Use must comply with the Llama 2 Acceptable Use Policy, which section 1(b)(iv) of the Agreement incorporates: https://ai.meta.com/llama/use-policy Section 1(b)(v) forbids using the Llama Materials, or any output or results of the Llama Materials, to improve any other large language model (excluding Llama 2 or derivative works thereof). Section 2 requires a separate licence from Meta for licensees whose products exceeded 700 million monthly active users on the Llama 2 release date. Section 1(b) sets the conditions for distributing the Llama Materials: a copy of the Agreement and the required "Notice" text file.


1. Setup

The helpers hold the downloads, the PyTorch models and the ISS drivers; tiny_llama_chipy holds the ChiPy ops and the decode step. The GPNPU target is the default HWConfig: QC-U with 16 MB of OCM (On-Chip Memory), 16 MACs per PE, one core at 1.7 GHz and 128 GB/s of DDR bandwidth.

from pathlib import Path

import numpy as np
import torch

import tiny_llama_chipy as tl_chipy
import tiny_llama_helpers as tl

%matplotlib inline

PROMPT = "Once upon a time"
NEW_TOKENS = 24  # tokens generated on the ISS in section 8
EVAL_WINDOWS = 16  # held-out windows scored on the ISS in section 9
torch.manual_seed(0)
<torch._C.Generator at 0x7fc94a57dfb0>

2. Download the Model

Three files come from the public sdk-cli-models bucket: the llama2.c stories15M.pt checkpoint, the Llama-2 SentencePiece tokenizer, and tokenized TinyStories windows of 257 tokens (32 for calibration, 64 held out for evaluation). The checkpoint is from llama2.c (MIT); the tokenizer, like the model code in llama2c_model.py, is Meta's, under the LLAMA 2 Community License Agreement; the windows are tokenized from TinyStories (CDLA-Sharing-1.0). The bucket serves each file with its licence notice beside it.

assets = tl.download_assets()
calib, val = tl.load_windows(assets)
tokenizer = tl.load_tokenizer(assets)
original = tl.load_original(assets)

n_params = sum(p.numel() for p in original.parameters())
print(f"stories15M: {n_params / 1e6:.1f}M parameters, vocabulary {tokenizer.vocab_size()}")
print(
    f"TinyStories windows: {len(calib)} calibration + {len(val)} held-out, {val.shape[1]} tokens each"
)
stories15M: 15.2M parameters, vocabulary 32000
TinyStories windows: 32 calibration + 64 held-out, 257 tokens each

3. A Deployable PyTorch Twin

Before quantizing, we rewrite the model into the form the GPNPU computes. Every rewrite is exact in float, and the cell below proves it:

RewriteWhy
Fold each RMSNorm gamma into the Linear that consumes it (wq/wk/wv, w1/w3, lm_head)the device only needs an unweighted norm
Permute wq/wk rows per head from interleaved RoPE pairs to half-split orderlinalg::RoPE is half-split; q·k is unchanged when q and k share the permutation
Untie the lm_head from the embeddingit absorbs the final-norm gamma
Expose attention's matmuls, softmax and the SwiGLU product as modulesquadric_pyquant quantizes modules

The twin also has a step(tok, pos, cache) method: decoding one token at a time with a KV cache must reproduce the full-sequence forward.

twin = tl.DeployableLlama.from_original(original)
full_err, step_err = tl.check_twin(original, twin, val)
print(f"max |original - twin| logits:        {full_err:.2e}")
print(f"max |full forward - KV-cache decode|: {step_err:.2e}")
assert full_err < 1e-3, "the twin must match the original model"
assert step_err < 1e-3, "KV-cache decoding must match the full forward"

fp32_ppl, fp32_top1 = tl.perplexity(twin, val)
print(
    f"fp32 perplexity {fp32_ppl:.3f}, next-token top-1 {fp32_top1:.1%} (64 windows x 256 positions)"
)
max |original - twin| logits:        3.62e-05
max |full forward - KV-cache decode|: 2.96e-05
fp32 perplexity 2.833, next-token top-1 71.6% (64 windows x 256 positions)

4. Quantize with quadric_pyquant

The device datapath is W8A8, symmetric and per-tensor: int8 weights, int8 activations at every GEMV input, and a FixedPoint32 residual stream. quantize_twin follows the standard flow (tag_all_modules → generate_quant_config → quantize_model → run_calibration → freeze_model_weights) with one addition found by ablation: the input of each MLP down projection, silu(w1·h) · (w3·h), carries outlier channels. An exact SmoothQuant fold moves them into the weights: each channel is divided by s in w3's rows and multiplied by s in w2's columns (s = amax(act)^0.7 / amax(w2 col)^0.3), which leaves the float model unchanged.

qmodel = tl.quantize_twin(tl.DeployableLlama.from_original(original), calib)
int8_ppl, int8_top1 = tl.perplexity(qmodel, val)

print(f"{'':22s}{'perplexity':>12s}{'top-1':>9s}")
print(f"{'fp32':22s}{fp32_ppl:12.3f}{fp32_top1:9.1%}")
print(f"{'W8A8 (quadric_pyquant)':22s}{int8_ppl:12.3f}{int8_top1:9.1%}")
assert int8_ppl < 1.1 * fp32_ppl, "W8A8 perplexity should stay within 10% of fp32"
                        perplexity    top-1
fp32                         2.833    71.6%
W8A8 (quadric_pyquant)       2.953    70.5%

5. The CCL Kernels

tiny_llama_ops.hpp implements the decode step as six CCL kernels. Each decoder layer runs five of them, and each fuses the elementwise work into its GEMV epilogue so no intermediate vector makes an extra trip to DDR:

KernelComputesOutput
embedone row of the FixedPoint32 embedding table (cgc::embedding)FX32
rmsQuantx / rms(x) → int8 (first layer's input)int8
linearTfused Q|K|V GEMV; the 288 × 32000 lm_head in 16 double-buffered DMA chunksFX32
attentionQKVV → int8, half-split RoPE on Q/K → int8, KV-cache append at pos, multi-head attention over positions 0..posint8
linearTResRmsWo or W2 GEMV + residual add + RMSNormFX32 residual, int8
gateUpSiluW1|W3 GEMV with SiLU(gate) · up in the epilogueint8

The GEMV core gives each PE one output column: the weights are packed as v4i8 words (four consecutive K values of one column per word), one read flow hands each PE its whole column, the activation vector is broadcast, and a single fullyConnectedTileBlock MAC produces the dot product. Here is that loop inside linearT:

header = Path("tiny_llama_ops.hpp").read_text()
start = header.index("  for(std::int32_t c = 0; c < numChunks; c++) {")
print(header[start : header.index("    constexpr std::int32_t lastCols", start)])
  for(std::int32_t c = 0; c < numChunks; c++) {
    if(c + 1 < numChunks) {  // prefetch the next chunk into the other buffer
      auto next = chunk(c + 1);
      memCpy(next, ocmB[(c + 1) & 1]);
    }
    {
      typename G::ReadB  readB(ocmB[c & 1]);
      typename G::WriteC writeC(ocmC);
      alignas(core_array::numMacsPerCore) typename G::ReadB::MinRoiArray qB;
      typename G::WriteC::MinRoiArray qC;
      typename G::Bcast bcast(ocmA);  // x is broadcast to every PE once per chunk
      bcast.stageMROI();
      for(std::int32_t t = 0; t < G::tiles; t++) {
        readB.read(qB);  // each PE receives the K values of its own output column
        qC[t] = dequant<combinedFrac, OutElem::fractionalBits>(
          nn::fullyConnectedTileBlock<std::int32_t, G::colsOfA>(qB), comb);
      }
      writeC.write(qC);
    }

6. The ChiPy Decode Step

Each kernel has a ChiPy twin: a torch.nn.Module decorated with @chipy.torch_ccl_custom_op whose forward() reproduces the kernel's arithmetic (int8 codes, integer accumulation, the device's round-half-up). build_decode_step composes one token's worth of them in a @chipy.func: embed, six layers of five ops, and the LM head. The token id, position, RoPE row and KV buffer are runtime inputs, so one compiled program serves every position; the weights are baked in as constants. Here is the attention op's contract:

source = Path("tiny_llama_chipy.py").read_text()
start = source.index('@chipy.torch_ccl_custom_op(\n    ccl_func_name="tiny_llama::attentionQKV"')
print(source[start : source.index("        pos = _index(pos, MAXSEQ)", start)])
@chipy.torch_ccl_custom_op(
    ccl_func_name="tiny_llama::attentionQKV",
    reserved_l2m=0,
    io_in_ext_mem=True,
    read_only=(
        "qkv", "sin", "cos", "kv", "pos", "layer_off",
        "q_inv", "k_inv", "v_inv", "comb", "sm_y", "qkv_scale",
    ),
)  # fmt: skip
class AttentionOp(torch.nn.Module):
    """Decode attention over this layer's region of the shared KV buffer."""

    _int8_output = True
    _frac_bits = (0,)

    def __init__(self, layer):
        """Bind the op to one layer's region of the KV buffer."""
        super().__init__()
        self.layer = layer
        self.qk_cal = calibration_point(f"L{layer}.qk")

    def template_param_factory(self):
        """Return <numHeads, seqLen, qkFracBits>."""
        return (NH, MAXSEQ, shared_frac(self.qk_cal))

    def forward(
        self, qkv, sin, cos, kv, pos, layer_off, q_inv, k_inv, v_inv, comb, sm_y, qkv_scale
    ):
        """RoPE, append K/V at ``pos``, attend over positions 0..pos, requantize to int8."""

Every FixedPoint32 value in the chain gets its fractional bits from a CalibrationPoint, recorded while the chain decodes real text on the host CPU. The fractional bits are shared across the six layers of each op, so all layers reuse one compiled instantiation of each kernel.

params = tl_chipy.extract_params(qmodel)
decode_step = tl_chipy.build_decode_step(params)
n_ops = tl_chipy.calibrate(decode_step, calib[:2])
print(f"Decode step: {len(decode_step.ops)} CCL custom ops, {n_ops} calibrated on 2 x 128 tokens")
Decode step: 33 CCL custom ops, 31 calibrated on 2 x 128 tokens

7. Compile with CGC

CGC compiles the ChiPy program together with tiny_llama_ops.hpp. In the generated C++, every layer's attention call receives the same KV buffer and its own byte offset into it; the attention kernel writes the new token's K and V at pos and attends over the cached positions:

decode_build = tl.compile_decode_step(decode_step, "tiny_llama_decode")
print(f"Compiled in {decode_build['compile_s']:.0f} s")

generated = (tl.build_dir("tiny_llama_decode").parent / "tiny_llama_decode.cpp").read_text()
for line in generated.splitlines():
    if "tiny_llama::attentionQKV" in line:
        print(line.strip()[:120] + " ...")
Compiled in 52 s
tiny_llama::attentionQKV<6,256,25>(ext_tensor_2, input_sin, input_cos, input_kv, input_pos, 0, FixedPoint32<27>(1.083232 ...
tiny_llama::attentionQKV<6,256,25>(ext_tensor_9, input_sin, input_cos, input_kv, input_pos, 147456, FixedPoint32<27>(1.3 ...
tiny_llama::attentionQKV<6,256,25>(ext_tensor_16, input_sin, input_cos, input_kv, input_pos, 294912, FixedPoint32<27>(1. ...
tiny_llama::attentionQKV<6,256,25>(ext_tensor_23, input_sin, input_cos, input_kv, input_pos, 442368, FixedPoint32<27>(1. ...
tiny_llama::attentionQKV<6,256,25>(ext_tensor_30, input_sin, input_cos, input_kv, input_pos, 589824, FixedPoint32<27>(1. ...
tiny_llama::attentionQKV<6,256,25>(ext_tensor_37, input_sin, input_cos, input_kv, input_pos, 737280, FixedPoint32<27>(1. ...

8. Generate Text on the ISS

Each ISS invocation executes one decode step. The host passes in the token and its position, takes the greedy argmax of the returned logits as the next token, and carries the KV buffer from one step to the next. The prompt is consumed one token per step, then the model generates.

At every step the ChiPy chain also runs on the host CPU in lockstep: same token, same position, and a copy of the same input KV buffer the ISS received. Both sides compute from identical inputs, so each step is checked on its own (section 9) rather than as a free-running sequence where one early difference would change all the text after it.

iss_tokens, iss_cycles, steps = tl.generate_on_iss(
    decode_build, decode_step, tokenizer, PROMPT, NEW_TOKENS
)
print(tokenizer.decode(iss_tokens))
print(
    f"\n{len(steps)} decode steps on the ISS, {np.mean(iss_cycles):,.0f} cycles per step on average"
)
Once upon a time, there was a little girl named Lily. She loved to play outside in the sunshine. One day,

28 decode steps on the ISS, 590,329 cycles per step on average

9. Compare Against the PyTorch Reference

The ChiPy chain on the host CPU is the oracle of every kernel; the W8A8 quadric_pyquant model is the reference the quantization was calibrated on. The gates, all asserted:

CheckReferenceGate
every generation step (lockstep, identical inputs)CPU chainlogit cosine ≥ 0.999; same argmax unless the top-2 logits are within 0.05; every K/V row the ISS wrote has cosine ≥ 0.99 with the chain's; every other byte of the returned KV buffer equals the input buffer
next token of 16 held-out windows (positions 0..254 cached by the CPU chain)CPU chainlogit cosine ≥ 0.999 and the same argmax on every window
the same 16 predictionsW8A8 PyTorch modellogit cosine ≥ 0.998 on every window, ≥ 0.9996 on average; relative L2 error ≤ 0.06 on average

The last gate catches what the first two cannot: the chain and the kernels share the scales read from the quantized model, so an error in reading those scales would pass a chain-only check. Cosine checks logit direction; relative L2 error ‖ISS − W8A8‖ / ‖W8A8‖ also checks magnitude, which cosine and argmax cannot constrain. The 6% mean bound allows numerical differences from FixedPoint32 arithmetic while rejecting an incorrectly scaled LM head.

The embedding table uses 30 fractional bits. At 24 bits, losing small embedding values can flip int8 rounding ties and amplify the difference from W8A8 through the cached prefix.

The KV buffer the ISS returns is the next step's input, and the CPU chain starts every step from it, so a step that damaged earlier positions would go unnoticed by the per-step gates. Every position other than the one being written must therefore come back byte-identical.

The K/V gate on the newly written row is a cosine, not byte equality. The kernels compute in FixedPoint32 and the chain in float32, so a value that lands within ~1e-5 of a rounding tie can round the other way on the device; one flipped int8 code early in a step moves a few codes in the later layers of that step. Most steps are byte-identical, a step with a flip still scores above 0.995, and a K/V row from the wrong token or position scores near 0.

step_worst = min(steps, key=lambda s: s["cosine"])
step_flips = [s for s in steps if s["argmax_iss"] != s["argmax_cpu"] and s["margin"] >= 0.05]
kv_cos = min(s["kv_cosine"] for s in steps)
kv_damaged = [s["pos"] for s in steps if not s["kv_preserved"]]
kv_diff, kv_equal = max(s["kv_max_diff"] for s in steps), min(s["kv_equal"] for s in steps)
print(
    f"Generation (lockstep): min cosine {step_worst['cosine']:.6f}, {len(step_flips)} argmax changes"
)
print(
    f"K/V written by the ISS: min row cosine {kv_cos:.5f}, max |diff| {kv_diff}, "
    f"{kv_equal:.2%} of bytes identical in the worst step; "
    f"history preserved at {len(steps) - len(kv_damaged)}/{len(steps)} steps"
)

windows = val[:EVAL_WINDOWS]
iss_logits, chain_logits = tl.score_last_token(decode_build, decode_step, windows)
with torch.no_grad():
    w8a8_logits = qmodel(windows[:, :-1])[:, -1].numpy()
cosines = [tl.cosine(c, d) for c, d in zip(chain_logits, iss_logits)]
chain_agree = int((iss_logits.argmax(1) == chain_logits.argmax(1)).sum())
w8a8_agree = int((iss_logits.argmax(1) == w8a8_logits.argmax(1)).sum())
w8a8_cos = [tl.cosine(q, d) for q, d in zip(w8a8_logits, iss_logits)]
w8a8_rel = np.linalg.norm(iss_logits - w8a8_logits, axis=1) / np.linalg.norm(w8a8_logits, axis=1)
print(
    f"Held-out next token vs CPU chain: min cosine {min(cosines):.6f}, argmax {chain_agree}/{EVAL_WINDOWS}"
)
print(
    f"Held-out next token vs W8A8 PyTorch: min cosine {min(w8a8_cos):.5f}, "
    f"mean {np.mean(w8a8_cos):.5f}, relative L2 error {w8a8_rel.mean():.4f} on average, "
    f"top-1 agreement {w8a8_agree}/{EVAL_WINDOWS}"
)
Generation (lockstep): min cosine 0.999944, 0 argmax changes
K/V written by the ISS: min row cosine 0.99856, max |diff| 3, 92.42% of bytes identical in the worst step; history preserved at 28/28 steps
Held-out next token vs CPU chain: min cosine 0.999901, argmax 16/16
Held-out next token vs W8A8 PyTorch: min cosine 0.99869, mean 0.99978, relative L2 error 0.0270 on average, top-1 agreement 16/16
targets = windows[:, -1].numpy()
with torch.no_grad():
    fp32_logits = twin(windows[:, :-1])[:, -1].numpy()
rows = {"fp32 PyTorch": fp32_logits, "W8A8 PyTorch": w8a8_logits, "GPNPU (ISS)": iss_logits}
print(f"{'next token of 16 windows':26s}{'perplexity':>12s}{'top-1':>8s}")
for name, logits in rows.items():
    ppl, top1 = tl.next_token_scores(logits, targets)
    print(f"{name:26s}{ppl:12.3f}{top1:8.0%}")

assert step_worst["cosine"] >= 0.999, f"ISS logits diverged from the CPU chain at {step_worst}"
assert not step_flips, f"ISS argmax differs from the CPU chain at {step_flips}"
assert kv_cos >= 0.99, f"ISS K/V writes diverged from the CPU chain (row cosine {kv_cos:.5f})"
assert not kv_damaged, f"the ISS changed cached K/V history at positions {kv_damaged}"
assert min(cosines) >= 0.999, f"held-out ISS logits diverged from the CPU chain: {cosines}"
assert (
    chain_agree == EVAL_WINDOWS
), f"held-out ISS argmax matches the CPU chain on {chain_agree}/{EVAL_WINDOWS}"
assert min(w8a8_cos) >= 0.998, f"held-out ISS logits diverged from the W8A8 model: {w8a8_cos}"
assert np.mean(w8a8_cos) >= 0.9996, f"held-out ISS vs W8A8 mean cosine {np.mean(w8a8_cos):.5f}"
assert w8a8_rel.mean() <= 0.06, f"held-out ISS vs W8A8 relative L2 error {w8a8_rel.mean():.4f}"
print("PASS: ISS decode matches the PyTorch reference")
next token of 16 windows    perplexity   top-1
fp32 PyTorch                     3.688     69%
W8A8 PyTorch                     3.599     69%
GPNPU (ISS)                      3.589     69%
PASS: ISS decode matches the PyTorch reference

10. Performance

For the profile we compile the decode step without the KV output (a deployment keeps the cache resident on the GPNPU) and wrap every kernel in an ISS profile region. The cost of a step grows with the number of cached positions, because attention reads them all, so we measure it at several positions. The weights (15 MB of int8) stream from DDR once per token: at 128 GB/s and 1.7 GHz that is a floor of about 200K cycles per token, and the lm_head alone runs close to its own floor.

decode_step.options["kv_output"] = False
perf_build = tl.compile_decode_step(decode_step, "tiny_llama_perf", profile=True)
empty_kv = torch.zeros(tl_chipy.KV_SHAPE, dtype=torch.int8)
cycles_by_pos = {
    pos: tl.run_decode_step(perf_build, 1, pos, empty_kv)[2] for pos in (1, 64, 128, 255)
}
regions = tl.region_profile("tiny_llama_perf")  # the last run: position 255

for pos, cycles in cycles_by_pos.items():
    print(f"position {pos:3d}: {cycles:9,d} cycles  ({1.7e9 / cycles:,.0f} tokens/s at 1.7 GHz)")
position   1:   554,675 cycles  (3,065 tokens/s at 1.7 GHz)
position  64:   640,567 cycles  (2,654 tokens/s at 1.7 GHz)
position 128:   725,612 cycles  (2,343 tokens/s at 1.7 GHz)
position 255:   879,893 cycles  (1,932 tokens/s at 1.7 GHz)
fig = tl.plot_profile(regions, cycles_by_pos, cycles_by_pos[255])


Summary

Modelstories15M: Llama-2 architecture, 15M parameters, 6 layers, 256-token context, TinyStories
PipelinePyTorch → deployable twin → quadric_pyquant W8A8 → ChiPy decode step of CCL kernels → CGC → ISS
TargetQC-U, 16 MB OCM, 16 MACs per PE, 1 core, 1.7 GHz, 128 GB/s
Quantizationint8 weights and activations (symmetric, per-tensor), SmoothQuant on the MLP down-projection input, FixedPoint32 residual stream
Custom opsembed, rmsQuant, linearT, attentionQKV, linearTResRms, gateUpSilu (+ kvSnapshot for host-carried caches)

Key takeaways

  1. The PyTorch model is the single source of truth. The float reference, the quantized model and the per-kernel CPU oracles are all torch.nn.Module code; the ISS output is checked against the same modules.
  2. Decode is a GEMV problem. Mapping one output column per PE, with the activation broadcast once, keeps the whole PE array busy on a one-row product and brings the 32000-wide LM head close to its DDR bandwidth floor.
  3. Fusing elementwise work into the GEMV epilogues (residual add, RMSNorm, SwiGLU) takes a decoder layer to five kernels with no intermediate DDR round trips.
  4. ChiPy composes the kernels into one program per token, with the position as a runtime input and the KV cache in DDR, so one compile serves the whole generation.

Citation

@article{eldan2023tinystories,
  title={TinyStories: How Small Can Language Models Be and Still Speak Coherent English?},
  author={Eldan, Ronen and Li, Yuanzhi},
  journal={arXiv preprint arXiv:2305.07759},
  year={2023}
}

@article{touvron2023llama2,
  title={Llama 2: Open Foundation and Fine-Tuned Chat Models},
  author={Touvron, Hugo and others},
  journal={arXiv preprint arXiv:2307.09288},
  year={2023}
}

Sign in to your account

Don't have an account? 
By signing in, you are agreeing to our Terms of Use and Privacy Policy.
Quadric // One architecture. Every algorithm.

Develop.

Simulate.

Profile.

Collaborate.