Model Demo: LFM2.5-350M Text Generation

NOTE: The Jupyter Notebook below is included in the Chimera SDK and can be run interactively by running the following CLI command:

$ quadric sdk notebook

From the Jupyter Notebook window in your browser, select the notebook named /quadric/sdk-cli/examples/models/lfm2_5_350m/lfm2_5_350m.ipynb.

LFM2.5-350M Text Generation on Chimera GPNPU

This notebook takes LFM2.5-350M, Liquid AI's hybrid language model (gated short convolutions plus grouped-query attention; the LFM2-350M architecture with new weights and a new tokenizer), from its Hugging Face PyTorch checkpoint to autoregressive text generation on the Chimera GPNPU: import Liquid AI's quantization-aware-distilled Q4_0 weights bit-exact, run them with 16-bit activations, compose a decode step from CCL (Chimera Compute Language) kernels with ChiPy, compile it with CGC (Chimera Graph Compiler), and generate tokens on the ISS (Instruction Set Simulator) with the recurrent state carried from step to step.

Why the PyTorch path?

The model never leaves PyTorch. The same torch.nn.Module code is the float reference and the CPU oracle of every GPNPU kernel: each ChiPy op is a module whose forward() computes exactly what its CCL kernel computes. There is no ONNX export to maintain, and LFM2.5's short-convolution layers, which have no standard ONNX decode form, are just another module.

Why a decode step with explicit state?

A generated token only needs the new token's projections plus what the model remembers about the past. LFM2.5 remembers two different things: each short-convolution layer keeps the last two B*x rows of its kernel-3 convolution, and each attention layer keeps a KV cache. The decode step reads and updates both in one DDR buffer, so each token is a single pass of matrix-vector products (GEMVs) over the 350M weights. The GEMVs map one output column per PE: each PE streams the weight column it owns while the activation is broadcast, so the whole PE array works on a one-row product.

Pipeline

Model: LFM2.5-350M (16 layers: 10 gated short-convolution + 6 GQA attention layers; width 1024, 16 query / 8 KV heads of 64, SwiGLU MLP 4608, vocabulary 65536 (64402 used by the tokenizer); 354M parameters; WikiText-2 and chat-formatted UltraChat windows for calibration, UltraChat for evaluation)

Host CPU in this notebook. Every stage of a decode token (embedding, all 16 layers and the LM head) runs on the GPNPU. The host CPU only tokenizes the prompt, takes the argmax of the returned logits, and passes the recurrent state from one ISS invocation to the next. PyTorch and the ChiPy chain also run on the host as the references the GPNPU output is checked against.

1. Setup

lfm2_helpers holds the downloads, the PyTorch models, the quantization recipe and the ISS drivers; lfm2_chipy holds the ChiPy ops and the decode step. The GPNPU target is QC-P with 8 MB of OCM (On-Chip Memory) and 8 MACs per PE: one core at 1.7 GHz.

import inspect
from pathlib import Path

import numpy as np
import torch

import lfm2_chipy
import lfm2_helpers as lfm2

%matplotlib inline

PROMPT = "The capital of France is"
NEW_TOKENS = 24  # tokens generated on the ISS in section 9
EVAL_WINDOWS = 32  # chat windows scored by the ChiPy chain in section 7
torch.manual_seed(0)

<torch._C.Generator at 0x72e469185fd0>

2. Download the Model

The checkpoint and tokenizer come from Hugging Face (LiquidAI/LFM2.5-350M, LFM Open License v1.0), pinned to a fixed revision. Four files come from the public sdk-cli-models bucket: the device parameters imported from Liquid AI's Q4_0 checkpoint in section 4, tokenized WikiText-2 windows of 257 tokens, chat-formatted windows of 257 tokens (conversations from UltraChat 200k rendered with the model's chat template), and the greedy generation of the original Hugging Face model for the prompt used in section 9.

Model license. The checkpoint and imported device parameters remain subject to the LFM Open License v1.0. Commercial use is free for legal entities with annual revenue below USD 10 million; the legal entity includes affiliates under the license. Commercial users at or above that revenue level should contact Liquid AI for commercial licensing terms. Each customer must assess its own eligibility: the Quadric SDK license does not grant additional rights to the model. download_assets() also downloads LICENSE and NOTICE.txt alongside the parameters; keep them with any redistributed assets. The notice identifies the weight conversions made by Quadric.

assets = lfm2.download_assets()
calib, val = lfm2.load_windows(assets)
chat = lfm2.load_chat_windows(assets)
tokenizer = lfm2.load_tokenizer(assets)
golden = lfm2.load_golden(assets)
sd = lfm2.load_state_dict(assets)

n_params = sum(t.numel() for t in sd.values())
print(
    f"LFM2.5-350M: {n_params / 1e6:.0f}M parameters, tokenizer vocabulary {tokenizer.get_vocab_size()}"
)
print(
    f"WikiText-2 windows: {len(calib)} calibration + {len(val)} held-out, {val.shape[1]} tokens each"
)
print(f"chat windows: {len(chat['calib'])} calibration + {len(chat['val'])} evaluation")

config.json: 0.00B [00:00, ?B/s]



tokenizer.json: 0.00B [00:00, ?B/s]



model.safetensors:   0%|          | 0.00/709M [00:00<?, ?B/s]


LFM2.5-350M: 354M parameters, tokenizer vocabulary 64402
WikiText-2 windows: 64 calibration + 128 held-out, 257 tokens each
chat windows: 64 calibration + 64 evaluation

3. A Deployable PyTorch Twin

lfm2.Original is the Hugging Face model written out op by op. We rewrite it into lfm2.DeployableLFM2, the float reference the GPNPU output is scored against. Every rewrite is exact in float:

The cell checks the twin against the original, the step-by-step decode against the full forward, and the twin's greedy generation against the original Hugging Face model's (hf_golden.json, 40 tokens). A negative control reverses the conv taps of one layer, which must break the match.

twin = lfm2.DeployableLFM2.from_hf(sd)
twin_err, step_err, neg_err = lfm2.check_twin(sd, twin, val)
print(f"max |original - twin| logits:          {twin_err:.2e}")
print(f"max |full forward - step-by-step decode|: {step_err:.2e}")
print(f"negative control (layer 0 taps flipped):  {neg_err:.2e}")
assert twin_err < 1e-3 and step_err < 1e-3, "the twin must match the original model"
assert neg_err > 1.0, "the negative control must break the match"

prompt_ids = golden["prompt_ids"]
twin_ids = lfm2.greedy(twin, prompt_ids, 40)
assert twin_ids == golden["gen_ids"], "twin greedy generation must equal the HF model's"
print(
    "twin greedy generation == Hugging Face (40 tokens):", repr(tokenizer.decode(twin_ids[1:])[:80])
)

max |original - twin| logits:          8.20e-05
max |full forward - step-by-step decode|: 9.20e-05
negative control (layer 0 taps flipped):  2.94e+01
twin greedy generation == Hugging Face (40 tokens): 'The capital of France is Paris. The French language is considered as the most im'

4. Import Liquid AI's QAD Q4_0 Weights

A decode token streams every weight from DDR once, so the weight bytes set the floor of the token time. Liquid AI publishes a quantization-aware-distilled (QAD) LFM2.5-350M in GGUF (LiquidAI/LFM2.5-350M-GGUF, LFM2.5-350M-QAD-Q4_0.gguf; QAD blog post): the model is distilled onto the Q4_0 grid instead of being rounded to it after training. Q4_0 stores int4 codes with one fp16 scale per block of 32 inputs, which is exactly the W4 block-32 symmetric format of our GEMV kernel, so every Linear is imported bit-exact:

We also quantized the BF16 checkpoint ourselves with quadric_pyquant (GPTQ, W4 block-32 with the most sensitive Linears at W8). On the same evaluation (128 held-out chat windows, the device numerics emulated on a GPU), GPTQ reached KL(fp32 ‖ quantized) 0.124, the QAD import 0.100 with every Linear at 4 bits, so the notebook uses QAD. Here is the import:

print(inspect.getsource(lfm2.import_qad_gguf))

def import_qad_gguf(gguf_path, lm_bits=8):
    """Map the QAD Q4_0 GGUF onto the device parameters (the format of lfm2_5_350m_qparams.pt).

    * Every Linear keeps its Q4_0 codes and fp16 block-32 scales bit-exact: W4 block-32 symmetric,
      the GEMV kernel's format, with codes [K, N] and scales [K/32, N].
    * The RMSNorm gammas stay separate (``gamma1`` = operator_norm, ``gamma2`` = ffn_norm): folding
      them into the weights would move the codes off the Q4_0 grid.
    * The tied Q6_K embedding is dequantized and re-quantized to int8 per row for ``embed``; the lm
      head is ``embedding * embedding_norm`` (the final norm folded in), RTN block-32 at W8
      (``lm_bits=8``) or W4 asymmetric (``lm_bits=4``, faster).
    Needs the ``gguf`` package.
    """
    from gguf import GGUFReader
    from gguf.quants import dequantize

    tensors = {t.name: t for t in GGUFReader(str(gguf_path)).tensors}

    def floats(name, shape):
        t = tensors[name]
        values = np.asarray(dequantize(t.data, t.tensor_type), np.float32)
        return torch.from_numpy(values.copy()).reshape(shape)

    def q4_0(names):  # block = fp16 d + 16 bytes; code j < 16: low nibble - 8, else high - 8
        codes, scales = [], []
        for name in names:
            raw = np.asarray(tensors[name].data).reshape(tensors[name].data.shape[0], -1, 18)
            scales.append(raw[:, :, :2].copy().view(np.float16)[..., 0].astype(np.float32))
            nibbles = raw[:, :, 2:]
            low, high = (nibbles & 15).astype(np.int8) - 8, (nibbles >> 4).astype(np.int8) - 8
            codes.append(np.concatenate([low, high], -1).reshape(raw.shape[0], -1))
        codes, scales = np.concatenate(codes, 0), np.concatenate(scales, 0)  # [N, K], [N, K/32]
        return {
            "codes": torch.from_numpy(codes.T.copy()),
            "scales": torch.from_numpy(scales.T.copy()),
            "bits": 4,
        }

    out = {}
    for i in range(NL):
        blk = f"blk.{i}."
        names = {k: f"{blk}{v}.weight" for k, v in _GGUF_NAMES.items()}
        if i in ATTN:
            out[f"layers.{i}.mix.qkv"] = q4_0([names["q"], names["k"], names["v"]])
            out[f"layers.{i}.mix.o"] = q4_0([names["o"]])
            norms = [floats(f"{blk}attn_{n}_norm.weight", (HD,)) for n in "qk"]
            out[f"layers.{i}.qk_norm"] = torch.stack(norms)
        else:
            out[f"layers.{i}.mix.in_proj"] = q4_0([names["in_proj"]])
            out[f"layers.{i}.mix.o"] = q4_0([names["out_proj"]])
            out[f"layers.{i}.conv_w"] = floats(f"{blk}shortconv.conv.weight", (DIM, KC))
        out[f"layers.{i}.w13"] = q4_0([names["w1"], names["w3"]])
        out[f"layers.{i}.w2"] = q4_0([names["w2"]])
        out[f"layers.{i}.gamma1"] = floats(f"{blk}attn_norm.weight", (DIM,))
        out[f"layers.{i}.gamma2"] = floats(f"{blk}ffn_norm.weight", (DIM,))
    table = floats("token_embd.weight", (VOCAB, DIM))
    codes, scales, zero_point = rtn_blocks(
        table * floats("token_embd_norm.weight", (DIM,)), lm_bits, asym=lm_bits == 4
    )
    out["lm"] = {
        "codes": codes.T.to(torch.int8).contiguous(),
        "scales": scales.T.contiguous(),
        "bits": lm_bits,
    }
    if zero_point is not None:
        out["lm"]["zp"] = zero_point.T.round().to(torch.int32).contiguous()
    row_scale = table.abs().amax(1, keepdim=True).clamp(min=1e-12) / 127
    out["embed"] = (torch.clamp(torch.round(table / row_scale), -128, 127) * row_scale).contiguous()
    return out

The import needs the gguf package and the 219 MB GGUF file, so the notebook loads its output from the sdk-cli-models bucket (downloaded in section 2). lfm2.import_qad_gguf on hf_hub_download(lfm2.QAD_REPO, lfm2.QAD_FILE, revision=lfm2.QAD_REVISION) regenerates the file exactly. On the 128 held-out chat windows the imported model measured KL 0.100 against the BF16 model and 89.6% top-1 agreement on the assistant-response tokens. Section 7 checks the accuracy again with the GPNPU arithmetic itself.

qparams = lfm2.load_qparams(assets)

summary = lfm2.describe_qparams(qparams)
print(f"{'weights':10s}{'Linears':>8s}{'codes MB':>10s}{'scales MB':>11s}")
for tag, (count, codes, scales) in sorted(summary.items()):
    print(f"{tag:10s}{count:8d}{codes / 1e6:10.1f}{scales / 1e6:11.1f}")
streamed = sum(codes + scales for _, codes, scales in summary.values())
print(
    f"streamed per token: {streamed / 1e6:.0f} MB (the fp32 checkpoint is {4 * n_params / 1e6:.0f} MB)"
)
q4_scales = torch.cat(
    [e["scales"].flatten() for e in qparams.values() if isinstance(e, dict) and e["bits"] == 4]
)
print(f"negative Q4_0 block scales: {(q4_scales < 0).float().mean():.1%}")

weights    Linears  codes MB  scales MB
W4              64     143.7       18.0
W8               1      67.1        4.2
streamed per token: 233 MB (the fp32 checkpoint is 1418 MB)
negative Q4_0 block scales: 50.0%

5. The CCL Kernels

lfm2_ops.hpp implements the decode step as CCL kernels. A short-conv layer runs four of them and an attention layer five, and each fuses the elementwise work into its GEMV epilogue so that no intermediate vector makes an extra trip through DDR. The GEMV input itself travels between ops as planes: the producer's epilogue already holds its output one column per PE, so it splits it into the high and low int8 planes of the 16-bit activation (plus the dynamic shift) that the next GEMV's MEU consumes, and no GEMV runs a separate split pass:

The GEMV core (gemvStream) gives each PE one output column. The weights are stored tile-major: each column tile of 256 columns (one per PE) is one contiguous stream, copied DDR → OCM double-buffered with its block scales, and cut into K-chunks of 512 rows that alternate between two LRM (Local Register Memory) buffers. An asynchronous PLS load puts chunk c+1 into LRM while chunk c is MACed; each block of 32 rows is MACed twice on the int8 MEU (the high and low planes of the 16-bit activation) and scaled by its signed block scale. Here is that loop:

header = Path("lfm2_ops.hpp").read_text()
start = header.index("    std::uint32_t id0 = p0.issue(L0);")
print(header[start : header.index("  barrier(ob0);", start)])

    std::uint32_t id0 = p0.issue(L0);
    std::uint32_t is0 = q0.issue(S0);
    qbar(id0);
    for(std::int32_t c = 0; c < nChunks; c += 2) {
      const std::uint32_t id1 = p1.issue(L1);  // lands while chunk c is MACed
      const std::uint32_t is1 = q1.issue(S1);
      if(c + 2 < nChunks) {
        memCpy(wV, ob0, 0, 0, (c + 2) * chunkRows, 0);
        memCpy(sV, os0, 0, 0, (c + 2) * sRows, 0);
        p0.rebind(ob0);
        q0.rebind(os0);
      }
      qbar(id0);
      qbar(is0);
      chunk(L0, S0, c);
      if(c + 2 < nChunks) {
        id0 = p0.issue(L0);
        is0 = q0.issue(S0);
        memCpy(wV, ob1, 0, 0, (c + 3) * chunkRows, 0);
        memCpy(sV, os1, 0, 0, (c + 3) * sRows, 0);
        p1.rebind(ob1);
        q1.rebind(os1);
      }
      qbar(id1);
      qbar(is1);
      chunk(L1, S1, c + 1);
    }
  }

6. The ChiPy Decode Step

Each kernel has a ChiPy twin: a torch.nn.Module decorated with @chipy.torch_ccl_custom_op whose forward() reproduces the kernel's arithmetic (int16 activation codes, exact integer accumulation, FixedPoint32 storage, the device's round-half-up). build_decode_step composes one token of them in a @chipy.func: embed, 16 layers of four or five ops, and the LM head (73 custom ops). The token id, position, RoPE row and the state buffer are runtime inputs, so one compiled program serves every position; the weights are baked in as constants. Here is the contract of the in_proj + short-conv op: its forward() is the oracle, and on the device it reads and writes its region of the state buffer in place:

source = Path("lfm2_chipy.py").read_text()
start = source.index('@chipy.torch_ccl_custom_op(\n    ccl_func_name="lfm2::w4gemvTConv"')
print(source[start : source.index("@chipy.torch_ccl_custom_op(", start + 1)])

@chipy.torch_ccl_custom_op(
    ccl_func_name="lfm2::w4gemvTConv",
    reserved_l2m=0,
    io_in_ext_mem=True,
    read_only=("x", "state", "state_off", "w", "s", "taps"),
)
class ConvInOp(torch.nn.Module):
    """in_proj GEMV + gated short conv in one op; y is returned as the out_proj's planes.

    bcx = h @ W, bx = B*x, y = C * (w0*h0 + w1*h1 + w2*bx), history <- (h1, bx).

    The oracle is the in_proj GEMV oracle followed by the short-conv oracle, with their own
    calibration points (``L{i}.in``, ``.bx``, ``.conv``, ``.y``).
    """

    _frac_bits = (0,)  # planes out

    def __init__(self, layer, entry, taps):
        """Pack in_proj's codes and scales; quantize the depthwise taps [D, 3] (oldest first)."""
        super().__init__()
        self.layer = layer
        packed = _GemvWeights(entry)
        object.__setattr__(self, "_matrix", packed.matrix)
        self.w, self.s, self.sfrac = packed.w, packed.s, packed.sfrac
        self.cal = calibration_point(f"L{layer}.in")
        self.wfrac = max_frac(taps)
        taps_q = np.round(taps.astype(np.float64).T * 2.0**self.wfrac).astype(np.int32)
        self.taps = taps_q.reshape(1, 1, KC, DIM)
        object.__setattr__(self, "_taps", torch.as_tensor(taps_q / 2.0**self.wfrac).float())
        self.bx_cal = calibration_point(f"L{layer}.bx")
        self.conv_cal = calibration_point(f"L{layer}.conv")
        self.y_cal = calibration_point(f"L{layer}.y")

    def template_param_factory(self):
        """Return <sFrac, inFrac, bxFrac, convFrac, wFrac, yFrac>."""
        return (
            self.sfrac,
            shared_frac(self.cal),
            shared_frac(self.bx_cal),
            shared_frac(self.conv_cal),
            self.wfrac,
            shared_frac(self.y_cal),
        )

    def forward(self, x, state, state_off, w, s, taps):
        """Return y as planes; the oracle updates its conv history (the device, the state edge)."""
        bcx = act_dot(x, self._matrix)
        self.cal(bcx)
        bcx = store(bcx, lambda: shared_frac(self.cal)).float()
        b_gate, c_gate, xx = bcx.reshape(3, DIM)
        bx = b_gate * xx
        self.bx_cal(bx)
        bx = store(bx, lambda: shared_frac(self.bx_cal))
        oracle = ORACLE_STATE["state"]
        hist = oracle["conv"][self.layer] if oracle is not None else torch.zeros(KC - 1, DIM)
        conv = self._taps[0] * hist[0] + self._taps[1] * hist[1] + self._taps[2] * bx
        self.conv_cal(conv)
        conv = store(conv, lambda: shared_frac(self.conv_cal))
        if oracle is not None:
            oracle["conv"][self.layer] = torch.stack([hist[1], bx])
        y = c_gate * conv
        self.y_cal(y)
        return encode_planes(store(y, lambda: shared_frac(self.y_cal)))

All recurrent state lives in one int8 DDR buffer: first the two FixedPoint32 B*x rows of each conv layer, then the fp16 K cache (packed as v2i16 words, the layout the attention kernel's QK matmul reads) and V cache of each attention layer, for 256 positions:

decode_step = lfm2_chipy.build_decode_step(qparams)
print(f"Decode step: {len(lfm2_chipy.all_ops(decode_step))} CCL custom ops")
print(f"State buffer: {lfm2_chipy.STATE_BYTES / 1e6:.2f} MB, int8 {lfm2_chipy.STATE_SHAPE}")
for layer in (0, 1, 2, 5):
    kind = "attention" if layer in lfm2.ATTN else "short conv"
    offset = lfm2_chipy.kv_offset(layer) if layer in lfm2.ATTN else lfm2_chipy.conv_offset(layer)
    print(f"  layer {layer:2d} ({kind:10s}): byte offset {offset:9,d}")

Decode step: 73 CCL custom ops
State buffer: 3.23 MB, int8 (1, 1, 3152, 1024)
  layer  0 (short conv): byte offset         0
  layer  1 (short conv): byte offset     8,192
  layer  2 (attention ): byte offset    81,920
  layer  5 (attention ): byte offset   606,208

7. Calibrate and Score the GPNPU Arithmetic

Every FixedPoint32 value in the chain gets its fractional bits from a CalibrationPoint, recorded while the chain decodes one WikiText-2 and one chat window on the host CPU. The fractional bits are shared across the layers of each op (the minimum over the group), so layers of the same shape can share one compiled instantiation of a kernel.

After calibration the ChiPy chain is the GPNPU's arithmetic, so its output is the deployed model's. LFM2.5-350M is a chat model, so we score it on chat-formatted text against the float twin on the same inputs: the KL divergence KL(fp32 ‖ chain) of the next-token distributions and how often the two pick the same top-1 token, over all positions and over the assistant-response positions alone. The gate is KL below 0.2 and top-1 agreement above 80%.

n_ops = lfm2_chipy.calibrate(decode_step, [calib[0], chat["calib"][0]])
print(f"{n_ops} ops calibrated on 2 x 256 tokens (one WikiText-2, one chat window)")

fid = lfm2_chipy.cpu_fidelity(
    decode_step, twin, chat["val"][:EVAL_WINDOWS], chat["val_astart"][:EVAL_WINDOWS]
)
print(f"{EVAL_WINDOWS} chat windows x 256 positions, ChiPy chain (GPNPU math) vs fp32 twin:")
print(f"  KL(fp32 || chain)   {fid['kl']:.4f}   assistant tokens {fid['kl_assistant']:.4f}")
print(f"  top-1 agreement     {fid['agree']:.1%}   assistant tokens {fid['agree_assistant']:.1%}")
print(f"  perplexity          fp32 {fid['ref_ppl']:.3f}, chain {fid['ppl']:.3f}")
assert fid["kl"] < 0.2 and fid["agree"] > 0.8, "the GPNPU arithmetic must track the fp32 model"

71 ops calibrated on 2 x 256 tokens (one WikiText-2, one chat window)
32 chat windows x 256 positions, ChiPy chain (GPNPU math) vs fp32 twin:
  KL(fp32 || chain)   0.1040   assistant tokens 0.0674
  top-1 agreement     86.3%   assistant tokens 88.8%
  perplexity          fp32 13.841, chain 12.976

8. Compile with CGC and Run One Decode Step

CGC compiles the ChiPy program together with lfm2_ops.hpp. This build wraps every kernel in an ISS profile region (LFM2_PROFILE) and has no state output: a deployment keeps the state resident. In the generated C++, every attention layer receives the same state buffer and its own byte offset into it:

profile_build = lfm2.compile_decode_step(decode_step, "lfm2_decode_profile", profile=True)
print(f"Compiled in {profile_build['compile_s']:.0f} s")

generated = (lfm2.build_dir("lfm2_decode_profile").parent / "lfm2_decode_profile.cpp").read_text()
for line in generated.splitlines():
    if "lfm2::attention<" in line:
        print(line.strip()[:120] + " ...")

Compiled in 24 s
lfm2::attention<8,256,1024,25,22,28>(ext_tensor_14, input_sin, input_cos, input_state, input_pos, 81920, lfm2__attention ...
lfm2::attention<8,256,1024,25,22,28>(ext_tensor_33, input_sin, input_cos, input_state, input_pos, 606208, lfm2__attentio ...
lfm2::attention<8,256,1024,25,22,28>(ext_tensor_52, input_sin, input_cos, input_state, input_pos, 1130496, lfm2__attenti ...
lfm2::attention<8,256,1024,25,22,28>(ext_tensor_65, input_sin, input_cos, input_state, input_pos, 1654784, lfm2__attenti ...
lfm2::attention<8,256,1024,25,22,28>(ext_tensor_78, input_sin, input_cos, input_state, input_pos, 2179072, lfm2__attenti ...
lfm2::attention<8,256,1024,25,22,28>(ext_tensor_91, input_sin, input_cos, input_state, input_pos, 2703360, lfm2__attenti ...

Each ISS invocation executes one decode token. To run position p of a held-out window, the ChiPy chain first decodes positions 0..p-1 on the host CPU and writes its state into the device's buffer layout (reference_step); the ISS then runs token p from that state, and the chain computes the same token as the reference. The cost of a token grows with the position because attention reads every cached key and value, so we run several positions:

POSITIONS = (0, 20, 64, 128, 200, 255)
window = val[0].tolist()
iss_logits, chain_logits, cycles_by_pos = {}, {}, {}
for pos in POSITIONS:
    state, chain_logits[pos] = lfm2_chipy.reference_step(decode_step, window[: pos + 1])
    iss_logits[pos], _, cycles_by_pos[pos] = lfm2.run_decode_step(
        profile_build, window[pos], pos, state
    )
    print(
        f"position {pos:3d}: {cycles_by_pos[pos]:10,d} cycles ({1.7e9 / cycles_by_pos[pos]:5.1f} tokens/s at 1.7 GHz)"
    )

position   0:  5,329,252 cycles (319.0 tokens/s at 1.7 GHz)
position  20:  5,334,810 cycles (318.7 tokens/s at 1.7 GHz)
position  64:  5,351,751 cycles (317.7 tokens/s at 1.7 GHz)
position 128:  5,378,174 cycles (316.1 tokens/s at 1.7 GHz)
position 200:  5,394,874 cycles (315.1 tokens/s at 1.7 GHz)
position 255:  5,407,418 cycles (314.4 tokens/s at 1.7 GHz)

regions = lfm2.region_profile("lfm2_decode_profile")  # the last run: position 255
lfm2.print_region_table(regions, cycles_by_pos[POSITIONS[-1]])

region         calls      cycles   share
  w4loop          65   5,031,932   93.1%
w4gemvT           23   3,507,176   64.9%
w4gemvTResG       31   1,235,629   22.9%
w4gemvTConv       10     427,205    7.9%
attention          6     145,381    2.7%
w4gemvTRes         1      62,303    1.2%
  w4split         65      14,789    0.3%
rmsG               1       3,059    0.1%
embed              1       2,931    0.1%

9. Generate Text on the ISS

For generation we compile the same calibrated decode step again, now with the snapshot kernel, which returns the state buffer as a second output. The host passes in the token and its position, takes the greedy argmax of the returned logits as the next token, and feeds the returned state into the next step. The prompt is consumed one token per step, then the model generates.

decode_step.options["snapshot"] = True
generate_build = lfm2.compile_decode_step(decode_step, "lfm2_decode_generate")
print(f"Compiled in {generate_build['compile_s']:.0f} s")

prompt_ids = tokenizer.encode(PROMPT).ids
assert prompt_ids == golden["prompt_ids"], "the tokenizer must reproduce the golden prompt"
iss_ids, iss_cycles = lfm2.generate_on_iss(generate_build, prompt_ids, NEW_TOKENS)
print(tokenizer.decode(iss_ids[1:]))
print(
    f"\n{len(iss_cycles)} decode steps on the ISS, {np.mean(iss_cycles):,.0f} cycles per step on average"
)

Compiled in 24 s
The capital of France is Paris. The French Revolution was a significant event in the history of France and the world. It was a time of great

29 decode steps on the ISS, 5,473,754 cycles per step on average

10. Compare Against the PyTorch Reference

Three checks against the ChiPy chain on the host CPU (the oracle of every kernel), all asserted:

1. Single decode steps (section 8): logit cosine ≥ 0.9999 and the same argmax at every position.
2. Generation: the same prompt, decoded greedily by the chain, must give the same tokens as the ISS. The gate is at least 90% of the generated tokens identical, which leaves room for a fixed-point rounding tie to flip a token.
3. Negative control: the ISS logits of position 255 compared against the chain's logits for position 200 must fail the cosine gate, so the gate can tell outputs apart.

The generation is also compared against the original Hugging Face BF16 model in fp32. That one is reported, not asserted. The two texts part after " Paris. The French": the QAD weights themselves, dequantized and run in fp32, generate exactly the GPNPU's text, so the difference comes from Liquid AI's QAD checkpoint, not from the device.

for pos in POSITIONS:
    cos = lfm2.cosine(chain_logits[pos], iss_logits[pos])
    same = chain_logits[pos].argmax() == iss_logits[pos].argmax()
    print(f"position {pos:3d}: cosine {cos:.6f}, argmax {'match' if same else 'DIFFERS'}")
    assert cos >= 0.9999 and same, f"ISS decode step diverged from the ChiPy chain at {pos}"
neg_cos = lfm2.cosine(chain_logits[200], iss_logits[255])
print(f"negative control (position 255 vs 200): cosine {neg_cos:.4f}")
assert neg_cos < 0.9999, "the cosine gate must reject a mismatched step"

chain_ids = lfm2_chipy.cpu_generate(decode_step, prompt_ids, NEW_TOKENS)
new = slice(len(prompt_ids), len(prompt_ids) + NEW_TOKENS)
gen_agree = np.mean([a == b for a, b in zip(iss_ids[new], chain_ids[new])])
hf_agree = np.mean([a == b for a, b in zip(iss_ids[new], golden["gen_ids"][new])])
print(
    f"\nGeneration: {gen_agree:.0%} of {NEW_TOKENS} tokens match the ChiPy chain, {hf_agree:.0%} match HF fp32"
)
print("ISS         :", repr(tokenizer.decode(iss_ids[new])))
print("ChiPy chain :", repr(tokenizer.decode(chain_ids[new])))
print("HF fp32     :", repr(tokenizer.decode(golden["gen_ids"][new])))
assert gen_agree >= 0.9, "ISS generation diverged from the ChiPy chain"
print("PASS: ISS decode matches the PyTorch reference")

position   0: cosine 0.999999, argmax match
position  20: cosine 0.999999, argmax match
position  64: cosine 0.999999, argmax match
position 128: cosine 0.999999, argmax match
position 200: cosine 0.999999, argmax match
position 255: cosine 0.999999, argmax match
negative control (position 255 vs 200): cosine 0.1696

Generation: 100% of 24 tokens match the ChiPy chain, 21% match HF fp32
ISS         : ' Paris. The French Revolution was a significant event in the history of France and the world. It was a time of great'
ChiPy chain : ' Paris. The French Revolution was a significant event in the history of France and the world. It was a time of great'
HF fp32     : ' Paris. The French language is considered as the most important language in the country. The French language has a rich history and'
PASS: ISS decode matches the PyTorch reference

11. Performance

A token streams 233 MB of weight codes and block scales from DDR, and almost all of its cycles are in the GEMV loops (w4loop: 94.5% at position 0, 93.4% at position 200). Those loops run at the rate the PLS delivers 1 KB weight rows to the PE array (about 21 cycles per row): with the MACs removed they take 99% of the time they take with them. Attention is the only kernel whose cost grows with the position: 0.07M cycles at position 0, 0.13M at 200 and 0.14M at 255.

fig = lfm2.plot_profile(regions, cycles_by_pos)

The decode step reached this cost in steps, each measured at position 0 on the same QC-P configuration and checked against the ChiPy chain. The kernels were first developed on LFM2-350M, the same architecture, with GPTQ weights and a W8 LM head; from the QAD import on, the steps were measured on this notebook's LFM2.5-350M model:

Tradeoff in the LM head. The W8 LM head streams 34 MB more per token than a W4 one and is now 28% of the token. With the W4 asymmetric head (a zero point per block of 32, supported by w4gemvT) a token cost 0.60M cycles less at position 0 before the last five steps above, and on the 32 chat windows of section 7 the KL of the GPNPU arithmetic rises from 0.104 to 0.114 (top-1 agreement 86.3% to 85.1%). The notebook keeps W8; lfm2.import_qad_gguf(path, lm_bits=4) builds the faster variant.

Summary

Key takeaways

1. The PyTorch model is the single source of truth. The float reference and the per-kernel CPU oracles are all torch.nn.Module code, and the ISS output is checked against the same modules.
2. A hybrid recurrent/attention model decodes with one state buffer. Conv histories and KV caches share a DDR buffer that each mixer updates in place, so one compiled program serves every position.
3. Decode is a GEMV problem, and W4 halves it. Mapping one output column per PE, streaming the weights asynchronously and fusing the elementwise work into the epilogues puts 93-95% of the token in the weight-streaming loop, which runs at the PLS row rate.
4. Quantization-aware weights import bit-exact. Q4_0's int4 codes with fp16 block-32 scales are the GEMV kernel's W4 format, so Liquid AI's QAD checkpoint maps onto the device one to one, and it beat our own GPTQ on chat KL (0.100 vs 0.124). The only kernel changes are signed scales and RMSNorm gammas applied after each normalization.

Citation

@article{liquidAI2026350M,
  author={Liquid AI},
  title={LFM2.5-350M: No Size Left Behind},
  journal={Liquid AI Blog},
  year={2026},
  note={www.liquid.ai/blog/lfm2-5-350m-no-size-left-behind}
}

@article{liquidai2025lfm2,
  title={LFM2 Technical Report},
  author={Liquid AI},
  journal={arXiv preprint arXiv:2511.23404},
  year={2025}
}

@article{merity2016pointer,
  title={Pointer Sentinel Mixture Models},
  author={Merity, Stephen and Xiong, Caiming and Bradbury, James and Socher, Richard},
  journal={arXiv preprint arXiv:1609.07843},
  year={2016}
}