llama.cpp on Chimera GPNPU

Introduction

llama.cpp is an inference engine for LLMs written in C/C++. It loads models in the GGUF format, builds a GGML compute graph per token, and dispatches that graph to whichever backends are registered in the build.

Quadric's fork adds ggml-epu, a GGML backend that offloads those graph operations to a Chimera GPNPU. The backend registers a device named EPU0 ("Quadric Chimera GPNPU") alongside the CPU backend, so all of the standard llama.cpp front-ends — llama-cli, llama-server, llama-bench and the rest — run on the GPNPU without any change to how they are invoked.

Device code is compiled separately with the Quadric LLVM toolchain (qllvm) and loaded at runtime as .qo binaries. The same tree builds against the cycle-accurate ISS and against FPGA hardware, selected at configure time with -DEPU_TARGET=ISS|FPGA.

This guide follows the backend's development branch, so it can describe features newer than the llama.cpp build in a given SDK release; Run says where to find the description of that build.

GPNPU backend support

Scope of this integration. Chimera is a programmable GPNPU whose operator support can be extended through software. This experimental GGML backend exposes the operators and configurations currently integrated and validated for llama.cpp. The CPU fallback paths described below reflect the current scope of this integration, rather than an architectural requirement or a fixed limitation of Chimera.

Execution model. GGML asks each registered backend which operations it can run (supports_op), then schedules every node onto a backend that claims it. Anything the GPNPU does not claim — an unsupported operator, an unvalidated shape, an unsupported quantization — falls back to the CPU backend. A model therefore always runs; how much of it runs on the GPNPU depends on how many nodes the backend claims. A decline is normally silent, with one exception: refusing a fused-attention node logs an EPU: flash attention declined ... warning listing every reason it failed, one line per distinct reason set and attention shape. llama.cpp resolves flash attention once per context, so a run that asked for it would otherwise lose it silently.

One case sits between the two. An attention node whose mask is neither plain causal, nor a single contiguous window, nor a gatherable set of runs is claimed by the GPNPU but computed on the host inside the backend, since no kernel expresses that shape. It is correct and considerably slower than the kernel, and the backend logs a warning the first time it happens. Sliding-window models reach it during prefill, as can a long-lived server slot whose KV cache has become fragmented.

Operators. The backend implements the operator set a decoder-only transformer needs. The set below is the one built for QC Perform; a different fabric geometry compiles a subset, and any operator whose kernel is absent is declined and runs on the CPU.

Numerics and quantization. Kernels compute in FP16 with fixed-point intermediates. The backend accepts GGUF weights in every quantization; what differs is where they are unpacked:

• Q8_0, Q5_0, Q4_0 and Q4_K stay quantized in device memory and are dequantized in-kernel — Q8_0 and Q5_0 through the 4×int8 path, Q4_0 and Q4_K through the packed 4-bit path, Q4_K with its per-block minimum applied on the device as well. In the GGUF file these occupy ~1.06 B/element for Q8_0 and Q5_0 and ~0.56 B/element for Q4_0 and Q4_K, and the host-side dequantization is removed. Each matmul weight additionally keeps a packed copy resident on the device, so a decode step re-reads it rather than re-packing the weight on every call. A tensor whose blocks dequantize past the on-device limit is named in an EPU: <tensor>: ... its MUL_MAT runs on the CPU warning at load, and its matmul does run on the CPU: correct, but the weight is copied across on every compute. Pass -ot '<tensor>=CPU' to keep that weight on the CPU outright.
• Every other quantization — the K-quants Q6_K, Q5_K, Q3_K and Q2_K among them — is dequantized once on the host when the tensor is uploaded and held on the device as FP16. Compute then runs on the GPNPU in the same way. A Q4_K_M file holds both kinds: its Q4_K tensors take the first path and its Q6_K tensors this one.

Q8_0 and Q4_0 are therefore the recommended formats: of the formats the device reads directly, they are the two verified for every model that runs fully on the GPNPU, at the accuracy shown in Perplexity. Q5_0 takes the same direct path where the table lists it, but it is stored in the same int8 lanes as Q8_0, so it costs Q8_0's bytes at lower precision; Q4_0 is the one that halves them.

Each model is qualified per quantization. The model support table records what has been verified for each.

SOLVE_TRI is off by default. The backend declines the triangular solve unless GGML_EPU_SOLVE_TRI=1 is set, and the node falls back to the CPU. The kernel is correct, but it is slower than that fallback on the node it exists for, the Qwen3.6-MoE delta-net solve. Qwen3.6-MoE is not a supported model.

SwiGLU range limit. SwiGLU runs in one of two fixed-point formats, chosen per tile from the data. The default format holds |gate|, |up| < 8192, a positive gate below 128 and |silu(gate) * up| < 8192; a tile past those bounds is redone in the wide format, which holds |gate|, |up| < 8192 and |silu(gate) * up| < 32768. Past the wide bounds the kernel saturates: an operand clamps to the edge of its input range and the product to the edge of the output range, so the element keeps its sign but not its magnitude. The kernel counts each such element and the backend logs a warning:

EPU: SwiGLU fixed-point range breach: <count> element(s) at n_ff=<n_ff> (<path>) break the widest format's ...; saturated (<total> so far for SwiGLU)

The warning is logged once per FFN width and path; later breaches there are logged at debug level, and every line carries the running total. Saturated elements are clamped, not exact, so compare the run against the CPU backend.

Other range warnings. The same warning, with the operator in place of SwiGLU, reports elements that saturated at a fixed-point bound in L2_NORM (inputs past +-4096), in the RMS_NORM weight (counted from the weight as it is uploaded) and in the fused FFN's activation conversion. Each line names the site and the bound that was broken.

SOLVE_TRI range limit. The triangular solve runs in a fixed-point format whose domain is +-32768. An element of A or B past that bound saturates to its edge and is counted in a SOLVE_TRI range warning of the same form. The solve's own accumulator still wraps rather than saturates: forward substitution multiplies the input error by the condition number of A, so a badly conditioned A can push the solution past the domain and return wrapped values of the wrong sign, with no error reported. supports_op accepts a shape on its dimensions alone and cannot see conditioning, so the backend does not decline these. The delta-net operand this op exists for stays inside the domain for structural reasons; docs/backend/quadric.md in the llama.cpp source tree (/quadric/llama-cpp/docs/backend/quadric.md in the SDK environment) carries the derivation. Treat SOLVE_TRI on arbitrary matrices as unverified and check the result against the CPU backend.

Attention shapes and the context ceiling. Flash-attention kernels are compiled per model geometry (query heads, KV heads, head dimension) and per context bucket. Each shape carries a measured context ceiling, beyond which attention falls back to the CPU rather than dispatching an unvalidated kernel:

This ceiling is compared against --ctx-size, not against the tokens actually in flight. llama.cpp resolves flash attention once per context, from a worst-case probe graph, so a context larger than the first column disables GPNPU attention for every layer of that context — however short the prompt. The default context size is the model's trained maximum, which is far above these ceilings for every model here, so pass -c explicitly at or below the model's ceiling. The backend logs a warning naming the value to use when this happens.

Between the two columns, a decode step runs the prefill kernel: correct, but it costs roughly the full bucket of attention work per generated token. For Qwen 2.5 0.5B and 1.5B the second column is the decode kernel's limit of 1024 Kᵀ elements per PE; for the other rows it is the largest bucket verified for decode.

Serving features. Beyond answering a single prompt, the following llama.cpp serving behaviours run on the GPNPU, each covered by cases in the backend's test suite:

Context shift and prefix reuse both leave the KV cache fragmented — the live tokens are no longer one contiguous range — and attention handles that case directly, compacting the surviving cells before each dispatch rather than approximating the window. Coverage spans three model geometries, at head dimensions 64, 128 and 256.

The KV cache itself is held as FP16; the -ctk / -ctv cache-quantization options are not supported.

Profiling. Set GGML_EPU_PROFILE=1|2|3 for a per-operator wall/host/device summary and Chrome-trace output, and GGML_EPU_ISS_PROFILE=1 (ISS builds only) for cycle-accurate per-dispatch profiles. The user-facing runtime environment variables are documented in /quadric/llama-cpp/docs/backend/quadric.md.

Model support

The quantizations listed per row are those verified with the end-to-end inference test, which compares GPNPU output against a CPU reference for the same graph, token by token. Both recommended formats, q8_0 and q4_0, are verified for every model that runs fully on the GPNPU.

Every model above runs end-to-end on the GPNPU.

Quantizations outside a row's list are not claimed for that model. Coverage is per model rather than global, and the lists differ in how far verification has been taken rather than in what the backend can load.

The Qwen 2.5 7B, Qwen 3 and Llama rows point at community conversions: Meta publishes no GGUF builds of its models at all, the model authors' own Qwen 3 repos carry a narrower set of quantizations than the ones listed here, and Qwen's own 7B repo shards most of its quantizations and carries fewer of them. Another publisher's GGUF of the same model and quantization, including one you convert yourself from the original weights with convert_hf_to_gguf.py, is expected to behave the same, but the files verified are the ones linked here.

Perplexity: CPU vs GPNPU

Accuracy is reported as the relative difference between the GPNPU and llama.cpp's own CPU backend on the same tree, model file and text — not as an absolute GPNPU perplexity, since perplexity values are not comparable across models, tokenizers or implementations. Each model is evaluated with llama-perplexity over ten fixed 512-token windows of wikitext-2-raw/wiki.test.raw, run once on each backend, and the reported figure is the mean of 100 * (PPL_GPNPU / PPL_CPU - 1) across those windows.

These figures were measured on an SDK build the shipped image does not carry, and have not been re-measured on it; treat them as indicative. Measured at Q8_0 on QC Perform — a 16×16 GPNPU configuration with 8 MB L2 memory — on FPGA, with flash attention on, on an SDK build that carries a mitigation for the head-dimension-128 defect listed under Known limitations and in which Q8_0 and Q4_0 weights took the exact int8 fold path at every batch size. The shipped SDK carries neither: an FPGA build against it runs the unmitigated flash-attention kernel, and any build takes the fold for single-row matmuls only, which is what decode runs, with prefill matmuls on fp16 activations on QC Perform:

The ten-site means span -0.431 % to +0.066 % of the CPU reference. Note that the CPU backend is the reference, not ground truth: for Q8_0 weights it also quantizes each matrix-multiply input to 8-bit blocks, which the GPNPU does not, so a negative difference does not mean the device is more accurate. The spread between windows is reported as measured and varies by model.

For scale, the absolute CPU perplexity over the full Wikitext-2 test set:

Per-site values and the measurement method are in tools/perplexity/README_quadric.md in the llama.cpp source tree — /quadric/llama-cpp/tools/perplexity/README_quadric.md in the SDK environment — which also carries a model beyond the ones listed here.

Known limitations

Beyond the operator, model and context-ceiling limits stated above, a build configured for the FPGA fabrics (-DEPU_TARGET=FPGA with -DEPU_HW_PROFILE=qcp_haps200 or qcn_haps100, see Build from source) has the limitations below. The prebuilt front-ends target the ISS, where neither applies.

qcp_haps200 (QC Perform, 16×16).

• Flash attention returns corrupted output for models with a head dimension of 128 or 256, which is every model in the support table except Qwen 2.5 0.5B Instruct and Llama 3.2 1B Instruct. Run those models with -fa off: attention's matrix multiplies then run on the CPU, and the output matches the CPU reference.

qcn_haps100 (QC Nano, 8×8).

• Flash attention is not available on this fabric. Attention's matrix multiplies run on the CPU for every model, and the output matches the CPU reference.

Run

llama.cpp ships prebuilt in the Chimera SDK from release 26.09 onwards, alongside the SDK tools and the toolchain. Its front-ends — llama-cli, llama-completion, llama-server, llama-bench and llama-perplexity — are on PATH inside the SDK environment, so they are invoked by name from any directory.

Open a shell in the SDK environment with quadric sdk interactive (see Interactive Sessions in the quick start guide), then check that the GPNPU device is registered:

## on the host
quadric sdk interactive

## inside the session
llama-cli --list-devices

EPU0 should be listed alongside the CPU device, with its memory: on the ISS the host's RAM, on an FPGA build the board DDR the profile compiles in (16 GiB for qcp_haps200, 4 GiB for qcn_haps100) less a 16 MB arena. The model, its KV cache and the working buffers have to fit; an allocation past the pool fails with an EPU: out of device memory line naming the request, the free space and the pool. The backend's own device-side copies — a packed copy of every on-device quantized weight, about the file's size again, and the attention caches — are allocated at the first compute, after --fit has placed layers, so leave that headroom, or pass an explicit -ngl or raise the margin with -fitt. The session enters the active SDK version, printed as it starts; that version must be 26.09 or later, or llama-cli is not found. LLAMA_CPP_HASH in the session names the llama.cpp commit the front-ends were built from. This guide tracks the head of the epu-backend branch of quadric-io/llama.cpp rather than a release, so it can describe switches and behaviour added after the commit an image was built from. The session holds the llama.cpp source at that commit under /quadric/llama-cpp; when the session lacks something described here, read /quadric/llama-cpp/ggml/src/ggml-epu/llama-cpp-user-guide.md and /quadric/llama-cpp/docs/backend/quadric.md, which describe the build you are running.

The session starts in the directory you launched it from, mounted at /ws, so a model file kept there is passed as a bare -m model.gguf. To reach other host directories, pass --mount /host/path:/container/path to quadric sdk interactive when launching the session.

The device kernels are installed next to the backend, and the backend looks for them there, so nothing has to be run from a particular directory. To point a front-end at a different set of kernels — one you have just built, say — set GGML_EPU_KERNEL_PATH to the directory holding them.

Single prompt

llama-cli \
  -hf Qwen/Qwen2.5-0.5B-Instruct-GGUF:Q8_0 \
  --device EPU0 \
  -ngl 99 \
  -c 4096 \
  -st \
  -p "Hello!" \
  -n 64

• -hf — fetch the model from Hugging Face and cache it, naming any repository and quantization from Model support. It needs outbound HTTPS; use -m /path/to/model.gguf instead for a file you already have, or where the session has no network access.
• --device EPU0 — pin to the GPNPU. Optional when it is the only non-CPU device, but explicit is safer.
• -ngl 99 — offload all layers. Without this, layers stay on the CPU and the GPNPU is never exercised.
• -c — context size, at or below this model's ceiling from Attention shapes and the context ceiling. Without it the default is the model's trained maximum, which disables GPNPU attention for the whole context. 4096 is the Qwen 2.5 0.5B ceiling; use 512 for either Qwen 3 model.
• -st — answer the prompt and exit. llama-cli is a chat client: without it, -p is consumed as the first user turn and the session then waits for input instead of returning. Drop -st and -p for an interactive chat session.

Server

llama-server \
  -hf Qwen/Qwen2.5-0.5B-Instruct-GGUF:Q8_0 \
  --device EPU0 \
  -ngl 99 \
  -c 4096 \
  --host 127.0.0.1 --port 8080

-c carries the same meaning as above. Leave the KV cache at its FP16 default: the -ctk / -ctv quantized-cache options are not supported on this backend and end the run at context creation.

Before the server reports model loaded it runs one forward pass to warm the model up. That pass takes minutes on the ISS. Add --no-warmup to skip it, and the server starts without running any graph.

While a pass is on the device, its position is reported every 30 seconds so a long wait can be told apart from a hang, for example:

W EPU: graph still running - node 126 / 821, 31 s elapsed

The line is progress, not a fault. Passes that finish inside the interval report nothing.

This serves the llama.cpp Web UI at http://127.0.0.1:8080 and an OpenAI-compatible API. quadric sdk interactive publishes no ports, so whether the host can reach them depends on the container runtime: under Podman the session has its own network and both are reachable only from inside the session; under Docker on Linux the session shares the host network and a browser on the host reaches them as well. From inside the session, start the server in the background (append & to the command above) and query it from the same shell:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages": [{"role": "user", "content": "What is a GPNPU?"}], "max_tokens": 64}'

Build from source

The SDK environment already contains a build, so this section is only needed to rebuild the backend after changing it. Everything the build needs is in the SDK environment: the llama.cpp source tree at /quadric/llama-cpp, the Quadric SDK, the Quadric LLVM toolchain that compiles the device kernels, and a C++17 compiler with CMake.

Point the build at the SDK

The SDK environment already exports QLLVM_ROOT_PATH and SDK_INSTALL_PATH, the latter being where the SDK headers are read from. Only QuadricSdk_DIR has to be set:

export QuadricSdk_DIR=/quadric/sdk_install/lib/CMake/QuadricSdk   # dir holding QuadricSdkConfig.cmake

Configure and build

-DCMAKE_BUILD_TYPE=Release matches the shipped binaries; without it the rebuild is unoptimized. -DLLAMA_BUILD_TESTS=OFF keeps the configure step offline: the test suite downloads googletest from GitHub when it is enabled. The SDK environment also records the exact command its own binaries were built with, in /quadric/llama-cpp/build/bin/configure.sh.

cd /quadric/llama-cpp

cmake -B build \
  -DGGML_EPU=ON \
  -DCMAKE_BUILD_TYPE=Release \
  -DLLAMA_BUILD_TESTS=OFF \
  -DQuadricSdk_DIR="$QuadricSdk_DIR" \
  -DEPU_SDK_PATH="$SDK_INSTALL_PATH" \
  -DQLLVM_ROOT_PATH="$QLLVM_ROOT_PATH"

cmake --build build -j$(nproc)

A successful configure prints the discovered SDK libraries and the GPNPU geometry it is building for:

-- EPU SDK found at /path/to/sdk
-- Found libquadric_host: /path/to/sdk/build/libquadric_host.a
-- EPU Configuration:
--   NUM_CORES: 16
--   NUM_BORDERS: 2
--   NUM_PE_MACS: 8
--   OCM_SIZE: 8192 KiB

The full build produces the host binaries in build/bin and the device kernels (ggml_epu_kernels.qo plus its per-shape variants) alongside them. To build just the front-ends and the kernels:

cmake --build build --target llama-cli llama-server epu-all-kernels -j$(nproc)

Add -DEPU_TARGET=FPGA to the configure step to build for the hardware instead of the ISS. An FPGA build has to match a qualified fabric geometry, which -DEPU_HW_PROFILE=<profile> selects; /quadric/llama-cpp/docs/backend/quadric.md lists the profiles this release qualifies, along with non-default core counts and L2 memory sizes.

For the test suite, the profiling knobs and the runtime environment variables, see /quadric/llama-cpp/docs/backend/quadric.md.