llama.cpp on Chimera GPNPU

Introduction

llama.cpp is an inference engine for LLMs written in C/C++. It loads models in the GGUF format, builds a GGML compute graph per token, and dispatches that graph to whichever backends are registered in the build.

Quadric's fork adds ggml-epu, a GGML backend that offloads those graph operations to a Chimera GPNPU. The backend registers a device named EPU0 ("Quadric Chimera GPNPU") alongside the CPU backend, so all of the standard llama.cpp front-ends — llama-cli, llama-server, llama-bench and the rest — run on the GPNPU without any change to how they are invoked.

Device code is compiled separately with the Quadric LLVM toolchain (qllvm) and loaded at runtime as .qo binaries. The same tree builds against the cycle-accurate ISS and against FPGA hardware, selected at configure time with -DEPU_TARGET=ISS|FPGA.

GPNPU backend support

Scope of this integration. Chimera is a programmable GPNPU whose operator support can be extended through software. This experimental GGML backend exposes the operators and configurations currently integrated and validated for llama.cpp. The CPU fallback paths described below reflect the current scope of this integration, rather than an architectural requirement or a fixed limitation of Chimera.

Execution model. GGML asks each registered backend which operations it can run (supports_op), then schedules every node onto a backend that claims it. Anything the GPNPU does not claim — an unsupported operator, an unvalidated shape, an unsupported quantization — falls back to the CPU backend. A model therefore always runs; how much of it runs on the GPNPU depends on how many nodes the backend claims.

One case sits between the two. An attention node whose mask is neither plain causal, nor a single contiguous window, nor a gatherable set of runs is claimed by the GPNPU but computed on the host inside the backend, since no kernel expresses that shape. It is correct and considerably slower than the kernel, and the backend logs a warning the first time it happens. Sliding-window models reach it during prefill, as can a long-lived server slot whose KV cache has become fragmented.

Operators. The backend implements the operator set a decoder-only transformer needs. The set below is the one built for QC Perform; a different fabric geometry compiles a subset, and any operator whose kernel is absent is declined and runs on the CPU.

Numerics and quantization. Kernels compute in FP16 with fixed-point intermediates. The backend accepts GGUF weights in every quantization; what differs is where they are unpacked:

• Q8_0, Q5_0 and Q4_0 stay quantized in device memory and are dequantized in-kernel — Q8_0 and Q5_0 through the 4×int8 path, Q4_0 through the packed 4-bit path. In the GGUF file these occupy ~1.06 B/element for Q8_0 and Q5_0 and ~0.56 B/element for Q4_0, and the host-side dequantization is removed. Each matmul weight additionally keeps a packed copy resident on the device, so a decode step re-reads it rather than re-packing the weight on every call.
• Every other quantization — the K-quants (Q6_K, Q5_K_M, Q4_K_M, Q3_K_M, Q2_K) among them — is dequantized once on the host when the tensor is uploaded and held on the device as FP16. Compute then runs on the GPNPU in the same way.

Q8_0 and Q4_0 are therefore the recommended formats: they are the two the device reads directly and the two verified for every model that runs fully on the GPNPU, at the accuracy shown in Perplexity. Q5_0 takes the same direct path where the table lists it, but it is stored in the same int8 lanes as Q8_0, so it costs Q8_0's bytes at lower precision; Q4_0 is the one that halves them.

Each model is qualified per quantization. The model support table records what has been verified for each.

Attention shapes and the context ceiling. Flash-attention kernels are compiled per model geometry (query heads, KV heads, head dimension) and per context bucket. Each shape carries a measured context ceiling, beyond which attention falls back to the CPU rather than dispatching an unvalidated kernel:

This ceiling is compared against --ctx-size, not against the tokens actually in flight. llama.cpp resolves flash attention once per context, from a worst-case probe graph, so a context larger than the first column disables GPNPU attention for every layer of that context — however short the prompt. The default context size is the model's trained maximum, which is far above these ceilings for every model here, so pass -c explicitly at or below the model's ceiling. The backend logs a warning naming the value to use when this happens.

Between the two columns, a decode step runs the prefill kernel: correct, but it costs roughly the full bucket of attention work per generated token.

Serving features. Beyond answering a single prompt, the following llama.cpp serving behaviours run on the GPNPU, each covered by cases in the backend's test suite:

Context shift and prefix reuse both leave the KV cache fragmented — the live tokens are no longer one contiguous range — and attention handles that case directly, compacting the surviving cells before each dispatch rather than approximating the window. Coverage spans three model geometries, at head dimensions 64, 128 and 256.

The KV cache itself is held as FP16; the -ctk / -ctv cache-quantization options are not supported.

Profiling. Set GGML_EPU_PROFILE=1|2|3 for a per-operator wall/host/device summary and Chrome-trace output, and GGML_EPU_ISS_PROFILE=1 (ISS builds only) for cycle-accurate per-dispatch profiles. The backend reference carries the full list of runtime environment variables.

Model support

The quantizations listed per row are those verified with the end-to-end inference test, which compares GPNPU output against a CPU reference for the same graph, token by token. Both recommended formats, q8_0 and q4_0, are verified for every model that runs fully on the GPNPU.

Every model above runs end-to-end on the GPNPU except Llama 3.1 8B, whose FFN down-projection matmul this release runs on the CPU by default. Everything else in that model, attention included, runs on the device. GGML_EPU_DOWNPROJ_ON_EPU=1 moves the down-projection onto the GPNPU as well; it is the A/B handle for re-testing that placement, not the configuration this release qualifies.

Quantizations outside a row's list are not claimed for that model. Coverage is per model rather than global, and the lists differ in how far verification has been taken rather than in what the backend can load.

The Qwen 3 and Llama rows point at community conversions: Meta publishes no GGUF builds of these models at all, and the model authors' own Qwen 3 repos carry a narrower set of quantizations than the ones listed here. Another publisher's GGUF of the same model and quantization, including one you convert yourself from the original weights with convert_hf_to_gguf.py, is expected to behave the same, but the files verified are the ones linked here.

Perplexity: CPU vs GPNPU

Accuracy is reported as the relative difference between the GPNPU and llama.cpp's own CPU backend on the same tree, model file and text — not as an absolute GPNPU perplexity, since perplexity values are not comparable across models, tokenizers or implementations. Each model is evaluated with llama-perplexity over ten fixed 512-token windows of wikitext-2-raw/wiki.test.raw, run once on each backend, and the reported figure is the mean of 100 * (PPL_GPNPU / PPL_CPU - 1) across those windows.

Measured at Q8_0 on QC Perform — a 16×16 GPNPU configuration with 8 MB L2 memory — on FPGA, against the SDK build this release ships, in which Q8_0 and Q4_0 weights take the exact int8 fold path:

The ten-site means span -0.431 % to +0.066 % of the CPU reference. Note that the CPU backend is the reference, not ground truth: for Q8_0 weights it also quantizes each matrix-multiply input to 8-bit blocks, which the GPNPU does not, so a negative difference does not mean the device is more accurate. The spread between windows is reported as measured and varies by model.

For scale, the absolute CPU perplexity over the full Wikitext-2 test set:

Per-site values and the measurement method are in tools/perplexity/README_quadric.md in the llama.cpp source tree — /quadric/llama-cpp/tools/perplexity/README_quadric.md in the image — which also carries a model beyond the ones listed here.

Run

This page documents the Chimera SDK image, quadric.io/sdk-cli:$ver, which ships llama.cpp prebuilt alongside the SDK and the toolchain. Its front-ends — llama-cli, llama-completion, llama-server, llama-bench and llama-perplexity — are on PATH, so they are invoked by name from any directory. Throughout this page, $ver is the release version of the image you are running.

The image's entry point is the sdk command, so a front-end is reached either from a shell in the container, or by naming it as the entry point:

docker run -it --entrypoint bash quadric.io/sdk-cli:$ver
llama-cli --list-devices

docker run --entrypoint llama-cli quadric.io/sdk-cli:$ver --list-devices

EPU0 should be listed alongside the CPU device.

The device kernels are installed next to the backend, and the backend looks for them there, so nothing has to be run from a particular directory. To point a front-end at a different set of kernels — one you have just built, say — set GGML_EPU_KERNEL_PATH to the directory holding them.

Single prompt

llama-cli \
  -hf Qwen/Qwen2.5-0.5B-Instruct-GGUF:Q8_0 \
  --device EPU0 \
  -ngl 99 \
  -c 4096 \
  -st \
  -p "Hello!" \
  -n 64

• -hf — fetch the model from Hugging Face and cache it, naming any repository and quantization from Model support. It needs outbound HTTPS; use -m /path/to/model.gguf instead for a file you already have, or where the container has no network access.
• --device EPU0 — pin to the GPNPU. Optional when it is the only non-CPU device, but explicit is safer.
• -ngl 99 — offload all layers. Without this, layers stay on the CPU and the GPNPU is never exercised.
• -c — context size, at or below this model's ceiling from Attention shapes and the context ceiling. Without it the default is the model's trained maximum, which disables GPNPU attention for the whole context. 4096 is the Qwen 2.5 0.5B ceiling; use 512 for either Qwen 3 model.
• -st — answer the prompt and exit. llama-cli is a chat client: without it, -p is consumed as the first user turn and the session then waits for input instead of returning. Drop -st and -p for an interactive chat session.

Server

llama-server \
  -hf Qwen/Qwen2.5-0.5B-Instruct-GGUF:Q8_0 \
  --device EPU0 \
  -ngl 99 \
  -c 4096 \
  --host 127.0.0.1 --port 8080

-c carries the same meaning as above. Leave the KV cache at its FP16 default: the -ctk / -ctv quantized-cache options are not supported on this backend and end the run at context creation.

This serves the llama.cpp Web UI at http://127.0.0.1:8080 and an OpenAI-compatible API:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages": [{"role": "user", "content": "What is a GPNPU?"}], "max_tokens": 64}'

Build from source

The image already contains a build, so this section is only needed to rebuild the backend after changing it. Everything the build needs is in the image: the llama.cpp source tree at /quadric/llama-cpp, the Quadric SDK, the Quadric LLVM toolchain that compiles the device kernels, and a C++17 compiler with CMake.

Point the build at the SDK

The image already exports QLLVM_ROOT_PATH and SDK_INSTALL_PATH, the latter being where the SDK headers are read from. Only QuadricSdk_DIR has to be set:

export QuadricSdk_DIR=/quadric/sdk_install/lib/CMake/QuadricSdk   # dir holding QuadricSdkConfig.cmake

Configure and build

-DCMAKE_BUILD_TYPE=Release matches the shipped binaries; without it the rebuild is unoptimized. -DLLAMA_BUILD_TESTS=OFF keeps the configure step offline: the test suite downloads googletest from GitHub when it is enabled. The image also records the exact command its own binaries were built with, in /quadric/llama-cpp/build/bin/configure.sh.

cd /quadric/llama-cpp

cmake -B build \
  -DGGML_EPU=ON \
  -DCMAKE_BUILD_TYPE=Release \
  -DLLAMA_BUILD_TESTS=OFF \
  -DQuadricSdk_DIR="$QuadricSdk_DIR" \
  -DEPU_SDK_PATH="$SDK_INSTALL_PATH" \
  -DQLLVM_ROOT_PATH="$QLLVM_ROOT_PATH"

cmake --build build -j$(nproc)

A successful configure prints the discovered SDK libraries and the GPNPU geometry it is building for:

-- EPU SDK found at /path/to/sdk
-- Found libquadric_host: /path/to/sdk/build/libquadric_host.a
-- EPU Configuration:
--   NUM_CORES: 16
--   NUM_BORDERS: 2
--   NUM_PE_MACS: 8
--   OCM_SIZE: 8192 KiB

The full build produces the host binaries in build/bin and the device kernels (ggml_epu_kernels.qo plus its per-shape variants) alongside them. To build just the front-ends and the kernels:

cmake --build build --target llama-cli llama-server epu-all-kernels -j$(nproc)

Add -DEPU_TARGET=FPGA to the configure step to build for the hardware instead of the ISS. An FPGA build has to match a qualified fabric geometry, which -DEPU_HW_PROFILE=<profile> selects; the backend reference lists the profiles this release qualifies, along with non-default core counts and L2 memory sizes.

For the test suite, the profiling knobs and the full environment-variable reference, see the backend reference.