Model Demo: MPIIFaceGaze Gaze Estimation

NOTE: The Jupyter Notebook below is included in the Chimera SDK and can be run interactively by running the following CLI command:

$ quadric sdk notebook

From the Jupyter Notebook window in your browser, select the notebook named /quadric/sdk-cli/examples/models/mpiifacegaze/mpiifacegaze.ipynb.

MPIIFaceGaze Gaze Estimation on Chimera GPNPU

This notebook estimates where people in a photo are looking. MediaPipe Face Mesh finds each face, the face is normalized by its head pose, and the MPIIFaceGaze network, compiled with CGC (Chimera Graph Compiler), predicts the gaze direction on the Chimera GPNPU. The ISS (Instruction Set Simulator) result is checked against ONNX Runtime on the same quantized graph.

Why the full face?

Earlier appearance-based methods crop the eyes and regress the gaze from those patches alone. MPIIFaceGaze feeds the whole face and learns a spatial weights map that emphasizes the facial regions that carry information about the gaze, so head orientation and eye appearance are read together. The input is a normalized face: the image is warped so that a virtual camera looks at the face center from a fixed distance with the head upright. The network therefore sees faces that differ only in the gaze and the head rotation, not in their position or scale in the photo.

Pipeline

Model: MPIIFaceGaze resnet_simple from ptgaze (trained on MPIIFaceGaze, 224x224 BGR face, 2.9 M parameters as exported), output (pitch, yaw) in radians in the normalized camera space.

Note — split pipeline.

• On the Chimera GPNPU: the MPIIFaceGaze gaze network
• On the host CPU: the MediaPipe Face Mesh landmark network

The full pipeline can run end-to-end on the GPNPU — see yolox_e2e_chipy.ipynb or retina_net_e2e_chipy.ipynb for examples that compose CCL custom ops with ChiPy to keep every stage on-chip.

License Notice

The model is the MPIIFaceGaze resnet_simple checkpoint of ptgaze, published with the MIT licence at hysts/ptgaze-mpiifacegaze-resnet-simple. It was trained on the MPIIFaceGaze dataset, which the Max Planck Institute for Informatics publishes under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License and with the statement "The data is only to be used for non-commercial scientific purposes": https://creativecommons.org/licenses/by-nc-sa/4.0/

Its backbone was initialized from torchvision's ImageNet-1K weights (IMAGENET1K_V1).

The dataset page asks that work using the dataset cite Zhang et al., It's Written All Over Your Face: Full-Face Appearance-Based Gaze Estimation, CVPR Workshops 2017, reproduced under Citation below.

1. Setup

Besides the Chimera SDK, the notebook uses mediapipe for face landmark detection, installed with the example requirements (examples/requirements.txt). The face normalization and the gaze geometry live in two helper files next to the notebook: mpiifacegaze_helpers.py (download, face detection, normalization, drawing) and ptgaze_face_model.py (the 3D face template and the head pose, adapted from ptgaze).

import numpy as np
from mpiifacegaze_helpers import (
    angle_between_deg,
    detect_faces,
    download_model,
    preprocess,
    show_detections,
    show_gazes,
    to_gaze_vector,
)
from tvm.contrib.epu.chimera_job.chimera_job import ChimeraJob

from sdk_cli.lib.inference import InferenceEngine, batch_inference

%matplotlib inline

2. Download the Pre-Quantized Model

The model is provided pre-built in Quadric's public model store. It has been exported to ONNX (opset 16) and quantized to symmetric int8 with the Quadric quantization pipeline, calibrated on 116 normalized faces. The tensor ranges ship as a separate .tranges file.

For the export, the 1x1 convolution that produces the single-channel spatial weights was widened to an equivalent 256-channel convolution with repeated weights, so the weight map is multiplied with the 256x14x14 feature map channel by channel. The result is mathematically identical to the original graph.

onnx_path, tranges_path = download_model()
print(f"Model: {onnx_path.name}")
print(f"Tensor ranges: {tranges_path.name}")

Downloading mpiifacegaze_opt_sym_int8_q.onnx ...
Downloading mpiifacegaze_opt_sym_int8_q.onnx.tranges ...
Model: mpiifacegaze_opt_sym_int8_q.onnx
Tensor ranges: mpiifacegaze_opt_sym_int8_q.onnx.tranges

3. Compile with CGC

ChimeraJob takes the quantized ONNX and its tensor ranges and compiles the graph for the Chimera GPNPU. The input image is 1x3x224x224; the output gaze is 1x2, (pitch, yaw) in radians.

cgc_job = ChimeraJob(model_p=str(onnx_path), trange_file=str(tranges_path))
cgc_job.compile(quiet=True)
print(cgc_job)

╒═════════════════════╤════════════════════════════════════════════════════════════════════════════╕
│ Module Name         │ mpiifacegaze_opt_sym_int8_q_QC_U_1d7_16MB_4kB_128GBps_128GBps_16_OFF_x1_x1 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────┤
│ ONNX File           │ mpiifacegaze_opt_sym_int8_q.onnx                                           │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────┤
│ Product Target      │ QC-U                                                                       │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────┤
│ Number of Cores     │ 1                                                                          │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────┤
│ ISS Clock Frequency │ 1.700                                                                      │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────┤
│ L2M Size            │ 16MB                                                                       │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────┤
│ LRM Size            │ 4kB                                                                        │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────┤
│ External Read BW    │ 128GBps                                                                    │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────┤
│ External Write BW   │ 128GBps                                                                    │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────┤
│ MACS per PE         │ 16                                                                         │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────┤
│ Max L2M             │ 3.548MB                                                                    │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────┤
│ Max LRM             │ 1.758kB                                                                    │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────┤
│ Max Temp Ext Bytes  │ 0.000MB                                                                    │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────┤
│ Network GMACs       │ 1.415                                                                      │
╘═════════════════════╧════════════════════════════════════════════════════════════════════════════╛

╒════╤════════╤════════╤══════════════════╤══════════════════════════╤═══════╕
│    │ Type   │ Name   │ shape            │ type                     │ mse   │
╞════╪════════╪════════╪══════════════════╪══════════════════════════╪═══════╡
│  0 │ Input  │ image  │ [1, 3, 224, 224] │ tensor[FixedPoint32<29>] │ n/a   │
├────┼────────┼────────┼──────────────────┼──────────────────────────┼───────┤
│  1 │ Output │ gaze   │ [1, 2]           │ tensor[FixedPoint32<31>] │ n/a   │
╘════╧════════╧════════╧══════════════════╧══════════════════════════╧═══════╛

4. Detect and Normalize the Faces

The model does not take a face crop but a normalized face image, produced on the host in three steps:

The face center is the mean of both eyes' corners and both mouth corners. The camera parameters of the demo photo are unknown, so, as in ptgaze, the camera is assumed to have a focal length equal to the image width, the principal point at the image center and no distortion.

We use veer_daniel.jpg, which shows two people; to try nigel_daniel.jpg, swap the comment.

image_paths = [
    "../../common/calibration/face/veer_daniel.jpg",
    # "../../common/calibration/face/nigel_daniel.jpg",
]

faces_per_image = detect_faces(image_paths)

/usr/local/lib/python3.10/dist-packages/google/protobuf/symbol_database.py:55: UserWarning: SymbolDatabase.GetPrototype() is deprecated. Please use message_factory.GetMessageClass() instead. SymbolDatabase.GetPrototype() will be removed soon.
  warnings.warn('SymbolDatabase.GetPrototype() is deprecated. Please '


veer_daniel.jpg: 2 face(s)

The left panel shows the detected faces and landmarks; the others are the normalized faces, which are the actual model inputs.

for image_path, faces in faces_per_image.items():
    show_detections(image_path, faces)

5. Run Gaze Estimation on ONNX Runtime and the ISS

Each normalized face is converted to the model input (BGR, ImageNet mean and standard deviation in BGR order) and run twice: through ONNX Runtime on the quantized graph (CHIMERA_ORT_INT8), and on the ISS (CHIMERA_ISS_INT8). batch_inference runs the faces of an image in parallel threads. The predicted (pitch, yaw) is turned into a 3D gaze vector and rotated back from the normalized camera to the camera of the photo.

engines = [InferenceEngine.CHIMERA_ORT_INT8, InferenceEngine.CHIMERA_ISS_INT8]

predictions = {}
for image_path, faces in faces_per_image.items():
    inputs = [preprocess(face["normalized_image"]) for face in faces]
    predictions[image_path] = {}
    for engine in engines:
        outputs = batch_inference(engine, cgc_job, inputs, threads=min(len(inputs), 6))
        predictions[image_path][engine] = [
            to_gaze_vector(output[0], face) for face, output in zip(faces, outputs)
        ]

2026-10-23 00:58 - WARNING - epu - chimera_job - ORT is not threadsafe -- forcing single threaded batch execution
100%|████████████████████████████████████████████| 2/2 [00:00<00:00, 10.90it/s]
Processing: 100%|████████████████████████████████| 2/2 [00:10<00:00,  5.16s/it]

Each arrow starts at the 3D face center and points 10 cm along the predicted gaze, projected onto the photo. ONNX Runtime and the ISS are shown side by side.

for image_path, faces in faces_per_image.items():
    gaze_vectors = {
        engine: [vector for _, vector in results]
        for engine, results in predictions[image_path].items()
    }
    show_gazes(image_path, faces, gaze_vectors)

6. Compare the ISS Against ONNX Runtime

The reference is ONNX Runtime on the same quantized graph. For every face we measure the angle between the ISS and ONNX Runtime gaze vectors. Gate: every face within 0.5°. The cell fails if any face exceeds it.

GATE_DEG = 0.5

diffs_deg = []
for image_path, results in predictions.items():
    ort_results = results[InferenceEngine.CHIMERA_ORT_INT8]
    iss_results = results[InferenceEngine.CHIMERA_ISS_INT8]
    for idx, ((ort_angles, ort_vector), (iss_angles, iss_vector)) in enumerate(
        zip(ort_results, iss_results), start=1
    ):
        diff_deg = angle_between_deg(ort_vector, iss_vector)
        diffs_deg.append(diff_deg)
        ort_pitch, ort_yaw = np.rad2deg(ort_angles)
        iss_pitch, iss_yaw = np.rad2deg(iss_angles)
        print(
            f"face {idx}: ORT (pitch, yaw) = ({ort_pitch:+.2f}, {ort_yaw:+.2f}) deg, "
            f"ISS = ({iss_pitch:+.2f}, {iss_yaw:+.2f}) deg, angle = {diff_deg:.2f} deg"
        )

assert diffs_deg, "No face was detected, so nothing was compared."
assert max(diffs_deg) <= GATE_DEG, f"ISS vs ORT: {max(diffs_deg):.2f} deg > {GATE_DEG} deg"
print(f"PASS: {len(diffs_deg)} face(s), max ISS vs ORT angle {max(diffs_deg):.2f} deg")

face 1: ORT (pitch, yaw) = (-7.78, +9.38) deg, ISS = (-7.78, +9.38) deg, angle = 0.00 deg
face 2: ORT (pitch, yaw) = (-9.15, +12.82) deg, ISS = (-9.15, +12.82) deg, angle = 0.00 deg
PASS: 2 face(s), max ISS vs ORT angle 0.00 deg

7. Run Statistics

The inference statistics of the ISS run: cycles, executions per second and memory traffic.

print(cgc_job)
cgc_job.plot_run_statistics()

╒═════════════════════╤════════════════════════════════════════════════════════════════════════════╕
│ Module Name         │ mpiifacegaze_opt_sym_int8_q_QC_U_1d7_16MB_4kB_128GBps_128GBps_16_OFF_x1_x1 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────┤
│ ONNX File           │ mpiifacegaze_opt_sym_int8_q.onnx                                           │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────┤
│ Product Target      │ QC-U                                                                       │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────┤
│ Number of Cores     │ 1                                                                          │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────┤
│ ISS Clock Frequency │ 1.700                                                                      │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────┤
│ L2M Size            │ 16MB                                                                       │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────┤
│ LRM Size            │ 4kB                                                                        │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────┤
│ External Read BW    │ 128GBps                                                                    │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────┤
│ External Write BW   │ 128GBps                                                                    │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────┤
│ MACS per PE         │ 16                                                                         │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────┤
│ Max L2M             │ 3.548MB                                                                    │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────┤
│ Max LRM             │ 1.758kB                                                                    │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────┤
│ Max Temp Ext Bytes  │ 0.000MB                                                                    │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────┤
│ Network GMACs       │ 1.415                                                                      │
╘═════════════════════╧════════════════════════════════════════════════════════════════════════════╛

╒════╤════════╤════════╤══════════════════╤══════════════════════════╤═══════╕
│    │ Type   │ Name   │ shape            │ type                     │ mse   │
╞════╪════════╪════════╪══════════════════╪══════════════════════════╪═══════╡
│  0 │ Input  │ image  │ [1, 3, 224, 224] │ tensor[FixedPoint32<29>] │ n/a   │
├────┼────────┼────────┼──────────────────┼──────────────────────────┼───────┤
│  1 │ Output │ gaze   │ [1, 2]           │ tensor[FixedPoint32<31>] │ 0.000 │
╘════╧════════╧════════╧══════════════════╧══════════════════════════╧═══════╛

Post-ISS Report 1.7 GHz ***
Fully placed-and-routed gate simulation: 
╒══════════════════════════════════╤═════════╕
│ Latency (ms)                     │ 0.59    │
├──────────────────────────────────┼─────────┤
│ FPS                              │ 1692.47 │
├──────────────────────────────────┼─────────┤
│ Average Power @ 3nm SSGNP (mW)   │ 1800.43 │
├──────────────────────────────────┼─────────┤
│ FPS per Watt @ 3nm SSGNP (FPS/W) │ 940.03  │
├──────────────────────────────────┼─────────┤
│ Ext Rd Bytes (MB)                │ 52.33   │
├──────────────────────────────────┼─────────┤
│ Ext Wr Bytes (MB)                │ 0.00    │
├──────────────────────────────────┼─────────┤
│ Avg Ext Rd BW (GBps)             │ 86.48   │
├──────────────────────────────────┼─────────┤
│ Avg Ext Wr BW (GBps)             │ 0.00    │
├──────────────────────────────────┼─────────┤
│ MAC Utilization                  │ 8.60%   │
╘══════════════════════════════════╧═════════╛
*** Data generated using 7nm SSGNP gatesim and scaled to 3nm

[SDK-CLI] : TotalCycles: 1,004,451
[SDK-CLI] : Executions/second: 1,692.47

compute      : ▇▇▇▇ 50.519K
data_array   : ▇▇ 29.004K
mac          : ▇▇▇▇▇▇▇▇▇▇▇ 125.862K
data_external: ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 250.39K
data_ocm     : ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 527.212K

for more information check run directory: /quadric/sdk-cli/examples/models/mpiifacegaze/ccl_build/mpiifacegaze_opt_sym_int8_q_QC_U_1d7_16MB_4kB_128GBps_128GBps_16_OFF_x1_x1/run/20261023_005849_b93e99


2026-10-23 00:59 - INFO - epu - chimera_job - Combined plots generated and saved to: 
/quadric/sdk-cli/examples/models/mpiifacegaze/ccl_build/mpiifacegaze_opt_sym_int8_q_QC_U_1d7_16MB_4kB_128GBps_128GBps_16_OFF_x1_x1/run/20261023_005849_b93e99/data/mpiifacegaze_opt_sym_int8_q_QC_U_1d7_16MB_4kB_128GBps_128GBps_16_OFF_x1_x1.combined.png





'/quadric/sdk-cli/examples/models/mpiifacegaze/ccl_build/mpiifacegaze_opt_sym_int8_q_QC_U_1d7_16MB_4kB_128GBps_128GBps_16_OFF_x1_x1/run/20261023_005849_b93e99/data'

Summary

Key takeaways

1. A normalized face input removes the head position and scale from the problem, so a compact network regresses the gaze directly as two angles.
2. The spatial-weights multiply was compiled as an equivalent 256-channel convolution, keeping the graph identical in value while matching the channel-wise broadcast the GPNPU executes.
3. ChimeraJob compiles the quantized ONNX and runs the same graph on ONNX Runtime and the ISS, so the on-chip result is checked against a host reference in one flow.

Citation

@inproceedings{zhang2017mpiifacegaze,
  title     = {It's Written All Over Your Face: Full-Face Appearance-Based Gaze Estimation},
  author    = {Zhang, Xucong and Sugano, Yusuke and Fritz, Mario and Bulling, Andreas},
  booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)},
  year      = {2017}
}

@inproceedings{zhang2018normalization,
  title     = {Revisiting Data Normalization for Appearance-Based Gaze Estimation},
  author    = {Zhang, Xucong and Sugano, Yusuke and Bulling, Andreas},
  booktitle = {Proceedings of the ACM Symposium on Eye Tracking Research and Applications (ETRA)},
  year      = {2018}
}