Model Demo: L2CS Fine-Grained Gaze Estimation

NOTE: The Jupyter Notebook below is included in the Chimera SDK and can be run interactively by running the following CLI command:

$ quadric sdk notebook

From the Jupyter Notebook window in your browser, select the notebook named /quadric/sdk-cli/examples/models/l2cs/l2cs.ipynb.

L2CS-Net Gaze Estimation

Abstract

L2CS-Net is a robust CNN-based model for predicting gaze in unconstrained settings. It regresses each gaze angle separately to improve the per-angel prediction accuracy, which will enhance the overall gaze performance. In addition, two identical losses, one for each angle, to improve network learning and increase its generalization are used. This model is evaluated with MPIIGaze and Gaze360 datasets collected with unconstrained settings.

The following is an example of L2CS-Net Gaze Estimation result. Gaze estimation is demonstrated with an arrows drawn from the center of a detected face with predicted yaw and pitch.

For details, please refer the paper.

• Ahmed A.Abdelrahman, Thorsten Hempel, Aly Khalifa, Ayoub Al-Hamadi. L2CS-NET: FINE-GRAINED GAZE ESTIMATION IN UNCONSTRAINED ENVIRONMENTS.

How It Works

The L2CS-Net works as a pipeline: a face detector model that operates on the full image and computes face locations, and the L2CS-Net model that operates on those locations and predicts gaze via regression.

The L2CS-Net github uses retinaface as a face detection model. In this notebook, we use Ultraface model which is a lightweight facedetection model designed for edge computing devices..

Notebook Outline

In this notebook, the following steps are taken to experiment the model.

• Set-Up
  • Package Installation
• L2CS Gaze Estimation Model
  • Pre-Quantized Model Download
  • Preprocessing
  • Compilation
• Demo
  • Face Detection
  • Gaze Estimation

Set-Up

Package Installation

!pip3 install -r ../../requirements.txt -q

WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv


import matplotlib.pyplot as plt
import numpy as np
from pathlib import Path
from PIL import Image

from torchvision.transforms import Compose, Normalize, ToTensor

from examples.models.zoo.zoo_utils import download_file
from examples.models.zoo.pose_estimators_zoo.pose_estimators_utils import (
    face_detection_per_image,
)
from tvm.contrib.epu.chimera_job.chimera_job import ChimeraJob
from sdk_cli.lib.inference import InferenceEngine, batch_inference
from sdk_cli.node_builtins.classical.gaze_estimator_postprocessing import (
    convert_raw_gaze_predictions_to_angles,
)
from sdk_cli.node_builtins.outputs.bbox_label_visualizer import draw_bbox
from sdk_cli.node_builtins.outputs.gaze_estimation_visualizer import draw_detected_gazes
from sdk_cli.utils.transforms import ResizePad

L2CS Gaze Estimation Model

Pre-Quantized Model Download

The L2CS-Net model is provided pre-built in Quadric's public model store: it has already been exported to ONNX, simplified, and quantized to asymmetric int8. We download it directly and go straight to compilation.

MODEL_NAME = "l2csnet"
model_input_size = (448, 448)  # (width, height)

## Pre-optimized, asymmetric-int8 L2CS-Net. Input `inputs0` is 1x3x448x448; the two
## gaze-regression heads `outputs0`/`outputs1` are each 1x90 (yaw/pitch bins).
onnx_file = f"{MODEL_NAME}-sim_opt_asym_int8_q.onnx"
S3_BASE = "https://sdk-cli-models.s3.us-east-2.amazonaws.com"

if not Path(onnx_file).exists():
    download_file(f"{S3_BASE}/{onnx_file}", onnx_file)

print(f"L2CS quantized ONNX ready: {onnx_file}")

L2CS quantized ONNX ready: l2csnet-sim_opt_asym_int8_q.onnx

Preprocessing

For this model, inputs should be normalized with the following parameters.

• size: (448, 448)
• mean: [0.485, 0.456, 0.406]
• std: [0.229, 0.224, 0.225]

dataset_mean, dataset_std = (0.485, 0.456, 0.406), (0.229, 0.224, 0.225)

transforms = Compose(
    [
        ResizePad(model_input_size),
        ToTensor(),
        Normalize(dataset_mean, dataset_std),
    ]
)

Compilation

## Define the chimerajob. The quantized ONNX embeds its tensor ranges, so no
## separate tensor-ranges file is required.
print(f"Define chimerajob with {onnx_file}")

cgc_job = ChimeraJob(model_p=onnx_file)

## Compile the model.
cgc_job.compile(quiet=True)
print(cgc_job)

Define chimerajob with l2csnet-sim_opt_asym_int8_q.onnx

╒═════════════════════╤════════════════════════════════════════════════════════════════════════════╕
│ Module Name         │ l2csnet_sim_opt_asym_int8_q_QC_U_1d7_16MB_4kB_128GBps_128GBps_16_OFF_x1_x1 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────┤
│ ONNX File           │ l2csnet-sim_opt_asym_int8_q.onnx                                           │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────┤
│ Product Target      │ QC-U                                                                       │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────┤
│ Number of Cores     │ 1                                                                          │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────┤
│ ISS Clock Frequency │ 1.700                                                                      │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────┤
│ L2M Size            │ 16MB                                                                       │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────┤
│ LRM Size            │ 4kB                                                                        │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────┤
│ External Read BW    │ 128GBps                                                                    │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────┤
│ External Write BW   │ 128GBps                                                                    │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────┤
│ MACS per PE         │ 16                                                                         │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────┤
│ Max L2M             │ 11.971MB                                                                   │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────┤
│ Max LRM             │ 2.750kB                                                                    │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────┤
│ Max Temp Ext Bytes  │ 0.000MB                                                                    │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────┤
│ Network GMACs       │ 16.349                                                                     │
╘═════════════════════╧════════════════════════════════════════════════════════════════════════════╛

╒════╤════════╤══════════╤══════════════════╤══════════════════════════╤═══════╕
│    │ Type   │ Name     │ shape            │ type                     │ mse   │
╞════╪════════╪══════════╪══════════════════╪══════════════════════════╪═══════╡
│  0 │ Input  │ inputs0  │ [1, 3, 448, 448] │ tensor[FixedPoint32<27>] │ n/a   │
├────┼────────┼──────────┼──────────────────┼──────────────────────────┼───────┤
│  1 │ Output │ outputs0 │ [1, 90]          │ tensor[FixedPoint32<27>] │ n/a   │
├────┼────────┼──────────┼──────────────────┼──────────────────────────┼───────┤
│  2 │ Output │ outputs1 │ [1, 90]          │ tensor[FixedPoint32<26>] │ n/a   │
╘════╧════════╧══════════╧══════════════════╧══════════════════════════╧═══════╛

Demo

Batch Inference

ChimeraJob supports batch execution for both ort and iss. This invokes specified number of threads to execute inference in parallel.

Here, we take advantage of this feature so that the inference on the target images finishes faster.

• Face Detection: invoke multiple threads for multiple input images.
• Gaze Estimation: invoke multiple threads for multiple detected faces across multiple input images.

Face Detection

The pipeline starts here. Here, we will experiment L2CS with nigel_daniel.jpg. To try with nigel_veer.jpg, uncomment the line.

all_image_paths = [
    "../../common/calibration/face/nigel_daniel.jpg",
    # '../../common/calibration/face/nigel_veer.jpg',
]

bboxes_per_image = face_detection_per_image(all_image_paths)

/usr/local/lib/python3.10/dist-packages/google/protobuf/symbol_database.py:55: UserWarning: SymbolDatabase.GetPrototype() is deprecated. Please use message_factory.GetMessageClass() instead. SymbolDatabase.GetPrototype() will be removed soon.
  warnings.warn('SymbolDatabase.GetPrototype() is deprecated. Please '

Display Face Detection Results

pic_len = 2
for image_path, outputs in bboxes_per_image.items():
    original_image = Image.open(image_path)
    ax, idx = {}, 1
    fig = plt.figure(figsize=(4 * pic_len, 4), tight_layout=True)

    # Display an original image.
    ax[idx] = fig.add_subplot(int("1%s%s" % (pic_len, idx)))
    ax[idx].imshow(original_image)
    ax[idx].set_title("Original Image")
    ax[idx].axis("off")

    # Display detected faces.
    idx += 1
    ax[idx] = fig.add_subplot(int("1%s%s" % (pic_len, idx)))
    ax[idx].imshow(draw_bbox(np.array(original_image), outputs, show_class=False))
    ax[idx].set_title("Detected Faces")
    ax[idx].axis("off")

    plt.show()

Gaze Estimation

Now, we run L2CS-Net for gaze estimation.

faces_per_image = {}
boxes_per_image = {}

for image_path, detections in bboxes_per_image.items():
    frame = np.array(Image.open(image_path))
    faces_per_image[image_path] = []
    boxes_per_image[image_path] = []
    for bbox in detections:
        x1, y1, x2, y2 = bbox[:4].astype(np.int32)
        score = bbox[4]
        if score > 0.5:
            extracted_image = frame[y1:y2, x1:x2, :]
            faces_per_image[image_path].append(Image.fromarray(extracted_image))
            boxes_per_image[image_path].append(bbox[:4].astype(np.int32))

engines = {
    InferenceEngine.CHIMERA_ORT_INT8: cgc_job,
    InferenceEngine.CHIMERA_ISS_INT8: cgc_job,
}

gaze_predictions_per_image = {}
for (image_path, face_images), boxes in zip(faces_per_image.items(), boxes_per_image.values()):
    original_image = Image.open(image_path)
    transformed_face_images = []
    for image in face_images:
        transformed_image = transforms(image)

        transformed_face_images.append(
            np.expand_dims(transformed_image.numpy(), axis=0).astype(np.float32)
        )

    gaze_predictions_per_image[image_path] = {}
    THREADS = min(len(face_images), 6)
    for inference_engine, engine in engines.items():
        all_inference_outputs = batch_inference(
            inference_engine,
            engine,
            transformed_face_images,
            threads=THREADS,
        )
        gaze_predictions_per_image[image_path][inference_engine] = []
        pitches_per_image, yaws_per_image = [], []

        for inference_outputs in all_inference_outputs:
            region_outputs = convert_raw_gaze_predictions_to_angles(
                inference_outputs[0], inference_outputs[1]
            )

            pitches_per_image.append(region_outputs[0])
            yaws_per_image.append(region_outputs[1])

        gaze_predictions_per_image[image_path][inference_engine].append(
            {
                "boxes": np.array(boxes),
                "pitches": pitches_per_image,
                "yaws": yaws_per_image,
            }
        )

2026-07-18 12:12 - WARNING - epu - chimera_job - ORT is not threadsafe -- forcing single threaded batch execution
100%|████████████████████████████████████████████| 2/2 [00:00<00:00,  2.08it/s]
Processing: 100%|████████████████████████████████| 2/2 [01:47<00:00, 53.86s/it]

Inference statistics of L2CS-Net is the following.

print(cgc_job)
cgc_job.plot_run_statistics()

╒═════════════════════╤════════════════════════════════════════════════════════════════════════════╕
│ Module Name         │ l2csnet_sim_opt_asym_int8_q_QC_U_1d7_16MB_4kB_128GBps_128GBps_16_OFF_x1_x1 │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────┤
│ ONNX File           │ l2csnet-sim_opt_asym_int8_q.onnx                                           │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────┤
│ Product Target      │ QC-U                                                                       │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────┤
│ Number of Cores     │ 1                                                                          │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────┤
│ ISS Clock Frequency │ 1.700                                                                      │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────┤
│ L2M Size            │ 16MB                                                                       │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────┤
│ LRM Size            │ 4kB                                                                        │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────┤
│ External Read BW    │ 128GBps                                                                    │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────┤
│ External Write BW   │ 128GBps                                                                    │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────┤
│ MACS per PE         │ 16                                                                         │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────┤
│ Max L2M             │ 11.971MB                                                                   │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────┤
│ Max LRM             │ 2.750kB                                                                    │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────┤
│ Max Temp Ext Bytes  │ 0.000MB                                                                    │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────┤
│ Network GMACs       │ 16.349                                                                     │
╘═════════════════════╧════════════════════════════════════════════════════════════════════════════╛

╒════╤════════╤══════════╤══════════════════╤══════════════════════════╤═══════╕
│    │ Type   │ Name     │ shape            │ type                     │ mse   │
╞════╪════════╪══════════╪══════════════════╪══════════════════════════╪═══════╡
│  0 │ Input  │ inputs0  │ [1, 3, 448, 448] │ tensor[FixedPoint32<27>] │ n/a   │
├────┼────────┼──────────┼──────────────────┼──────────────────────────┼───────┤
│  1 │ Output │ outputs0 │ [1, 90]          │ tensor[FixedPoint32<27>] │ 0.000 │
├────┼────────┼──────────┼──────────────────┼──────────────────────────┼───────┤
│  2 │ Output │ outputs1 │ [1, 90]          │ tensor[FixedPoint32<26>] │ 0.003 │
╘════╧════════╧══════════╧══════════════════╧══════════════════════════╧═══════╛

Post-ISS Report 1.7 GHz ***
Fully placed-and-routed gate simulation: 
╒══════════════════════════════════╤═════════╕
│ Latency (ms)                     │ 2.00    │
├──────────────────────────────────┼─────────┤
│ FPS                              │ 499.54  │
├──────────────────────────────────┼─────────┤
│ Average Power @ 3nm SSGNP (mW)   │ 2531.02 │
├──────────────────────────────────┼─────────┤
│ FPS per Watt @ 3nm SSGNP (FPS/W) │ 197.37  │
├──────────────────────────────────┼─────────┤
│ Ext Rd Bytes (MB)                │ 29.07   │
├──────────────────────────────────┼─────────┤
│ Ext Wr Bytes (MB)                │ 0.00    │
├──────────────────────────────────┼─────────┤
│ Avg Ext Rd BW (GBps)             │ 14.18   │
├──────────────────────────────────┼─────────┤
│ Avg Ext Wr BW (GBps)             │ 0.00    │
├──────────────────────────────────┼─────────┤
│ MAC Utilization                  │ 29.32%  │
╘══════════════════════════════════╧═════════╛
*** Data generated using 7nm SSGNP gatesim and scaled to 3nm

[SDK-CLI] : TotalCycles: 3,403,150
[SDK-CLI] : Executions/second: 499.54

compute      : ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 1.034M
data_array   : ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 501.839K
mac          : ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 1.495M
data_external: ▇ 30.418K
data_ocm     : ▇▇▇▇▇▇▇▇ 268.866K

for more information check run directory: /quadric/sdk-cli/examples/models/l2cs/ccl_build/l2csnet_sim_opt_asym_int8_q_QC_U_1d7_16MB_4kB_128GBps_128GBps_16_OFF_x1_x1/run/20260718_121245_ec273b


2026-07-18 12:14 - INFO - epu - chimera_job - Combined plots generated and saved to: 
/quadric/sdk-cli/examples/models/l2cs/ccl_build/l2csnet_sim_opt_asym_int8_q_QC_U_1d7_16MB_4kB_128GBps_128GBps_16_OFF_x1_x1/run/20260718_121245_ec273b/data/l2csnet_sim_opt_asym_int8_q_QC_U_1d7_16MB_4kB_128GBps_128GBps_16_OFF_x1_x1.combined.png





'/quadric/sdk-cli/examples/models/l2cs/ccl_build/l2csnet_sim_opt_asym_int8_q_QC_U_1d7_16MB_4kB_128GBps_128GBps_16_OFF_x1_x1/run/20260718_121245_ec273b/data'

Display Gaze Estimation Results

We are going to display gaze estimation results with ORT and ISS side-by-side so that they can be compared.

pic_len = len(engines.keys()) + 1

for image_path, all_outputs in gaze_predictions_per_image.items():
    ax, idx = dict(), 1
    fig = plt.figure(figsize=(4 * pic_len, 4), tight_layout=True)

    # Display an original image.
    original_image = Image.open(image_path)
    ax[idx] = fig.add_subplot(int("1%s%s" % (pic_len, idx)))
    ax[idx].imshow(original_image)
    ax[idx].set_title("Gaze Estimation: Original Image")
    ax[idx].axis("off")

    for inference_engine in engines.keys():
        boxes = all_outputs[inference_engine][0]["boxes"]
        pitches = all_outputs[inference_engine][0]["pitches"]
        yaws = all_outputs[inference_engine][0]["yaws"]
        detected_image = draw_detected_gazes(np.array(original_image), boxes, pitches, yaws)

        # Display gaze estimation results.
        idx += 1
        ax[idx] = fig.add_subplot(int("1%s%s" % (pic_len, idx)))
        ax[idx].imshow(detected_image)
        ax[idx].set_title(f"Gaze Estimation: {str(inference_engine).upper()}")
        ax[idx].axis("off")

    fig.show()