Accepted Model Formats - Quantized ONNX

Open Neural Network Exchange (ONNX)

The Chimera Graph Compiler (CGC) requires that Deep Neural Network (DNN) models be represented in the Open Neural Network Exchange (ONNX) format to run compilation.

ONNX is an open-source format for representing DNN models and is designed to enable AI developers to more easily use, compile, and deploy models trained with different Deep Learning (DL) frameworks, e.g. PyTorch, TensorFlow, TensorFlow Lite, Caffe, etc. ONNX accomplishes this by defining a common set of operators - the building blocks of deep learning models - and a conversion paradigm for representing these operators in each of the aforementioned DL frameworks.

Quadric has elected to use ONNX as its initial ingestion format for the CGC so that the toolchain is accessible to all developers, regardless of their preferred DL framework for training.

ONNX Operator Set (OpSet) Support

The Deep Learning community and its techniques are rapidly changing as more efficient and effective methods are discovered.

ONNX adapts to these changes by periodically releasing updates to their common set of operators that are used to build and represent DNNs. These updates are called ONNX OpSets. A list of all ONNX operators, past and present, and the OpSet with which they were introduced or edited can be found here.

Currently, CGC supports ONNX graphs defined using any of the following ONNX OpSets:

SUPPORTED_ONNX_OPSETS = [12, 13, 14, 15, 16]

By default, the Chimera SDK will attempt to convert any models to the following ONNX OpSet if not explicitly specified by a user:

DEFAULT_ONNX_OPSET = 16

NOTE: Not all operators defined in these OpSets are supported! These are the only ONNX OpSets with which CGC will attempt compilation.

Converting ONNX OpSets

If your ONNX model is already defined in an OpSet version other than the supported versions, you can attempt to convert to a support OpSet using the following Python function:

from pathlib import Path
from typing import Optional, Union

import onnx
from onnx import version_converter


def convert_to_onnx_opset(
        onnx_model: onnx.ModelProto,
        onnx_opset_version: int,
        export_path: Optional[Union[str, Path]] = None
    ):
    """Converts an ONNX model to a different ONNX opset version and saves it to disk.

    Parameters
    ----------
    model: onnx.ModelProto
        ONNX model to convert.
    onnx_opset_version: int
        ONNX opset version with which converted model will be represented.
    export_path : Union[str, Path]
        The file path to save the exported ONNX model.

    Returns
    -------
    converted_model: onnx.ModelProto
        ONNX model represented in `onnx_opset_version` opset.
    """
    # Get the opset version of the model
    opset_version = onnx_model.opset_import[0].version
    if opset_version != onnx_opset_version:
        converted_model = version_converter.convert_version(onnx_model, onnx_opset_version)

        onnx.checker.check_model(converted_model)

        if export_path:
            onnx.save(converted_model, str(export_path))

        print(f"Model converted to ONNX opset {onnx_opset_version} and saved at `{export_path}`.")
    else:
        print(f"Model already represented in ONNX opset {onnx_opset_version}. Skipping conversion.")

        converted_model = onnx_model

    return converted_model

Unsupported Operators

The operator coverage of the Chimera SDK is growing rapidly, but not all ONNX operators are supported by CGC at this time. Further, some custom operators found in models defined in DL frameworks like PyTorch, TensorFlow, etc. will never be natively supported by ONNX. To learn more about this, refer to Tutorial: Custom Operator Insertion for Operators without an ONNX Equivalent.

When an unsupported operator is encountered during the graph import phase, the user has the option to partition the graph around the unrecognized operator and replace it with a custom operator. The input and output tensor shapes and names for this custom operator can be inferred from the graph and be used to automatically create the custom operators function declaration. A developer can then write a C++ implementation for the unsupported operator and reintegrate the new custom operator into the original graph to complete the functional compilation of the model with CGC. To learn more about this process, refer to Handling Unsupported or Custom Operators.

Quantization

Quantization in the context of DL, is the process of approximating a DNN that uses floating-point numbers (typically 32 or 64-bit) for inference with a DNN that uses lower-precision numbers.

If you're interested in learning more about Quantization, there are many public resources that do an excellent job of explaining this concept. A few that we recommend are:

• Qualcomm: Here's why quantization matter for AI - Great conceptual overview of quantization and why it's relevant to AI

• Distiller: Range-Based Linear Quantization - Great introduction to symmetric versus asymmetric quantization

• Deci.AI: The Ultimate Guide to Deep Learning Model Quantization and Quantization-Aware Training - Great summary of the differences between Post-Training Quantization (PTQ) and Quantization-Aware Training (QAT) and helpful tips for quantization in practice

As you can see, quantization is not a unique concept to Quadric, but is common for all DL developers that wish to run their models on edge devices with power, size, and memory constraints that are not as relevant to the cloud-compute platforms used for training.

Quadric specifically requires that models be partially or fully quantized to supported data types. Additionally, the Chimera SDK can perform naive quantization, if your model is not already quantized and you want to estimate your models performance without accuracy.

Partial vs. Full Quantization

During partial quantization, the data types of weights and biases of only the main compute operators such as convolutions to 8-bit integers, i.e. INT8, while keeping other operators like activations, softmax and pooling layers in floating point.

In comparison, fully quantized graphs use only integer types for the intermediate tensors. In other words, the inputs and outputs of all hidden or intermediate layers of the model are represented in integer format. Note: the inputs and outputs of the entire graph however, may still be in floating point.

Quadric runtime performs inference over fully quantized graphs. Converting from partially quantized to fully quantized graphs can be performed automatically by the front-end of CGC. Refer to Tutorial: Quantizing and Validating a Neural Network to see an example of how a model can be quantized and prepared for digestion by the CGC.

Naive, Hybrid, & Selective Quantization

In this section we briefly describe different quantization methodologies and in what situations they are most appropriate.

NOTE: The Chimera SDK can only perform naive quantization at this time. Quadric provides this functionality to make it easier for users to quickly quantize their model and benchmark performance on the Chimera processor architecture. If you choose to use the built-in naive quantization, please be aware that your model's accuracy will likely drop significantly and that this is not indicative of the potential accuracy you could achieve if using a more robust quantization methodology.

Naive Quantization

In naive quantization, all operators are quantized to INT8 precision, and are calibrated using the same method.

Naive quantization is the simplest quantization methodology to implement, but often results in a dramatic drop in model accuracy compared to the original floating-point model. This is because the same quantization method is applied to all operators, regardless of their sensitivity to quantization.

Most DNNs have some layers that are more sensitive to quantization than others. For example, in single-stage detection models, like YOLO, the last layers for classification and bounding-box regression are the most sensitive, so it’s a common practice to exclude them.

Using more sophisticated quantization methods, such as hybrid or selective quantization, often produce more accurate results in contrast to naive quantization.

Hybrid Quantization

In hybrid quantization, some operators are quantized to INT8 precision, and some are left in the original, floating-point data type, e.g. FP16 or FP32.

Hybrid quantization can yield significantly more accurate results than naive quantization; however, hybrid quantization requires that have prior knowledge of the neural network structure and its quantization-sensitive layers in order to reap its benefits. A sensitivity analysis, i.e. excluding layers one-by-one and watch the change in latency/accuracy, can be performed to learn what layers are quantization-sensitive.

Selective Quantization

In selective quantization, only some operators are quantized to INT8 precision while sensitive and non-friendly layers remain in floating-point precision. Further, the quantized operators may be quantized with different calibration methods and different granularity, e.g. per channel or per tensor.

Selective quantization gives users the most flexibility in the selection of quantization parameters for different layers of the network and often is used to maximize accuracy and minimize latency simultaneously.

Performing Naive Quantization with the Chimera SDK CLI

If you've already installed the Chimera SDK locally, you can run the following command to perform naive quantization on your ONNX model:

$ quadric sdk graph quantize <Path to Model>.onnx --synthetic-input

To learn more about this and other Chimera SDK CLI commands, refer to theChimera SDK Command Line Interface (CLI).

For a tutorial that walks through the process of quantizing a model, refer to the Tutorial: Quantizing and Validating a Neural Network.