Handling Unsupported or Custom Operators

Change is inevitable

It's an exciting time to work in AI. Deep Neural Networks (DNN) are continuously evolving to solve ever-more challenging problems in a more efficient way.

For example, some noteworthy vision models have shifted away from Convolutional Neural Network (CNN) based model architectures to mixed-mode or transformer-layer based models:

The figure above depicts the rapid innovation of model architectures in the Computer Vision (CV) space from 2016 to 2022. It also demonstrates the need for AI compute platforms to be flexible and fully-programmable to keep up-to-date with the pace of innovation of model architectures.

Chimera Empowers Developers

Quadric’s Chimera processors and SDK is designed to support the ever-changing needs of AI developers. While the operator coverage of the CGC is rapidly growing, Quadric's primary goal is not to port and optimize every model and kernel for the Chimera platform. Instead, we've designed a hardware-software ecosystem that empowers developers to quickly and easily port new operators defined in the latest, state-of-the-art models or implement custom, proprietary operators designed in-house themselves.

Custom Operator Workflow

When an unsupported operator or graph region is encountered during the import phase of CGC, the user has the option to partition the graph around the unrecognized operator(s) and replace it with a custom operator:

The input and output tensor shapes and names for this custom operator can be inferred from the graph and be used to automatically create the custom operators C++ function declaration. A new ONNX graph is then created with the operator(s) in that graph region replaced with the defined custom operator. A developer can then write a C++ implementation for the unsupported or custom operator to complete the functional compilation of the model with CGC.

The code-equivalent process of this involves the following easy steps:

1. Replace one ore more ONNX operators in a graph with either a QuadricCusomOpElementWise or a QuadricCustomOpusing the graphutils Python package included with the Chimera SDK.

2. Implement the custom operator(s) in C++. Continue reading this document and refer to Tutorial: Writing a Kernel in C++ API for more information on writing performant C++ kernels for the Chimera Architecture.

3. Compile the ONNX model with custom operator(s) inserted using the CGC by providing the custom operator C++ implement from step 2.

The following diagram shows how user-defined operators fit into the CGC compilation flow outlined on Overview of the CGC:

Types of Custom Ops

The CGC supports two types of custom operators:

• Change is inevitable
• Chimera Empowers Developers
• Custom Operator Workflow
• Types of Custom Ops
  • Element-wise Custom Operators
  • General-Purpose Custom Operators

Note: The custom operator type selected has significant impact on the overall performance of the graph compiled in CGC. We recommend that you read the descriptions of both types of custom operators carefully to understand when each type should be used.

Element-wise Custom Operators

An element-wise function that operates on corresponding elements between two tensors. Elements between tensors are said to be corresponding if the two elements occupy the same position within their respective tensors. In most cases, the position is determined by the indices used to locate each element.

Suppose we have the following two tensors:

>>> import torch

>>> t1 = torch.tensor([
    [1,2],
    [3,4]
])

>>> t2 = torch.tensor([
    [9,8],
    [7,6]
])

Both of these tensors are rank-2 tensors, i.e. two-dimensional, with a shape of (2,2).

This means that we have two axes that both have a length of two elements each. The elements of the first axis are arrays and the elements of the second axis are numbers.

Below is an example of performing an element-wise addition between tensors t1 and t2:

>>> sum_t = t1 + t2

>>> print(sum_t)

tensor([[10., 10.],
        [10., 10.]])

Since DNNs typically process high-dimensional tensors during inference, element-wise operations, like the one above, are extremely common. If the unsupported operator you wish to port or the custom operator you wish to implement is an element-wise operation, you will experience significant performance improvements by representing the custom op as a QuadricCusomOpElementWise.

An example of when you might want to use a QuadricCusomOpElementWise is if your model uses a novel activation function. In the example below, an unknown NewActivation operator is replaced in an ONNX graph with a QuadricCusomOpElementWise by setting the element_wise parameter to True:

## The following code fragment reads in input.onnx and replaces all nodes
## of type NewActivation with a QuadricCustomOpElementWise node. Notice the
## ccl_func_name argument provides the name of the C++ function that CGC will
## bind to.

post_processing = modify(
    "input.onnx",
    NewActivation,
    inference = True,
    ccl_func_name= "UserCPPFunctionName",
    element_wise = True
)

check(self, post_processing, "input_with_quadric_elementwise_custom_op.onnx")

User operation (NewActivation) in the Input Graph

Graph instantiated with CGC supported QuadricCusomOpElementWise node

General-Purpose Custom Operators

Any algorithm that is not element-wise operation should be represented with a general-purpose QuadricCusomOp. There are no computational restrictions to using a general-purpose QuadricCusomOp. Thus, a general-purpose custom operator can be used to implement an element-wise operation, but, if you choose to do so, your implementation will perform significantly below its potential.

Another benefit of general-purpose custom operators is that they can be used to replace a collection of ONNX operators. This feature is particularly useful for operators that are defined in Deep Learning (DL) frameworks, such as PyTorch or Tensorflow, but do not have a standard ONNX operator equivalent. To learn more about this particular use-case, refer to the Tutorial: Custom Operator Insertion for Operators without an ONNX Equivalent.

Below is an example code snippet and Netron graph visualization of an ONNX model before and after a general-purpose QuadricCusomOpis used to replace four unknown ONNX nodes by setting the element_wise parameter to False:

## The following code fragment reads in input.onnx and replaces all sub-graphs
## that match the required pattern with a QuadricCustomOp node. Notice the
## ccl_func_name argument provides the name of the C++ function that CGC will
## bind to.
from graphutils import FilterNode

match1 = FilterNode("match1", [self.fw], 0)
match2 = FilterNode("match2", [match1], 0)
match3 = FilterNode("match3", [match1], 0)
match4 = FilterNode("match4", [match2, match3], 0)

post_processing = modify(
    "input.onnx",
    match4,
    inference = True,
    ccl_func_name= "UserCPPFunctionName",
    element_wise = False
)

check(self, post_processing, "input_with_quadric_general-purpose_custom_op.onnx")

A sub-graph in the input ONNX

The sub-graph on the left replaced with QuadricCusomOp node