C++ Code Generation

Now that we have a fundamental understanding of the types of optimzations that CGC can implement and the decision-making process that it uses, we can elaborate on the "passes" or iterations over the graph that CGC takes to convert an abstract ONNX graph into human-readable C++ code.

The C++ code generated by CGC can easily be combined with more traditional C++ code written by a developer to build a complete application signal-processing chain.

CGC Passes & Intermediate Representations (IR)

After each pass by CGC, an Intermediate Representation (IR) of the graph is created with additional runtime context added. CGC builds upon the optimzation decisions from the previous passes to select the most optimal Chimera Compute Library (CCL) APIs to use for the graphs implementation.

Below is a table describing each of the passes made by CGC, including each's input IR, output IR, and a summary of the optimizations performed and context added by each pass:

Continue reading for a more detailed description of each CGC pass.

Front-End

The CGC Front-End accepts frozen ONNX graphs. The Front-End checks the ONNX graph for compatibility with further downstream CGC optimization modules. The checks that are reported are:

• Graph Quantization Support
• Operator Support
• Graph Structure Support

The Front-End surfaces and localizes any potential issues. It also provides references to the relevant documentation and support resources to allow you to resolve important issues.

When frozen graph representations are successfully ingested through the Front-End, the graph is converted into a TVM Relay representation for further optimizations.

To learn how to convert your model to an ONNX graph format, refer to the next section of this guide on Preparing a Model for Compilation.

Input: ONNX Graph

Output: TVM Relay

Target-Independent Optimizations

In this module, CGC makes Target-Independent Optimizations. These optimizations convert the TVM Relay to an optimized representation. This module performs the following types of optimization passes:

• Constant Folding

• Algebraic Simplification

• Operation Fusion

• Operation Reordering

Input: TVM Relay

Output: TVM Relay

Float → Fixed Point Quantization

Framework-level Quantization focuses on the quantization of multiply-accumulate operations. Oftentimes, this leaves operations such as Batch Normalization and Global Average Pool represented as single precision floats. The QB-series architecture can support fixed point 32, fixed point 16, fixed point 8, and int8 datatypes. This optimization module identifies remaining floating point data types and replaces them with a fixed-point representation. The output of this Optimization Module is a fully quantized TVM Relay.

It is recommend that you simulate the result of this Optimization Module using the Inference Simulator.

Input: TVM Relay

Output: TVM Relay

First Pass Lowering

Architecture-aware passes are introduced in the First Pass Lowering. The passes contained within this module are:

• Buffer Allocation

• Memory Splitting

• Loop Tiling

• Software Pipelining

• Hoisting

• Dead Code Elimination

• Custom Op Provisioning

The output of the First Pass Lowering Module is a TVM Relay with special Quadric-specific extensions.

Input: TVM Relay

Output: Quadric-extended TVM Relay

Second Pass Lowering

The Second Pass Lowering is also an Architecture-aware module that performs the following optimization passes:

• Memory Allocation

• Loop Unrolling and Assignment

• Pre-fetching Optimizations

The output from the pass is TVM’s Target Intermediate Representation (TIR)

Input: Quadric-extended TVM Relay

Output: Target Intermediate Representation (TIR)

Code Generation

The Code Generation Module takes TIR and converts the representation to Quadric CCL through a series of passes. Further, any custom operations that were provisioned in the First Pass lowering will be inserted into the code at this step to be compiled during the next step.

Input: TIR

Output: C++

Compile

Now that a full C++ codebase has been emitted by CGC, the next step is to compile the generated code. The Chimera C++ compiler is based on the popular LLVM compiler framework.

The output of the Compiler phase is a target-specific program in Chimera Assembly. The resulting assembly program can be simulated using the Chimera Instruction Set Simulator (ISS).

For more information about simulating the program see:

Input: C++

Output: Quadric Assembly

Assembler

The module will convert the Compile-generated assembly file into a byte-code representation binary file. This binary file is loaded onto a Chimera QB-series Core instance.

Input: Quadric Assembly

Output: Binary