After a model has been properly prepared, i.e. converted to the Open Neural Network Exchange (ONNX) format and been fully or partially quantized, the CGC will begin a series of graph optimizations to convert the imported DNN graph into an efficient C++ function.
Graph Costs
Before diving into the optimizations performed by the CGC, let's discuss the costs that are typically front-of-mind in inference runtime environments.
Compute Costs
One of the most fundamental jobs of CGC is to take high-dimensional tensor operations that are the building blocks of DNNs - each described solely by an operator name, a set of input tensors, and a set of attributes - and decompose them into loops of simpler operations expressible in C++ and CCL.
Because the tensors are high-dimensional, CGC has some degrees of freedom in deciding how to generate nested loop structures, i.e. over which dimension of the tensor is iterated. Some looping strategies can result in more efficient programs than others. CGC leverages knowledge of the Chimera Processor architecture being targeted when selecting loop structures to, among other things, maximize utilization of the Processor Elements (PE), particularly the Multiply-Accumulate (MAC) units.
For example, there are a finite number of Processor Elements (PE) in a given Chimera core and one looping strategy may yield better MAC utilization for a given operator in a network than another. CGC must balance this compute cost with the memory cost associated with flowing the high-dimensional tensor data into the PEs' Local Register Memories (LRM). While the looping strategy will not have an impact on memory movement cost for a single operator, it may prevent the surrounding operators from being fused together to reduce overall memory movement from LRM to L2 or DDR memory. Continue reading to the section on Operator Fusion to learn more about this process.
Memory Movement Costs
The other most fundamental job of CGC is to reduce the number of memory movement operations that are needed to move data in and out of the Processing Elements' (PE) Local Register Memories (LRM).
Memory movement costs associated with moving tensors between different types of memory become exponentially more expensive the further away the tensor is moved from the compute elements, i.e. the PEs and LRM. Memory movement operations are also one of the biggest reasons that most inference hardware platforms consume so much energy.
Due to the flexibility of the Chimera architecture, CGC can automatically run many different combinations of common neural network operations with minimal usage of any memory other than the 4KBs of LRM that exists for each PE. During compile time, the CGC considers every possible way to move data between each operator in a graph and its successors in an effort to select the compute and data transfer strategies that will minimize costly memory movements out of LRM.
In many situations, off-loading some tensors to L2 or even external, e.g. DDR, memory is unavoidable. When higher-level memories must be used, data will need to be transferred back from its off-loaded memory location to the same or a new destination memory location in LRM when it is needed again for compute. We refer to this constraint as the location constraint.
Lastly, CGC must consider when the off-loaded data is needed again for compute. CGC will make an effort to hide as much of the expensive memory movement operations by running them in parallel to other compute that does not utilize the off-loaded data. Optimizing when the data must return into LRM is referred to as the timing constraint.
Graph Optimization Types
In the following sections, we attempt to conceptualize and group the different types of optimizations that the CGC performs during its numerous passes over an input graph.
When reading through these optimzations, it's important to remember that all optimziation decisions are done with the singular goal of improving compute efficiency. Trade-offs between different optimzation techniques are extremely common. CGC considers all permutations of optimizations within a given pass and will always select the one that results in the greatest compute efficiency, i.e. the simplest looping strucutres that should result in the least number of execution cycles.
General vs. Architecture-Aware Optimizations
Optimizations that are not specific to the Chimera Processor architecture are called General Optimizations or Target-Independent Optimizations. These optimizations only rely upon information available in the graph itself, are not unique to Quadric, and similar implementations may be found in other DNN compilers.
Some examples of General Optimizations performed by the CGC are:
Graph simplification / canonization: The shape and structure of the graph is optimized to simplify and fuse operators, where possible. Refer to this wikipedia article for a high-level summary of graph canonization in graph theory.
Constant folding / propagation: Remove operators with purely constant arguments, if possible. Refer to this wikipedia article for more info.
Optimizations that use context of the target Chiemra Processor architecture are called Architecture-Aware Optimizations. Conceptually, these optimizations are the most important for understanding how the CGC is able to utilize the Chimera Processor architecture to run DNN graphs so efficiently.
Some examples of Architecture-Aware Optimizations performed by the CGC are:
Operator legalization / conversion: Convert ONNX operators to optimal Chimera GPNPU-specific operators. Many ONNX operators have several different Chimera GPNPU-specific C++ implementation. Here the CGC chooses which implementation to minimize data movement.
Operator fusion: Fuse compute that operatres over the same regions of memory to keep intermediate tensors in Local Register Memory (LRM).
Prefetching weights from external memory: Boost execution performance by fetching weights before they're needed when possible, given excess L2 memory available. Conceptually this is very similar to cache prefetching.
Compute vs. Memory Optimizations
Nearly all of the Architecture-Aware Optimzations performed by CGC involve analyzing the trade-offs between:
maximizing compute efficiency, i.e. maximizing the utilization of the compute array of Processing Elements (PE) so that compute can be performed in the fewest number of cycles, and
minimizing the memory management overhead so that moving or transforming data can be performed in the fewest number of cycles.
As mentioned at the beginning of the Graph Optimzations section of this page, CGC will always select the series of optimzations that results in the most efficient compute. In the Graph Costs section, we introduce the degrees of freedom for iterating over high dimensional data and how different loop structures can result in a more or less efficient implementation of the same operation. To reiterate, the most efficient program will be the one that has the simplest loop structure with the greatest utilization rate of the PEs.
The degrees of freedom for iterating over high dimensional data combined with the general-purpose design of the PEs, enable many array operations like Convolutions, Pooling Layers, Activations, etc. to be implemented in many different ways. Likewise, the number of possible implementations for an operator that CGC can choose from during the optimization process is called the operator's degrees of freedom.
The challenge that CGC faces when optimizing for program efficiency is deciding the individual operator implementations that will result in the most efficient compute over an entire graph.
Because all operator implementations are functionally the same, an operator's implementation can be distinguished by its input array's expected layout and its output array's expected layout.
When considering which operator implementation is most optimal, CGC compares the expected input and output layouts for sequential pairs of operators in the graph. In some situations, operators with fewer or no degrees of freedom can become bottlenecks in a particular graph implementation. If there are no direct matches between any of the inputs and outputs of successive operators in a graph, CGC must inject operations to rearrange or reformat the intermediate tensor data. These injected operations that are needed to satisfy two operator implementations mismatched input and output data layouts are called adapters.
When deciding which operator implementations to use, CGC strategically weighs the cost of adapters against the compute efficiencies of different operator implementations. In some scenarios, it might make sense for CGC to purposefully select an operator implementation with a less efficient compute that does not involve using adapters to rearrange intermediate tensor data in memory. The decision is simple: if the adapter loop structures needed to rearrange the data in memory are more expensive than the loops saved using a more efficient compute, then the more efficient operator implementation is less efficient in the context of the graph and will not be selected by CGC during its optimization passes.
Below we provide some examples of Architecture-Aware memory and compute optimzations that will be elaborated upon in the next section of this doc.
Some examples of Memory Optimizations performed by CGC are:
Some examples of the Compute Optimizations performed by CGC are:
Graph Optimizations
In this section we elaborate on some of the Graph Optimizations that were mentioned in the Graph Optimization Types section of this page.
Operator Fusion
Operator fusion, sometimes also referred to as kernel or layer fusion, is a key optimization technique for all DNN compilers and execution frameworks such as Pytorch, TensorFlow, TVM, and MNN. Generally speaking, operator fusion involves reducing the "costs" of two or more operators in succession by reconsidering their scope as if they were a single operator. In practice, this typically involves reducing the memory movement overhead of a series of operators by avoiding storing/loading their intermediate tensors to/from external memory.
Fusing convolutional layers, bias adds, and activation function layers is an extremely common choice for operator fusion. Convolutions and activation functions can be decomposed into a series of matrix multiplication operations followed by element-wise operations that look something like:
If these operations are performed in three sequential steps, the graph computation would look something like (left side of image below):
Input patches (x) and kernel weights (w) are loaded from global into local memory
Tensor product of input patches (x) and weights (w) are computed, outputs are stored in intermediate tensor (m) in local memory
Intermediate tensor output (m) is written from local memory into global memory
Bias value (b) and intermediate tensor output (m) are loaded from global memory into local memory
Bias value (b) is summed with the intermediate tensor (m) to produce the convolution outputs (z)
Convolution outputs (z) are written to global memory
Convolution outputs (z) are loaded into local memory
ReLU activation function is computed for convolution outputs (z), activation outputs (y) are stored in local memory
Activation outputs (y) are written to global memory to be used by next operator(s) in the graph
If the two operations are fused, the graph computation becomes simplified to something like (right side of image above):
Input patches (x), kernel weights (w), and bias values (b) are loaded from memory
Tensor product of input patches (x) and weights (w) are computed, outputs are stored in intermediate tensor (m) in local memory
Bias values (b) are summed with the intermediate tensor (m) to produce the convolution outputs (z)
ReLU activation function is computed for convolution outputs (z), activation outputs (y) are stored in local memory
Activation outputs (y) are written to global memory to be used by next operator(s) in the graph
By representing these three operations as a single operation, four steps are able to be removed: the need to write and read the intermediate tensor (m) and the convolution output tensor (z) out of and back into local memory.
Fusion in Local Memory (FILM)
While the concepts of operator fusion remain the same, CGC adopts a more restrictive definition of operator fusion, Fusion in Local Memory or FILM, which yields even greater performance. In CGC, two successive operations are considered to be fused only if the intermediate tensor between those two operations does not get stored to or loaded from L2 memory; in other words, intermediate tensors must remain in LRM, the lowest level of memory on the Chimera platforms.
When running a program on a Chimera GPNPU core, users typically provide input data and pre-trained weights initially stored in external memory. Based on this assumption, CGC generates code for inference that brings data from external memory into L2 memory and then into the PE's' LRM as needed.
When CGC is choosing which set of operator implementations are optimal in the context of a graph, it looks at the expected input and output data layouts of all successive sets of operators in the graph. If it's within the location constraints of two operators' implementations, CGC will not send intemediate tensors out to L2 memory unless the intermediate tensor data are larger than the available LRM. By not writing intermediate tensors to L2 memory, all of the clock cycles that would have been spent transfering that data are saved and the overall graph compute efficiency is improved.
Due to the generality of the Chimera PEs, many operations can be run in succession without any data transfer at all, including within the LRM iteself. If no data transfer is required, even within LRM, then the operators are said to be trivially fusible.
Many different pairs of important operators are trivially fusible by CGC, including:
- most activation functions and any operators that precede them,
- convolutional layers with 1x1 filter sizes and any operators that precede them.
Trivial fusion is always the most optimal option for running two operations in succession, and in practice, is chosen often in a network.
For two operators that are not trivially fusible, CGC can rearrange intermediate tensor data within each PE's LRM or perform inexpensive local data transfers to neighboring PE's LRM. In practice, data can be rearranged between most operations in neural networks to satisfy the successive operator implementation's expected input data layout. If the successive operator implementation's input data layout can be satisfied and there's sufficient space in LRM to rearrange the data without writing to L2 memory, then the pair of operators can be fused by CGC.
Even if two successive operators can be fused, CGC may opt to not fuse them. If, during data layout analysis of an entire graph, CGC discovers that fusing two operators would require a more expensive memory movement operation later in the graph's execution, CGC will purposefully not fuse two operators. CGC will only choose to fuse operators that results in the most efficient program for the entire graph.
General Convolution
General Convolution is a fallback implementation automatically selected by the CGC compiler when convolution parameters fall outside the scope of optimized native implementations. The compiler analyzes each convolution operation and determines the most efficient implementation strategy based on specific hardware constraints and optimization opportunities.
Selection Criteria
The compiler selects General Convolution when one or more of the following conditions are met:
Kernel Dimensions: Non-optimized kernel sizes or kernels with unequal height and width. Currently optimized kernels include square configurations of 1×1, 2×2, 3×3, 4×4, 5×5, and 7×7 under specific conditions.
Stride Configurations: Strides other than 1×1 or 2×2, with an exception for cases where stride equals kernel size and input dimensions (creating 1×1 output).
Dilation Parameters: Dilations other than 1×1.
Group Configurations:
- For standard convolutions: when groups ≠ 1
- For depthwise convolutions: when using kernel sizes other than 3×3, 5×5, or 7×7
- For groupwise convolutions: when using kernel sizes other than 1×1, 3×3, or 5×5, or when using stride 2×2
Padding Scenarios:
- Non-standard padding configurations
- Certain asymmetric padding patterns that don't meet optimization criteria
Implementation Details
When General Convolution is applied, the compiler uses a more flexible but potentially less optimized implementation that ensures correct execution while accommodating the non-standard parameters. This approach allows models to run without modification but may not achieve the peak performance possible with optimized native implementations.
Compilation Indicators
When General Convolution is applied, you'll see two key indicators during compilation:
- Per-node warnings that identify specific operations using General Convolution:
INFO - epu - fuse_weights - No optimization for node /model.0/conv/Conv_quant with properties { kernel_size: [6, 6], channels: 16, strides:[2, 2], pat_dtype, out_layout: }. Applying General Convolution algorithm. Performance for this node can be improved significantly. Please refer to the documentation or contact support.
- A compilation summary
CGC has used general convolution applied to some nodes. Performance estimate reflects future use of native convolution support. general conv nodes: /model.0/conv/Conv_quant
These indicators help identify potential optimization opportunities within your model architecture.
Optimization Roadmap
The range of natively optimized convolution configurations is continuously expanding. Operations currently using General Convolution may automatically benefit from performance improvements in future SDK releases as additional native implementations are added, without requiring model modifications.
FILM Regions
When processing a neural network graph, CGC typically fuses a pair of operations at a time and then continues to compose as many fused operators as possible. The series of operators of a neural network that are fused in local memory and therefore have no intermediate tensors written to or read from L2 or external memory are called FILM regions.
In practice, FILM regions typically consist of:
- a sequence of several convolutions mixed with quantization and activation functions, or
- a single fully-connected or matrix multiply operation, mixed with quantization and activation functions, or
- a single custom CCL operator.
Note: FILM regions may additionally consist of some number of prefetches.
Note: FILM region boundaries are consistent for a given neural network and array-size combination across all other parameters of a hardware configuration. For example, ResNet-50 on a QB16 will have the same FILM region boundaries for any L2M size, but it may have different FILM region boundaries than a ResNet-50 compiled to target a QB4.
FILM regions boundaries can be observed in the generated C++ for a given network on lines such as the following:
cgc::profileStart("region: 1 / 7");
Here is an entire FILM region for a MobileNet-V2 model, compiled for QB16 with 8 MB of L2M:
cgc::profileStart("region: 2 / 7");
UnbufferedReadFlow<TensorAccessor<MinRoiDescriptor<OcmTensor<std::int8_t, 1, 3, 64, 64>, AxisGroup<Direction::Channel>, Granularity::Square, true, 2, 2, 1, DefaultLRMType, -14>, AxisGroup<Direction::Width, Direction::Height>>, OcmTensor<std::int8_t, 1, 3, 224, 224>> ocm_tensor_0_flow_1(ocm_tensor_0, 0, 0, 0, 0);
OcmTensor<v2i8, 1, 48, 112, 112> ocm_tensor_1;
ocm_tensor_1.ptr = 516096;
UnbufferedWriteFlow<TensorAccessor<MinRoiDescriptor<OcmTensor<v2i8, 1, 48, 32, 32>, AxisGroup<Direction::Channel>, Granularity::Square, false, 1, 1, 1>, AxisGroup<Direction::Width, Direction::Height>>, OcmTensor<v2i8, 1, 48, 112, 112>> ocm_tensor_1_flow_0(ocm_tensor_1, 0, 0, 0, 0);
for (int32_t tb_y1 = 0; tb_y1 < 4; ++tb_y1) {
for (int32_t tb_x1 = 0; tb_x1 < 4; ++tb_x1) {
container::NDArray<qVar_t<int8_t>, 27> ocm_tensor_0_rf;
ocm_tensor_0_flow_1.read(ocm_tensor_0_rf);
dataAggregationStrideOf2x2ConvWithDataMov3x3<OcmTensor<std::int8_t, 1, 3, 64, 64>>(ocm_tensor_0_rf);
container::NDArray<qVar_t<int8_t>, 32> T_contrib_epu_qlinear_conv2d_rf;
BroadcastFlow<TensorAccessor<MinRoiDescriptor<OcmTensor<std::int8_t, 1, 1, 1, 1280>, AxisGroup<>, Granularity::Row, false, 1, 1, 1>>, OcmTensor<std::int8_t, 1, 1, 1, 1280>, OcmTensor<std::int8_t, 1, 1, 1, 1280>, 1> const_tensor_0_ocm_flow_0(const_tensor_0_ocm);
for (int32_t ch1 = 0; ch1 < 32; ++ch1) {
qVar_t<int32_t> _0 = nn::convTileBlockInt8<std::int32_t, 27, 1, 0, false>(ocm_tensor_0_rf) /* /features/features.0/features.0.0/Conv_quant */ ;
qVar_t<int32_t> _11 = qBroadcast<0, std::int32_t, BroadcastAction::POP>;
T_contrib_epu_qlinear_conv2d_rf[(ch1)] = ((qVar_t<int8_t>)math::min(math::max((cgc::fxRoundPosInf<2>(
#ifdef __epu__
__builtin_epu_fxsmul((_0 + _11), 76247355, 29)
#else
0
#endif
) + -128), -128), 127));
}
container::NDArray<qVar_t<int8_t>, 32> T_contrib_epu_qlinear_conv2d_rf1;
BroadcastFlow<TensorAccessor<MinRoiDescriptor<OcmTensor<std::int8_t, 1, 1, 1, 1024>, AxisGroup<>, Granularity::Row, false, 1, 1, 1>>, OcmTensor<std::int8_t, 1, 1, 1, 1024>, OcmTensor<std::int8_t, 1, 1, 1, 1024>, 1> const_tensor_1_ocm_flow_0(const_tensor_1_ocm);
cgc::initNonParticipatingCores<112, 112, 32, 1, 1, 1, 1, 1, false>(T_contrib_epu_qlinear_conv2d_rf, ((qVar_t<int8_t>)-128), tb_y1, tb_x1);
for (int32_t ch2 = 0; ch2 < 32; ++ch2) {
qVar_t<int32_t> _2 = nn::groupwiseConvTileBlockInt8<std::int32_t, 1, 3, 1, 0, false>(T_contrib_epu_qlinear_conv2d_rf, ch2) /* /features/features.1/conv/conv.0/conv.0.0/Conv_quant */ ;
qVar_t<int32_t> _3 = qBroadcast<0, std::int32_t, BroadcastAction::POP>;
T_contrib_epu_qlinear_conv2d_rf1[(ch2)] = ((qVar_t<int8_t>)math::min(math::max((cgc::fxRoundPosInf<2>(
#ifdef __epu__
__builtin_epu_fxsmul((_2 + _3), 134911888, 29)
#else
0
#endif
) + -128), -128), 127));
}
container::NDArray<qVar_t<int8_t>, 16> T_contrib_epu_qlinear_conv2d_rf2;
BroadcastFlow<TensorAccessor<MinRoiDescriptor<OcmTensor<std::int8_t, 1, 1, 1, 640>, AxisGroup<>, Granularity::Row, false, 1, 1, 1>>, OcmTensor<std::int8_t, 1, 1, 1, 640>, OcmTensor<std::int8_t, 1, 1, 1, 640>, 1> const_tensor_2_ocm_flow_0(const_tensor_2_ocm);
for (int32_t ch3 = 0; ch3 < 16; ++ch3) {
qVar_t<int32_t> _4 = nn::convTileBlockInt8<std::int32_t, 32, 1, 0, false>(T_contrib_epu_qlinear_conv2d_rf1) /* /features/features.1/conv/conv.1/Conv_quant */ ;
qVar_t<int32_t> _5 = qBroadcast<0, std::int32_t, BroadcastAction::POP>;
T_contrib_epu_qlinear_conv2d_rf2[(ch3)] = ((qVar_t<int8_t>)math::min(math::max((cgc::fxRoundPosInf<2>(
#ifdef __epu__
__builtin_epu_fxsmul((_4 + _5), 4853152, 29)
#else
0
#endif
) + 11), -128), 127));
}
container::NDArray<qVar_t<int8_t>, 96> T_contrib_epu_qlinear_conv2d_rf3;
BroadcastFlow<TensorAccessor<MinRoiDescriptor<OcmTensor<std::int8_t, 1, 1, 1, 2304>, AxisGroup<>, Granularity::Row, false, 1, 1, 1>>, OcmTensor<std::int8_t, 1, 1, 1, 2304>, OcmTensor<std::int8_t, 1, 1, 1, 2304>, 1> const_tensor_3_ocm_flow_0(const_tensor_3_ocm);
for (int32_t ch4 = 0; ch4 < 96; ++ch4) {
qVar_t<int32_t> _6 = nn::convTileBlockInt8<std::int32_t, 16, 1, 0, false>(T_contrib_epu_qlinear_conv2d_rf2) /* /features/features.2/conv/conv.0/conv.0.0/Conv_quant */ ;
qVar_t<int32_t> _7 = qBroadcast<0, std::int32_t, BroadcastAction::POP>;
T_contrib_epu_qlinear_conv2d_rf3[(ch4)] = ((qVar_t<int8_t>)math::min(math::max((cgc::fxRoundPosInf<2>(
#ifdef __epu__
__builtin_epu_fxsmul((_6 + _7), 1183293723, 29)
#else
0
#endif
) + -128), -128), 127));
}
ocm_tensor_1_flow_0.write(T_contrib_epu_qlinear_conv2d_rf3);
}
}
cgc::profileStop("region: 2 / 7");
The above FILM region consists of 4 fused convolutions - one depthwise convolution (the call to nn::groupwiseConvTileBlockInt8 and three non-depthwise convolutions (the calls to nn::convTileBlockInt8). For each of these CCL API calls, there is a comment at the end of the line of code that corresponds to the name of the ONNX node that was implemented by this call.
For example, when searching in the Netron ONNX graph visualization tool for the name in the comment next to the first call:
the node that was implemented will be highlighted in the graph:
Prefetched Transfers from External Memory
Inference programs contain numerous pre-defined constants, like model weights, which are typically loaded from external memory. Memory transfers from external memory often contribute the most overhead to programs; therefore, timing these memory transfers requires signficant consideration from CGC to keep programs efficient.
When scheduled properly, memory transfers and computation can be performed in parallel by the Chimera architecture. CGC strives to leverage this feature aggressively to "hide" the costs of memory transfers from external memory by scheduling them concurrently with computation, but still satisfying the timing contraint of the operator. As long as data is in local memory on the Chimera core when it is needed for compute, the processor can be maximally utilitzed which results in the most efficient programs. If data arrives late to local memory, the processor stalls while memory transfer is completing which hurts performance. Any cycles spent delaying computation while waiting for necessary data to arrive are called memory stall cycles.
CGC takes two approaches to timing data transfers from external memory:
On-Demand transfers or transfers that are intiated when an operation needs data, and
Prefetched transfers or transfers that are initiated prior to an operation's execution to reduce or eliminate memory stall cycles.
On-demand transfers are simple and straight-forward to implement, but they do not hide any of the memory transfer costs. While not performant, this approach is useful for simplified debugging sessions.
Prefetched transfers are more complicated to implement, but they will always result in more performant programs.
To see how impactful memory transfers timings can be, let’s consider this simple computational graph:
In the graph above, var1 and var2 are variables and const is the constant we are interested in fetching from external memory (in this case, our external memory is DDR).
The sequence diagram below illustrates an On-Demand Transfer. Notice how the Chimera processor must wait for data to arrive before being able to perform the add operation of the computational graph.
The next sequence diagram illustrates a Prefetched Transfer. In this example, CGC has identified a location in the source code where data could be feteched early from external memory (DDR) to L2 memory. The const memory block is prefetched into L2 memory while the mul operation between var1 and var2 is happening. When the processor is ready to begin the add operation, const is already in L2 memory and the processor wastes no time between the mul and add operations in the computational graph.
