Traditional Convolutions
Convolution [Filter NxN, Stride 1x1]
File:
/src/neuralNetBlocks/conv.hppLines 800–822 /**
* @brief Computes convolutions in int8 precision with a height, width stride of 2, 2.
* @ingroup convolutions
* @param qData The activations of the input.
*
* @tparam T The type of the input values.
* @tparam numInCh The type of the input values.
* @tparam filterSize The filter size of the convolution.
* @tparam shiftAmount The amount to shift the output of the convolution by.
* @tparam useVaping Whether or not the convolution will implement a Virtual Array Partitioning (VAP) strategy.
* @tparam sameWeights If 'True' the weights are popped once and used through out the convolution.
* @tparam NDArrayType The input array type of `qData`.
* @return qOutput A single output channel.
**/
// clang-format on
template <typename T,
std::uint32_t numInCh,
std::uint32_t filterSize = 1,
std::uint32_t shiftAmount = 0,
bool useVaping = false,
bool sameWeights = false,
typename NDArrayType>
INLINE qVar_t<T> convStrideOf2x2TileBlockInt8(NDArrayType& qData) {
Convolution [Filter 5x5, Stride 2x2]
File:
/src/neuralNetBlocks/conv.hppLines 618–656 /**
* @brief Performs a 5x5 conv with stride of 2 on a 2 channel packed input.
* This operation performs 4 3x3 convolutions to get the 5x5 conv output. The 4 strided input tiles can be mapped as
* follows:
*
* ```
* | 0, 1 | 0, 1 | 0, [1] |
* | 2, 3 | 2, 3 | 2, [3] |
* ---------------------------------
* | 0, 1 | 0, 1 | 0, [1] |
* | 2, 3 | 2, 3 | 2, [3] |
* ---------------------------------
* | 0, 1 | 0, 1 | 0, [1] |
* |[2], [3] | [2], [3] | [2], [3] |
* ```
*
* The weights corresponding to indexes inbetween [] need to be zeros for the corresponding convs.
*
* @tparam numInChn The number of input channels.
* @tparam filterSize The size of the convolutional filter.
* @tparam T The output qVar type
* @tparam shiftAmount The amount to shift the output of the convolution by
* @tparam useVaping Whether or not the convolution will implement a Virtual Array Partitioning (VAP)
* @tparam sameWeights if True the weights are popped once and used through out the convolution.
* @tparam heightOffset Padding related offset in height
* @tparam widthOffset Padding related offset in width
* @tparam NDArrayType The input array type of `qData`.
* @param qData The input data fetched, the neighbor moving will be handled inside the function
* @return qResult Convolution output with 5x5 filter size and stride of 2x2 for input `qData`.
*/
// clang-format on
template <typename T,
std::int32_t numInChn,
std::uint32_t filterSize = 1,
std::uint32_t shiftAmount = 0,
bool useVaping = false,
bool sameWeights = false,
typename NDArrayType>
INLINE qVar_t<T> conv5x5StrideOf2x2TileBlockInt8(NDArrayType& qData) {
Convolution [Filter NxN, Stride 2x2]
File:
/src/neuralNetBlocks/conv.hppLines 759–778 /**
* @brief Computes convolutions in int8 precision with a height, width stride of 2, 2.
* @ingroup convolutions
* @param qData The activations of the input.
* @tparam T The type of the input values.
* @tparam numInCh The type of the input values.
* @tparam filterSize The filter size of the convolution.
* @tparam shiftAmount The amount to shift the output of the convolution by.
* @tparam useVaping Whether or not the convolution will implement a Virtual Array Partitioning (VAP) strategy.
* @tparam sameWeights If 'true' the weights are popped once and used through out the convolution.
* @return qOutput A single output channel.
**/
// clang-format on
template <typename T,
std::uint32_t numInCh,
std::uint32_t filterSize = 1,
std::uint32_t shiftAmount = 0,
bool useVaping = false,
bool sameWeights = false>
INLINE qVar_t<T> convStrideOf2x2TileBlockInt8(qVar_t<std::int8_t> qData[]) {
Convolution [Filter NxN, Stride 2x2]
File:
/src/neuralNetBlocks/conv.hppLines 800–822 /**
* @brief Computes convolutions in int8 precision with a height, width stride of 2, 2.
* @ingroup convolutions
* @param qData The activations of the input.
*
* @tparam T The type of the input values.
* @tparam numInCh The type of the input values.
* @tparam filterSize The filter size of the convolution.
* @tparam shiftAmount The amount to shift the output of the convolution by.
* @tparam useVaping Whether or not the convolution will implement a Virtual Array Partitioning (VAP) strategy.
* @tparam sameWeights If 'True' the weights are popped once and used through out the convolution.
* @tparam NDArrayType The input array type of `qData`.
* @return qOutput A single output channel.
**/
// clang-format on
template <typename T,
std::uint32_t numInCh,
std::uint32_t filterSize = 1,
std::uint32_t shiftAmount = 0,
bool useVaping = false,
bool sameWeights = false,
typename NDArrayType>
INLINE qVar_t<T> convStrideOf2x2TileBlockInt8(NDArrayType& qData) {
Depthwise Convolutions
Depthwise Convolution [Filter NxN, Stride 1x1]
File:
/src/neuralNetBlocks/conv.hppLines 1180–1205 /**
* @brief Computes Groupwise convolutions in int8 precision
* @ingroup convolutions
* @param qData The activations of the input.
* @param outputIteration If function is looped for a very large convolution that must run in multiple iterations, `outputIteration` is used to calculate the offset in the output tensor's memory where partial convolutional outputs should be stored.
*
* @tparam T The type of the input values.
* @tparam groupSize The number of channels in a group.
* @tparam filterSize The filter size of the convolution.
* @tparam stride The stride of the convolution.
* @tparam shiftAmount The amount to shift the output of the convolution by.
* @tparam useVaping Whether or not the convolution will implement a Virtual Array Partitioning (VAP) strategy.
* @tparam sameWeights If 'True' the weights are popped once and used through out the convolution.
* @tparam NDArrayType The input array type of `qData`.
* @return qOutput A single output channel.
**/
// clang-format on
template <typename T,
std::uint32_t groupSize = 4,
std::uint32_t filterSize = 3,
std::uint32_t stride = 1,
FracRepType shiftAmount = 0,
bool useVaping = false,
bool sameWeights = false,
typename NDArrayType>
INLINE qVar_t<T> groupwiseConvTileBlockInt8(NDArrayType& qData, std::int32_t outputIteration = 0) {
Depthwise Convolution [Filter 3x3, Stride 2x2]
File:
/src/neuralNetBlocks/conv.hppLines 985–1091 /**
* @brief Depthwise conv function for 3x3 and stride 2x2 case.
*
* The data is laid out in such a way that two outputs are computed at a time using two intrinsic calls:
*
* A Typical 3x3 filter is labeled as:
*
* ```
* 0 │ 1 │ 2
* 3 │ 4 │ 5
* 6 │ 7 │ 8
* ```
*
* The Strided flows bring in packed rows and columns in a single core
* Each core has the bottom right 2x2 grid of activations present for the conv.
* Due to the v2i8 nature of the packing, consecutive channels are brought in together.
* This yields the following sequential data layout in qData:
* `4_0` `4_1` `5_0` `5_1` `7_0` `7_1` `8_0` `8_1`
*
* To compute the rest of the 3x3 convolution, local data movement is used
* to pull data in 4 byte increments from the West neighbor core which contains
* `3_0` `3_1` and `6_0` `6_1`, then data from the North neighbor core which contains
* `1_0` `1_1` `2_0` `2_1` and finally data from the North-West neighbor core which contains
* `0_0` `0_1`. Post Local Data Movement, the data is arranged in qMove as:
* `x x 3_0 3_1 x x 6_0 6_1`
* `1_0 1_1 2_0 2_1 x x 0_0 0_1`
*
* heightOffset/widthOffset > 0 implies how much to move the data North/West respectively. For example, if
* the border values are 0 then we would want the input (not represented below in its strided form which is the
* actual input) to go from.
*
* > <b>Note:</b> The double lines are border cores('||' or '=') and a square made of single lines represents the same
* convolution across 4 tiles corresponding to different filter/input indices. Each box corresponds to a conv in
* original representation without any offset.
*
* ```
* 0 0 ║ 0 0 0 0 0 0 ...
* 0 0 ║ 0 0 0 0 0 0 ...
* ════╬═════════════════════════
* ║ ┌─────┐ ┌─────┐ ┌─────┐
* 0 0 ║ │ 4 5 │ │ 4 5 │ │ 4 5 │ ...
* 0 0 ║ │ 7 8 │ │ 7 8 │ │ 7 8 │ ...
* ║ └─────┘ └─────┘ └─────┘
* ║ ┌─────┐ ┌─────┐ ┌─────┐
* 0 0 ║ │ 4 5 │ │ 4 5 │ │ 4 5 │ ...
* 0 0 ║ │ 7 8 │ │ 7 8 │ │ 7 8 │ ...
* ║ └─────┘ └─────┘ └─────┘
* ║ ┌─────┐ ┌─────┐ ┌─────┐
* 0 0 ║ │ 4 5 │ │ 4 5 │ │ 4 5 │ ...
* 0 0 ║ │ 7 8 │ │ 7 8 │ │ 7 8 │ ...
* ║ └─────┘ └─────┘ └─────┘
* ```
*
* to as follows for heightOffset = widthOffset = 1
*
* ```
* 0 0 ║ 0 0 0 0 0 ...
* ┌───╫───┐ ┌─────┐ ┌─────┐
* 0 │ 4 ║ 5 │ │ 4 5 │ │ 4 5 │ ...
* ══╪═══╬═══╪═╪═════╪═╪═════╪═
* 0 │ 7 ║ 8 │ │ 7 8 │ │ 7 8 │ ...
* └───╫───┘ └─────┘ └─────┘
* ┌───╫───┐ ┌─────┐ ┌─────┐
* 0 │ 4 ║ 5 │ │ 4 5 │ │ 4 5 │ ...
* 0 │ 7 ║ 8 │ │ 7 8 │ │ 7 8 │ ...
* └───╫───┘ └─────┘ └─────┘
* ┌───╫───┐ ┌─────┐ ┌─────┐
* 0 │ 4 ║ 5 │ │ 4 5 │ │ 4 5 │ ...
* 0 │ 7 ║ 8 │ │ 7 8 │ │ 7 8 │ ...
* └───╫───┘ └─────┘ └─────┘
* ```
*
* In the first diagram the top left corner conv gets converted to:
* ```
* 0 ║ 0 0 4 ║ 5 4
* ══╬════ ➤ ══╬════
* 0 ║ 4 5 7 ║ 8 7
* 0 ║ 7 8 4 ║ 5 4
* ```
*
* which means the 8 7 5 4 should be on the same core across the 4 tiles in that order
*
* As can be deciphered for the above diagram, this would require the following operations
*
* `4_0`, `4_1` -> Move West, Move North
* `5_0`, `5_1` -> Move North
* `7_0`, `7_1` -> Move West
* `8_0`, `8_1` ->
*
* and would also lead to a change in the of ops to `8_0` `8_1` `7_0` `7_1` `5_0` `5_1` `4_0` `4_1`.
* All this done with the help of applyCoreOffsetWE/NS calls for neighbor movements.
*
* @tparam heightOffset Padding related offset in height.
* @tparam widthOffset Padding related offset in width.
* @tparam useVaping Use VAPing (Virtual Array Partitioning) in the convolution.
* @tparam InNDArrayType Type of the input array (`qData`).
* @tparam OutNDArrayType Type of the output array (`qResult`).
* @param qData [in] The input data fetched, the neighbor movement within the function.
* @param outputIteration The output channel index for the first channel to be processed.
* @return A vector of `core_array::numAccumSums` integers.
*/
// clang-format on
template <std::int32_t heightOffset = 0,
std::int32_t widthOffset = 0,
bool useVaping = false,
typename InNDArrayType>
INLINE qVar_t<vNsi> depthwiseConv3x3StrideOf2x2TileBlock(InNDArrayType& qData, std::uint32_t outputIteration) {
Depthwise Dilated Convolution [Filter NxN, Dilation Factor DxD]
File:
src/neuralNetBlocks/conv.hppLines 1937–1970 /**
* @brief Computes Depthwise dilated convolutions in int8 precision. This operation is DDR only.
*
* @tparam dilationFactor In pytorch this is the spacing between kernel elements. In the quadric implementation this is the spacing between input tensor elements. Effectively both are same. Currently this operation only supports 1, 2, and 4 dilation factors.
* @tparam filterSize The filter size of the convolution. Only filterSize of 3 is supported.
* @tparam numFracBits number of fractional bits for quantization parameters
* @tparam DdrInOutShape The shape of the input/output DDR tensor
* @tparam DdrWeightShape The shape of the weight DDR tensor
* @tparam OcmAllocatorType Type of OCM memory allocator
*
* @param ddrIn The DDR input tensor
* @param ddrWeights The DDR weight tensor (contains weights for conv as well as bias)
* @param ddrOut The DDR output tensor
* @param ocmMemAlloc ocm allocator to be used locally to allocate ocm memory space
* @param scale Scale parameter for quantization
* @param xZeroPoint Zero point quantization parameter for input tensor (needs to be 0)
* @param wZeroPoint Zero point quantization parameter for weight tensor (needs to be 0)
* @param yZeroPoint Zero point quantization parameter for output tensor
*/
// clang-format on
template <std::int32_t dilationFactor,
std::uint32_t filterSize = 3,
FracRepType numFracBits = 31,
typename DdrInOutShape,
typename DdrWeightShape,
typename OcmAllocatorType>
INLINE void depthwiseDilatedConvInt8(DdrInOutShape& ddrIn,
DdrWeightShape& ddrWeights,
DdrInOutShape& ddrOut,
OcmAllocatorType& ocmMemAlloc,
FixedPoint32<numFracBits> scale,
std::int8_t xZeroPoint,
std::int8_t wZeroPoint,
std::int8_t yZeroPoint) {
Groupwise Convolutions
Groupwise Convolution [Filter NxN, Stride MxM]
File:
/src/neuralNetBlocks/conv.hppLines 1180–1205 /**
* @brief Computes Groupwise convolutions in int8 precision
* @ingroup convolutions
* @param qData The activations of the input.
* @param outputIteration If function is looped for a very large convolution that must run in multiple iterations, `outputIteration` is used to calculate the offset in the output tensor's memory where partial convolutional outputs should be stored.
*
* @tparam T The type of the input values.
* @tparam groupSize The number of channels in a group.
* @tparam filterSize The filter size of the convolution.
* @tparam stride The stride of the convolution.
* @tparam shiftAmount The amount to shift the output of the convolution by.
* @tparam useVaping Whether or not the convolution will implement a Virtual Array Partitioning (VAP) strategy.
* @tparam sameWeights If 'True' the weights are popped once and used through out the convolution.
* @tparam NDArrayType The input array type of `qData`.
* @return qOutput A single output channel.
**/
// clang-format on
template <typename T,
std::uint32_t groupSize = 4,
std::uint32_t filterSize = 3,
std::uint32_t stride = 1,
FracRepType shiftAmount = 0,
bool useVaping = false,
bool sameWeights = false,
typename NDArrayType>
INLINE qVar_t<T> groupwiseConvTileBlockInt8(NDArrayType& qData, std::int32_t outputIteration = 0) {
FP16 Convolutions
FP16 Convolution [Filter NxN, Stride 1x1 - Auto Buffer]
File:
/src/neuralNetBlocks/conv.hppLines 257–286 /**
* @brief this is a wrapper of the other version of convTileBlockFp16, this one allocates its internal buffer when
* and should be preferred in handwritten code.
*
* @tparam OutT The output data type, which must be a FixedPoint type.
* @tparam numInCh The number of input channels for the convolution.
* @tparam filterSize The size of the convolution filter. Defaults to 1.
* Supported sizes are 1, 3, and 5.
* @tparam useVaping A boolean flag to enable vaping. Defaults to false.
* @tparam sameWeights if True the weights are popped once and used through out the convolution.
* @tparam inSize inferred, the size of input array
* @tparam drainBroadcastFifo When true, consumes leftover broadcast FIFO elements after the MAC
* operation. Defaults to false.
* @tparam numFracBits inferred, the number of frac bits of input
*
* @param qData An NDArray of qVar_t<FixedPoint32<numFracBits>> representing the
* input data for the convolution.
*
* @return A qVar_t<OutT> containing the result of the convolution after
* shifting.
*/
template <typename OutT,
std::uint32_t numInCh,
std::uint32_t filterSize = 1,
bool useVaping = false,
bool sameWeights = false,
std::size_t inSize,
bool drainBroadcastFifo = false,
FracRepType numFracBits>
INLINE qVar_t<OutT> convTileBlockFp16(container::NDArray<qVar_t<FixedPoint32<numFracBits>>, inSize>& qData) {
FP16 Groupwise Convolutions
FP16 Groupwise Convolution [Filter NxN, Stride MxM]
File:
/src/neuralNetBlocks/conv.hppLines 1279–1303 /**
* @brief Computes Groupwise convolutions in fp16 precision
* @ingroup convolutions
* @param qData The activations of the input.
* @param outputIteration If function is looped for a very large convolution that must run in multiple iterations, `outputIteration` is used to calculate the offset in the output tensor's memory where partial convolutional outputs should be stored.
*
* @tparam T The type of the input values.
* @tparam groupSize The number of channels in a group.
* @tparam filterSize The filter size of the convolution.
* @tparam stride The stride of the convolution.
* @tparam shiftAmount The amount to shift the output of the convolution by.
* @tparam useVaping Whether or not the convolution will implement a Virtual Array Partitioning (VAP) strategy.
* @tparam sameWeights If 'True' the weights are popped once and used through out the convolution.
* @tparam NDArrayType The input array type of `qData`.
* @return qOutput A single output channel.
**/
// clang-format on
template <typename T,
std::uint32_t groupSize = 4,
std::uint32_t filterSize = 3,
std::uint32_t stride = 1,
bool useVaping = false,
bool sameWeights = false,
typename NDArrayType>
INLINE qVar_t<T> groupwiseConvTileBlockFp16(NDArrayType& qData, std::int32_t outputIteration = 0) {
