Quadric's Chimera architecture supports three different data access patterns from L2 Memory ↔︎ LRM (qVar) for data movement between different levels of memory hierarchy.
The Broadcast data access pattern or broadcasting for short, is a one-to-many data transfer in which each element within a region of interest (ROI) is written from L2 memory to the LRM of every PE in the array. Broadcasting data from L2 Memory to all PEs on the array is done using the dedicated Broadcast Bus which enables a user to send scalar data to all PEs with minimal clock cycles. The Broadcast Bus size is 64 bits and is only available on the west side of the Chimera GPNPU array.
BroadcastFlow Class
The class for broadcast, it provides capacity to stage a MinROI (required before consuming and a MinROI needs to be finished before reading the next can be started), read part or whole of MinROI in the multiple of 8 bytes, and maintain the offsets when moving MinROI inside the ROI.
/src/core/flows/broadcast_flow.hppLines 342–364 /**
* @brief The class for broadcast, it provides capacity to stage a MinROI (required before consuming and a MinROI
* needs to be finished before reading the next can be started), read part or whole of MinROI in the multiple of 8
* bytes, and maintain the offsets when moving MinROI inside the ROI
*
* @tparam Access the Accessor defining the MinRoi and RoiAxisGroup, other members will be ignored for broadcast
* @tparam OCMShape The OCM tensor type.
* @tparam RoiShape The ROI shape.
* @tparam numberOfPartitions Number of VAP partitions to create. Use `1` if not needed.
* @tparam autoStage Whether or not to call `stageMROI()` to stage the broadcast bus automatically. By default, it stages when: 1) the class is constructed 2) when a read is attempted and the previous MinROI is completely consumed. It comes with an overhead, this option should be disabled when lots of small read are performed to save cycles.
* @note RuntimeTensor support:
* - For RuntimeTensors, PLS counts are auto-clamped to min(MinRoi, tensor dims) at each staging.
* @tparam repetitions The number of times to repeat the flow.
* @tparam repetitionAxis The axis to repeat.
*/
template <typename Access,
typename OCMShape,
typename RoiShape = OCMShape,
std::uint8_t numberOfPartitions = 1,
bool autoStage = true,
std::int32_t repetitions = 1,
Direction repetitionAxis = Direction::Unused>
struct BroadcastFlow : detail::AutoStage<autoStage> {
Utility Methods
/src/core/flows/broadcast_flow.hppLines 414–425 /**
* @brief Read from a PE array via broadcast.
*
* The memory being read must be a multiple of 8 bytes and cannot be larger than what is
* remaining in the current staged MinROI (or the whole MinROI if starting a new MinROI).
*
* @tparam T The data type to read. Default is `OCMShape::elemType`.
* @tparam size The number elements to read.
* @return container::NDArray<qVar_t<T>, size>
*/
template <typename T = typename OCMShape::elemType, std::size_t size = numTiles<T>()>
INLINE container::NDArray<qVar_t<T>, size> read() {
Examples
In this example, we setup a BroadcastFlow for an internal DMA transfer which stores the broadcast values in an NDArray named broadcastRead. We then iterate over the broadcast values and store the sum of those values in an output variable out.
/tests/static_data_tests/broadcast_stream_unit.cppLines 129–137 BroadcastFlow<Access, OcmInShape> broadcastFlow(ocmInp);
constexpr std::size_t numTiles = broadcastFlow.numTiles();
const container::NDArray<qVar_t<int>, 256UL> broadcastRead = broadcastFlow.read();
qVar_t<std::int32_t> out = 0;
for(std::size_t i = 0; i < numTiles; i++) {
out += broadcastRead[i];
}
Queue a Tensor to be broadcasted using the Broadcast bus.
/src/core/flows/broadcast_flow.hppLines 431–436 /**
* @brief Setup the broadcast bus for the next MinROI.
* For RuntimeTensors, counts are auto-clamped to min(MinRoi, tensor) per dimension.
* For static tensors, compile-time counts from DataAccessTraits are used.
*/
void stageMROI() {
Examples
/tests/static_data_tests/broadcast_stream_unit.cppLines 107–117 BroadcastFlow<Access16, OcmIn16Shape, OcmIn16Shape, 1, false> broadcastFlow(ocmInp16);
broadcastFlow.stageMROI();
qVar_t<std::int32_t> out = 0;
for(std::int32_t i = 0; i < numBroadcasts16; i++) {
container::NDArray<qVar_t<std::int16_t>, numRegs16> arr = broadcastFlow.read<std::int16_t, numRegs16>();
for(std::size_t i = 0; i < numRegs16; i++) {
out += arr[i];
}
}
Public Members
/src/core/flows/broadcast_flow.hppLines 402–409 /**
* @brief Returns the size of the MinROI in bytes.
*
* @tparam T The data type used in the Broadcast operation. Default is `OCMShape::elemType`.
* @return std::int32_t The size of the MinROI in bytes.
*/
template <typename T = typename OCMShape::elemType>
static constexpr std::int32_t numTiles() {
Examples
/tests/static_data_tests/broadcast_stream_unit.cppLines 132–137 const container::NDArray<qVar_t<int>, 256UL> broadcastRead = broadcastFlow.read();
qVar_t<std::int32_t> out = 0;
for(std::size_t i = 0; i < numTiles; i++) {
out += broadcastRead[i];
}
How to use broadcasting?
- First declare the File:
/src/core/flows/broadcast_flow.hppSymbol: BroadcastFlowBroadcastFlow
BroadcastFlow<Access, OcmInOutShape> broadcastFlow(ocmInp);Then consume the broadcasted values.
container::NDArray<qVar_t<std::int32_t>, broadcastFlow.numTiles()> broadCastRead = broadcastFlow.read();
Broadcasting is a technique that is commonly used in conjunction with convolution-related operators. In particular, it is utilized to transfer convolution parameters, such as weights and biases, to the LRM.
To illustrate this process, consider the following code snippet:
/tests/static_data_tests/nn_conv3x3_i8_v2.cppLines 53–64 for(std::uint32_t tileNum = 0; tileNum < inputFlow.numMROIs(); tileNum++) {
auto qInput = inputFlow.read();
BroadcastFlow<BroadcastAccess, OcmWeightTensorShape> broadcastFlow(weightsTensor);
for(std::uint32_t channel = 0; channel < OcmOutShape::NUM_CHN; channel++) {
qVar_t<std::int32_t> accumOutput = nn::convTileBlockInt8<std::int32_t, OcmInputShape::NUM_CHN, 3>(qInput);
auto scaledOutput = static_cast<std::int32_t>(((accumOutput * scale) + bias) >> shift);
qOutput[channel] = static_cast<std::int8_t>(
scaledOutput > int8High ? int8High : (scaledOutput < int8Low ? int8Low : scaledOutput));
}
outputFlow.write(qOutput);
}
The provided code utilizes the
/src/core/flows/broadcast_flow.hppSymbol: BroadcastFlowBroadcastFlowocmWeights tensor to the PE. Subsequently, the convolution operation is performed for each channel in the output tensor using the /src/neuralNetBlocks/conv.hppSymbol: convTileBlockInt8convTileBlockInt8It should be noted that the
/src/neuralNetBlocks/conv.hppSymbol: convTileBlockInt8convTileBlockInt8As a result, the user need not concern themselves with the low-level details of the broadcast process. Instead, they can simply utilize the
/src/neuralNetBlocks/conv.hppSymbol: convTileBlockInt8convTileBlockInt8VAP (Virtual Array Partitioning)
The VAP (Virtual Array Partition) feature is an advanced technique that enables the efficient utilization and allocation of computational resources within the array. By partitioning the tile into smaller chunks, it allows broadcasting different weights for each partition, thereby optimizing the processing capabilities by increasing the core utilization.
Usage
To enable VAP, you can use our standard Broadcast API with the appropriate numberOfPartitions parameter:
/src/core/flows/broadcast_flow.hppLines 342–364 /**
* @brief The class for broadcast, it provides capacity to stage a MinROI (required before consuming and a MinROI
* needs to be finished before reading the next can be started), read part or whole of MinROI in the multiple of 8
* bytes, and maintain the offsets when moving MinROI inside the ROI
*
* @tparam Access the Accessor defining the MinRoi and RoiAxisGroup, other members will be ignored for broadcast
* @tparam OCMShape The OCM tensor type.
* @tparam RoiShape The ROI shape.
* @tparam numberOfPartitions Number of VAP partitions to create. Use `1` if not needed.
* @tparam autoStage Whether or not to call `stageMROI()` to stage the broadcast bus automatically. By default, it stages when: 1) the class is constructed 2) when a read is attempted and the previous MinROI is completely consumed. It comes with an overhead, this option should be disabled when lots of small read are performed to save cycles.
* @note RuntimeTensor support:
* - For RuntimeTensors, PLS counts are auto-clamped to min(MinRoi, tensor dims) at each staging.
* @tparam repetitions The number of times to repeat the flow.
* @tparam repetitionAxis The axis to repeat.
*/
template <typename Access,
typename OCMShape,
typename RoiShape = OCMShape,
std::uint8_t numberOfPartitions = 1,
bool autoStage = true,
std::int32_t repetitions = 1,
Direction repetitionAxis = Direction::Unused>
struct BroadcastFlow : detail::AutoStage<autoStage> {
To enable VAP, you can use our standard Broadcast API with the appropriate
/src/core/flows/broadcast_flow.hppSymbol: numberOfPartitionsnumberOfPartitionsNote: VAP is only supported for the following pairs of
/src/core/flows/broadcast_flow.hppSymbol: numberOfPartitionsnumberOfPartitions/src/core/flows/broadcast_flow.hppSymbol: numberOfPartitionsnumberOfPartitions| Chimera GPNPU | Supported Number of Partitions ( File: /src/core/flows/broadcast_flow.hppSymbol: numberOfPartitionsnumberOfPartitions |
|---|---|
| QB1 | 1 |
| QB4 | 1, 4 |
| QB16 | 1, 4, 16 |
Examples
Assume we have the following weight tensor of shape (1,1,1,32) and type int8:
If we use 4 partitions, i.e.
/src/core/flows/broadcast_flow.hppSymbol: numberOfPartitionsnumberOfPartitions= 4, and broadcast data to LRM, the tiles will look like following:Tile 1
Tile 2
Without VAP, the tiles will look like follows
Tile 1
Tile 2
Arrangement also depends on the number of partitions.
The diagram below illustrates how elements are arranged when the number of VAP partitions is set to 16. (This is only valid for QB16.) (Assume an int8 tensor)
Tile 1
The number of elements you can broadcast simultaneously depends on the data size, such as whether the element is an int8, int16, and so on. Consequently, the weight arrangement in the tile partitions also depends on the data type.
Please refer to the illustrations below to gain a better understanding of how the data is organized.
int8
Input tensor for broadcasting
If we rearrange this data in order we receive at LCM, it looks like follows.
The Quadric Broadcast Bus has a width of 64 bits, allowing it to transfer 8 units of int8 data in a single transfer. During the first broadcast iteration, it fetches the data up to the 31st value (32 values in total), which is equal to 8 (number of units of data) multiplied by 4 (number of partitions).
The remaining data will be fetched in the subsequent iterations. (if the input tensor size is greater than 32)
int16
Input tensor for broadcasting
If we rearrange this data in order we receive at LCM, it lookslike follows.
The Quadric Broadcast Bus has a 64-bit width, enabling it to transfer 4 units of int16 data in a single transfer. In the first broadcast iteration, the bus fetches data up to the 15th value (a total of 16 values), which is calculated as 4 (number of units of data) times 4 (number of partitions). The remaining data will be fetched in the subsequent iterations.
int32
Input tensor for broadcasting
If we rearrange this data in order we receive at LCM, it looks-like follows.
The Quadric Broadcast Bus features a 64-bit width, allowing it to transfer 2 units of int32 data in a single transfer.
During the first broadcast iteration, the bus fetches data up to the 7th value (a total of 8 values), which is calculated as 2 (number of units of data) times 4 (number of partitions).
The remaining data will be fetched in the subsequent iterations.
