The Flow APIs enable efficient data movement between different memory hierarchies in the Chimera architecture. These APIs abstract the complexities of data transfers and provide optimized buffering strategies for improved performance.
Flow API Overview
The system provides several categories of Flow APIs for different memory transfer scenarios:
OCM ↔ LRM Flows (Internal Flows)
For data transfers between On-Chip Memory (OCM) and Local Register Memory (LRM):
- - Buffered read flow from OCM to LRMFile:
/src/core/flows/pls_flow.hppSymbol: ReadFlowReadFlow - - Direct read flow from OCM to LRM (no ping-pong buffering)File:
/src/core/flows/pls_flow.hppSymbol: BufferedReadFlowBufferedReadFlow - - Buffered write flow from LRM to OCMFile:
/src/core/flows/pls_flow.hppSymbol: WriteFlowWriteFlow - - Direct write flow from LRM to OCM (no ping-pong buffering)File:
/src/core/flows/pls_flow.hppSymbol: UnbufferedWriteFlowUnbufferedWriteFlow
DDR ↔ LRM Flows (Integrated Flows)
For data transfers that span DDR to LRM with integrated buffering:
- - Integrated flow from DDR to LRM (combines DDR→OCM + OCM→LRM)File:
/src/core/flows/ddr_array_flow.hppSymbol: DdrArrayFlowDdrArrayFlow - - Integrated flow from LRM to DDR (combines LRM→OCM + OCM→DDR)File:
/src/core/flows/ddr_array_flow.hppSymbol: ArrayDdrFlowArrayDdrFlow
DDR ↔ OCM Flows (External Flows)
For high-level data transfers between DDR and OCM with advanced double buffering:
- - External read flow from DDR to OCM with double bufferingFile:
/src/core/flows/ext_flow.hppSymbol: ExtReadFlowExtReadFlow - - External write flow from OCM to DDR with double bufferingFile:
/src/core/flows/ext_flow.hppSymbol: ExtWriteFlowExtWriteFlow
Memory Hierarchy
The Chimera architecture consists of three main memory levels:
- DDR: External memory for large data storage
- OCM: On-Chip Memory for intermediate buffering
- LRM: Local Register Memory for computation
Flow APIs enable efficient data movement between these levels with optimized buffering strategies.
General Flows
File:
src/core/flows/ddr_array_flow.hppLines 456–489 /**
* @brief Creates an Array -> DDR/OCM flow that depends on the input tensor type.
*
* When `bufferOnLrm` is true (default), creates an `ArrayDdrFlow` that double-buffers through LRM.
* When `bufferOnLrm` is false, creates an `UnbufferedArrayDdrFlow`.
*
* @tparam Access The TensorAccessor.
* @tparam DdrTensorShape The DDR tensor type.
* @tparam RoiShape The shape of the region of interest (RoI). Default is `DdrTensorShape`.
* @tparam numMinRoiPerOcmBuffer Max number of minROIs in each OCM buffer.
* @tparam numOcmBuffers Number of OCM buffers. 2 enables async buffering for better cycle utilization, 1 saves OCM
* usage.
* @tparam OcmAllocatorType The OCM allocator type. Default is `MemAllocator`.
* @tparam side The stream side for the PLS. Default is `StreamSide::NORTH`.
* @tparam bufferOnLrm When true, buffers data through LRM (double-buffered). When false, creates an unbuffered flow.
*
* @param ddrTensor Tensor located in external memory.
* @param allocator Memory allocator.
* @param start Starting address. Default is `{}`.
*
* @return An `ArrayDdrFlow` or `UnbufferedArrayDdrFlow` depending on `bufferOnLrm`.
*/
template <typename Access,
typename DdrTensorShape,
typename RoiShape = DdrTensorShape,
std::uint32_t numMinRoiPerOcmBuffer = 2,
std::uint32_t numOcmBuffers = 2,
typename OcmAllocatorType = MemAllocator,
StreamSide side = StreamSide::NORTH,
bool bufferOnLrm = true,
std::enable_if_t<DdrTensorShape::location == TensorLocation::DDR, int> = 0>
INLINE auto createGeneralWriteFlow(DdrTensorShape& ddrTensor,
OcmAllocatorType& allocator,
const TensorOffset& start = {}) {
File:
/src/core/flows/ddr_array_flow.hppLines 414–448 /**
* @brief Creates an OCM -> Array flow that depends on the input tensor type.
*
* When `bufferOnLrm` is true (default), creates a `ReadFlow`.
* When `bufferOnLrm` is false, creates an `UnbufferedReadFlow` with an internal buffer
* (enabling the no-arg `read()` which returns a reference to the internal buffer).
*
* @tparam Access The TensorAccessor.
* @tparam OcmTensorShape The shape of the OCM tensor.
* @tparam RoiShape The shape of the region of interest (RoI). Default is `OcmTensorShape`.
* @tparam numMinRoiPerOcmBuffer Max number of minROIs in each OCM buffer. Not applicable in this version of general
* flow.
* @tparam numOcmBuffers Number of OCM buffers. Not applicable in this version of general flow.
* @tparam OcmAllocatorType The OCM allocator type. Default is `MemAllocator`.
* @tparam side The stream side for the PLS. Default is `StreamSide::SOUTH`.
* @tparam bufferOnLrm When true, buffers data through LRM. When false, creates an unbuffered flow.
*
* @param ocmTensor Tensor located in OCM.
* @param allocator Memory allocator.
* @param start Starting offset. Default is all zero, i.e. `{}`.
*
* @return A `ReadFlow` or `UnbufferedReadFlow` depending on `bufferOnLrm`.
*/
template <typename Access,
typename OcmTensorShape,
typename RoiShape,
std::uint32_t numMinRoiPerOcmBuffer = 2,
std::uint32_t numOcmBuffers = 2,
typename OcmAllocatorType = MemAllocator,
StreamSide side = StreamSide::SOUTH,
bool bufferOnLrm = true,
std::enable_if_t<OcmTensorShape::location == TensorLocation::OCM, int> = 0>
INLINE auto createGeneralReadFlow(OcmTensorShape& ocmTensor,
OcmAllocatorType& allocator,
const TensorOffset& start = {}) {
File:
src/core/flows/ddr_array_flow.hppLines 456–489 /**
* @brief Creates an Array -> DDR/OCM flow that depends on the input tensor type.
*
* When `bufferOnLrm` is true (default), creates an `ArrayDdrFlow` that double-buffers through LRM.
* When `bufferOnLrm` is false, creates an `UnbufferedArrayDdrFlow`.
*
* @tparam Access The TensorAccessor.
* @tparam DdrTensorShape The DDR tensor type.
* @tparam RoiShape The shape of the region of interest (RoI). Default is `DdrTensorShape`.
* @tparam numMinRoiPerOcmBuffer Max number of minROIs in each OCM buffer.
* @tparam numOcmBuffers Number of OCM buffers. 2 enables async buffering for better cycle utilization, 1 saves OCM
* usage.
* @tparam OcmAllocatorType The OCM allocator type. Default is `MemAllocator`.
* @tparam side The stream side for the PLS. Default is `StreamSide::NORTH`.
* @tparam bufferOnLrm When true, buffers data through LRM (double-buffered). When false, creates an unbuffered flow.
*
* @param ddrTensor Tensor located in external memory.
* @param allocator Memory allocator.
* @param start Starting address. Default is `{}`.
*
* @return An `ArrayDdrFlow` or `UnbufferedArrayDdrFlow` depending on `bufferOnLrm`.
*/
template <typename Access,
typename DdrTensorShape,
typename RoiShape = DdrTensorShape,
std::uint32_t numMinRoiPerOcmBuffer = 2,
std::uint32_t numOcmBuffers = 2,
typename OcmAllocatorType = MemAllocator,
StreamSide side = StreamSide::NORTH,
bool bufferOnLrm = true,
std::enable_if_t<DdrTensorShape::location == TensorLocation::DDR, int> = 0>
INLINE auto createGeneralWriteFlow(DdrTensorShape& ddrTensor,
OcmAllocatorType& allocator,
const TensorOffset& start = {}) {
File:
/src/core/flows/ddr_array_flow.hppLines 509–542 /**
* @brief Creates an Array -> OCM flow that depends on the input tensor type.
*
* When `bufferOnLrm` is true (default), creates a `WriteFlow`.
* When `bufferOnLrm` is false, creates an `UnbufferedWriteFlow`.
*
* @tparam Access The TensorAccessor.
* @tparam DdrTensorShape The DDR tensor type.
* @tparam RoiShape The shape of the region of interest (RoI). Default is `DdrTensorShape`.
* @tparam numMinRoiPerOcmBuffer Max number of minROIs in each OCM buffer. Not applicable in this version of general
* flow.
* @tparam numOcmBuffers Number of OCM buffers. Not applicable in this version of general flow.
* @tparam OcmAllocatorType The OCM allocator type. Default is `MemAllocator`.
* @tparam side The stream side for the PLS. Default is `StreamSide::NORTH`.
* @tparam bufferOnLrm When true, buffers data through LRM. When false, creates an unbuffered flow.
*
* @param ddrTensor Tensor located in external memory.
* @param allocator Memory allocator.
* @param start Starting address. Default is `{}`.
*
* @return A `WriteFlow` or `UnbufferedWriteFlow` depending on `bufferOnLrm`.
*/
template <typename Access,
typename DdrTensorShape,
typename RoiShape = DdrTensorShape,
std::uint32_t numMinRoiPerOcmBuffer = 2,
std::uint32_t numOcmBuffers = 2,
typename OcmAllocatorType = MemAllocator,
StreamSide side = StreamSide::NORTH,
bool bufferOnLrm = true,
std::enable_if_t<DdrTensorShape::location == TensorLocation::OCM, int> = 0>
INLINE auto createGeneralWriteFlow(DdrTensorShape& ddrTensor,
OcmAllocatorType& allocator,
const TensorOffset& start = {}) {
Read Flows
Read flows copy data from L2 memory to LRM.
File:
/src/core/flows/pls_flow.hppLines 947–959 /**
* @brief Class to write data from the LRM to the OCM by iterating through all minROIs in a given tensor.
* @ingroup flow
* @tparam Access The TensorAccessor class.
* @tparam OCMShape The OCM tensor type.
* @tparam RoiShape Region of Interest of the OCM to which we want to write. Default is `OCMShape`.
* @tparam side The side of the flow. Default is `Epu::StreamSide::NORTH`.
*/
template <typename Access,
typename OCMShape,
typename RoiShape = OCMShape,
Epu::StreamSide side = Epu::StreamSide::NORTH>
using WriteFlow = BufferedWriteFlow<Access, OCMShape, RoiShape, side>;
File:
/src/core/flows/pls_flow.hppLines 834–846 /**
* @brief Buffered read flow.
*
* @tparam Access The TensorAccessor class.
* @tparam OCMShape The OCM tensor type.
* @tparam RoiShape Region of Interest (RoI) of the OCM from which we want to read. Default is `OCMShape`.
* @tparam side The side of the flow. Default is `Epu::StreamSide::SOUTH`.
*/
template <typename Access,
typename OCMShape,
typename RoiShape = OCMShape,
Epu::StreamSide side = Epu::StreamSide::SOUTH>
using BufferedReadFlow = Flow<FlowType::Read, Access, RoiShape, side, OCMShape, true>;
File:
/src/core/flows/pls_flow.hppSymbol: ReadFlowReadFlowFile:
/src/core/flows/pls_flow.hppSymbol: BufferedReadFlowBufferedReadFlowFile:
/src/core/types/ct_containers.hppSymbol: NDArrayNDArrayFile:
/src/core/types/ct_containers.hppSymbol: NDArrayNDArrayFile:
/src/core/types/ct_containers.hppSymbol: NDArrayNDArrayFile:
/src/core/types/ct_containers.hppSymbol: NDArrayNDArrayFile:
/src/core/flows/pls_flow.hppSymbol: BufferedReadFlowBufferedReadFlowFile:
/src/core/flows/pls_flow.hppLines 911–927 /**
* @brief Similar to a WriteFlow. This does not utilize ping-pong in LRM which prevents concurrent flows with compute
* and hence degraded performance. Use this API in places where the min ROI needs to take up a large space in LRM and
* ping-pong (2x the buffer size) is not an option
*
* @tparam Access The TensorAccessor class.
* @tparam OCMShape The OCM tensor type.
* @tparam RoiShape Region of Interest (RoI) of the OCM from which we want to read. Default is `OCMShape`.
* @tparam side The side of the flow. Default is `Epu::StreamSide::NORTH`.
*
* @ingroup flow
*/
template <typename Access,
typename OCMShape,
typename RoiShape = OCMShape,
Epu::StreamSide side = Epu::StreamSide::NORTH>
using UnbufferedWriteFlow = Flow<FlowType::Write, Access, RoiShape, side, OCMShape, false>;
UnbufferedReadFlow API can also be used to read the data stored in an OCM tensor and write it into LRM. However, this does not utilize a ping-pong buffer in LRM which prevents concurrent flows with compute and hence degraded performance. Use this API in places where the min ROI needs to take up a large space in LRM and ping-pong (2x the buffer size) is not an option.
Write Flows
WriteFlow API helps to write the data stored in LRM into an OCM tensor. The reason this is also called as BufferedWriteFlow is, it does not write the LRM data directly into the output OCM tensor. First, it copies the data in the input NDArray to a ping-pong buffer NDArray (which is also allocated in the LRM) and then writes the data to the output OCM tensor. This very useful when it comes to performance improvements because with this, it is possible to write the previous minROI of processed data to the output OCM tensor while we process the current minROI of data that we already have in the input NDArray. However, this version takes twice as much space than the UnbufferedWriteFlow version.
File:
/src/core/flows/pls_flow.hppLines 947–959 /**
* @brief Class to write data from the LRM to the OCM by iterating through all minROIs in a given tensor.
* @ingroup flow
* @tparam Access The TensorAccessor class.
* @tparam OCMShape The OCM tensor type.
* @tparam RoiShape Region of Interest of the OCM to which we want to write. Default is `OCMShape`.
* @tparam side The side of the flow. Default is `Epu::StreamSide::NORTH`.
*/
template <typename Access,
typename OCMShape,
typename RoiShape = OCMShape,
Epu::StreamSide side = Epu::StreamSide::NORTH>
using WriteFlow = BufferedWriteFlow<Access, OCMShape, RoiShape, side>;
File:
/src/core/flows/pls_flow.hppLines 848–860 /**
* @brief Buffered write flow.
*
* @tparam Access The TensorAccessor class.
* @tparam OCMShape The OCM tensor type.
* @tparam RoiShape Region of Interest (RoI) of the OCM from which we want to read. Default is `OCMShape`.
* @tparam side The side of the flow. Default is `Epu::StreamSide::NORTH`.
*/
template <typename Access,
typename OCMShape,
typename RoiShape = OCMShape,
Epu::StreamSide side = Epu::StreamSide::NORTH>
using BufferedWriteFlow = Flow<FlowType::Write, Access, RoiShape, side, OCMShape, true>;
UnbufferedWriteFlow API can also be used to write the data stored in LRM into an OCM tensor. However, this does not utilize a ping-pong buffer in LRM which prevents concurrent flows with compute and hence degraded performance. Use this API in places where the min ROI needs to take up a large space in LRM and ping-pong (2x the buffer size) is not an option.
File:
/src/core/flows/pls_flow.hppLines 911–927 /**
* @brief Similar to a WriteFlow. This does not utilize ping-pong in LRM which prevents concurrent flows with compute
* and hence degraded performance. Use this API in places where the min ROI needs to take up a large space in LRM and
* ping-pong (2x the buffer size) is not an option
*
* @tparam Access The TensorAccessor class.
* @tparam OCMShape The OCM tensor type.
* @tparam RoiShape Region of Interest (RoI) of the OCM from which we want to read. Default is `OCMShape`.
* @tparam side The side of the flow. Default is `Epu::StreamSide::NORTH`.
*
* @ingroup flow
*/
template <typename Access,
typename OCMShape,
typename RoiShape = OCMShape,
Epu::StreamSide side = Epu::StreamSide::NORTH>
using UnbufferedWriteFlow = Flow<FlowType::Write, Access, RoiShape, side, OCMShape, false>;
Runtime Tensor Support in ReadFlow/WriteFlow (New)
The ReadFlow and WriteFlow APIs now natively support both static tensors (compile-time dimensions) and runtime tensors (runtime dimensions) with a single consistent interface. Runtime tensor support is directly integrated into the flow classes—no separate "unified" flow types are needed.
What ReadFlow/WriteFlow Support
Core Capabilities:
- Static Tensors: Traditional tensors with compile-time known dimensions
- Runtime Tensors: Tensors with dimensions determined at runtime via
RuntimeTensorwrapper - Double Buffering: Ping-pong buffering in LRM for overlapping compute and data movement (default, best performance)
- Unbuffered Mode: Direct transfers without ping-pong buffering to save LRM space
- Tensor Offsets: Partial read/write of tensor sub-regions
- ROI (Region of Interest): Process specific regions of tensors
- MinROI Iteration: Automatic iteration through minimum regions of interest
- Runtime Tensor Slicing: Create smaller views of larger OCM buffers at runtime
- Automatic Address Computation: Runtime dimension-aware address calculations
Current Limitations for Runtime Tensors
- Bordered Flows: Flows with border/halo regions are not yet supported for runtime tensors
- Multi-Layer Synchronization (MLS): Inter-core communication for convolution-like operations with border padding
- Sleep State Optimization: Power optimization by selectively sleeping cores based on data boundaries
- EAST Stream Side: Only NORTH and SOUTH sides are supported
- Constraint:
squareInternalHeightStridemust equal 1 for runtime tensors
Using ReadFlow/WriteFlow with Static Tensors
Static tensors have dimensions known at compile time:
// Define static tensor types
using OcmTensorType = Tensor<int16_t, TensorLocation::OCM, 1, 64, 32, 128>;
// Define access pattern
using MinRoiDesc = MinRoiDescriptor<OcmTensorType,
AxisGroup<Direction::Width, Direction::Height>,
Granularity::Square>;
using Access = TensorAccessor<MinRoiDesc>;
// Create flows with static tensors
OcmTensorType inputTensor;
OcmTensorType outputTensor;
ocmAllocator.allocate(inputTensor);
ocmAllocator.allocate(outputTensor);
ReadFlow<Access, OcmTensorType> readFlow(inputTensor);
WriteFlow<Access, OcmTensorType> writeFlow(outputTensor);
// Use the flows
typename ReadFlow<Access, OcmTensorType>::MinRoiArray lrmBuffer;
for(uint32_t i = 0; i < readFlow.numMROIs(); i++) {
readFlow.read(lrmBuffer);
// Process data in lrmBuffer...
writeFlow.write(lrmBuffer);
}
Using ReadFlow/WriteFlow with Runtime Tensors
Runtime tensors allow dimensions to be specified at runtime, enabling dynamic tensor slicing and flexible memory layouts:
// Allocate a large OCM buffer (base allocation)
using OcmTensorType = Tensor<int16_t, TensorLocation::OCM, 1, 64, 64, 256>;
OcmTensorType ocmBuffer;
ocmAllocator.allocate(ocmBuffer);
// Create RuntimeTensor view with smaller dimensions (runtime slicing)
using RuntimeTensorOcm = RuntimeTensor<OcmTensorType>;
RuntimeTensorOcm runtimeView(
ocmBuffer, // Base OCM buffer
1, 64, 32, 128, // Runtime dimensions (BCH, CHN, ROWS, COLS) - smaller than base
{0, 0, 0, 0} // Start offset
);
// Define access pattern (same MinRoiDesc as static case)
using MinRoiDesc = MinRoiDescriptor<OcmTensorType,
AxisGroup<Direction::Width, Direction::Height>,
Granularity::Square>;
using Access = TensorAccessor<MinRoiDesc>;
// Create flows with runtime tensors
// IMPORTANT: For runtime tensors, RoiShape must equal OcmShape type
ReadFlow<Access, RuntimeTensorOcm, RuntimeTensorOcm> readFlow(runtimeView);
WriteFlow<Access, RuntimeTensorOcm, RuntimeTensorOcm> writeFlow(runtimeView);
// Use flows exactly the same way as static tensors
typename ReadFlow<Access, RuntimeTensorOcm, RuntimeTensorOcm>::MinRoiArray lrmBuffer;
for(uint32_t i = 0; i < readFlow.numMROIs(); i++) {
readFlow.read(lrmBuffer);
// Process data...
writeFlow.write(lrmBuffer);
}
Key Points for Runtime Tensors:
- The
RoiShapetemplate parameter must be the same type asOcmShape(both must beRuntimeTensortypes) - Actual dimensions are specified when constructing the
RuntimeTensorinstance
Buffered vs Unbuffered Flow Usage
Choosing Buffered vs Unbuffered:
| Mode | LRM Usage | Performance | Use Case |
|---|---|---|---|
| Buffered (default) | 2x buffer size | High - overlaps compute with memory ops | Most workloads |
| Unbuffered | 1x buffer size | Lower - sequential operations | LRM-constrained workloads |
Buffered Flow (Default - Recommended):
// Uses ping-pong buffers (2x LRM buffers)
ReadFlow<Access, RuntimeTensorOcm, RuntimeTensorOcm> readFlow(ocmTensor);
WriteFlow<Access, RuntimeTensorOcm, RuntimeTensorOcm> writeFlow(ocmTensor);
typename ReadFlow<Access, RuntimeTensorOcm, RuntimeTensorOcm>::MinRoiArray lrmBuffer;
readFlow.read(lrmBuffer);
// While processing current data, next data is being fetched!
writeFlow.write(lrmBuffer);
Unbuffered Flow (LRM-Constrained):
// Uses single buffer (1x LRM buffer)
UnbufferedReadFlow<Access, RuntimeTensorOcm, RuntimeTensorOcm> readFlow(ocmTensor);
UnbufferedWriteFlow<Access, RuntimeTensorOcm, RuntimeTensorOcm> writeFlow(ocmTensor);
typename UnbufferedReadFlow<Access, RuntimeTensorOcm, RuntimeTensorOcm>::MinRoiArray buffer;
readFlow.read(buffer);
// Sequential operation - no overlap with next fetch
writeFlow.write(buffer);
Complete Example: Static vs Runtime Tensors
This example demonstrates that the same ReadFlow/WriteFlow API works with both static and runtime tensors:
// ===== Static Tensors using ReadFlow/WriteFlow =====
{
// Use Flow directly or the ReadFlow/WriteFlow aliases
using StaticReadFlow = Flow<FlowType::Read, Access, TestTensorOcm,
Epu::StreamSide::SOUTH, TestTensorOcm>;
using StaticWriteFlow = Flow<FlowType::Write, Access, TestTensorOcm,
Epu::StreamSide::NORTH, TestTensorOcm>;
StaticReadFlow readFlow(ocmStaticInput);
StaticWriteFlow writeFlow(ocmStaticOutput);
typename StaticReadFlow::MinRoiArray lrmBuffer;
for(std::uint32_t i = 0; i < readFlow.numMROIs(); i++) {
readFlow.read(lrmBuffer);
writeFlow.write(lrmBuffer);
}
memCpy(ocmStaticOutput, ddrOutStatic);
}
// ===== Runtime Tensors using ReadFlow/WriteFlow =====
{
using RuntimeTensorOcm = RuntimeTensor<TestTensorOcm>;
RuntimeTensorOcm runtimeOcmInp(ocmRuntimeInput,
TestTensorOcm::NUM_BCH,
TestTensorOcm::NUM_CHN,
TestTensorOcm::NUM_ROWS,
TestTensorOcm::NUM_COLS);
RuntimeTensorOcm runtimeOcmOut(ocmRuntimeOutput,
TestTensorOcm::NUM_BCH,
TestTensorOcm::NUM_CHN,
TestTensorOcm::NUM_ROWS,
TestTensorOcm::NUM_COLS);
// Use ReadFlow/WriteFlow directly with runtime tensors
using RuntimeReadFlow = ReadFlow<Access, RuntimeTensorOcm, RuntimeTensorOcm>;
using RuntimeWriteFlow = WriteFlow<Access, RuntimeTensorOcm, RuntimeTensorOcm>;
RuntimeReadFlow readFlow(runtimeOcmInp);
RuntimeWriteFlow writeFlow(runtimeOcmOut);
typename RuntimeReadFlow::MinRoiArray lrmBuffer;
for(std::uint32_t i = 0; i < readFlow.numMROIs(); i++) {
readFlow.read(lrmBuffer);
writeFlow.write(lrmBuffer);
}
memCpy(runtimeOcmOut, runtimeDdrOut);
}
Both code paths produce identical results, validating the unified implementation!
Future Development Plans
Runtime tensor support in ReadFlow/WriteFlow is under active development with planned enhancements:
Upcoming Features:
- Sleep State Support: Power optimization by selectively sleeping cores based on data boundaries
- DdrArrayFlow Integration: Extend runtime tensor capabilities to DDR↔LRM flows
- MLS Support: Multi-layer synchronization for inter-core communication in convolution operations
- Bordered Flow Support: Add support for halo regions and border padding for runtime tensors
Partial Read/Write Flows with Tensor Offsets
In some cases, it is required to read/write only a certain part (sub-section) of an OCM tensor. This is where the Tensor offsets come in handy. All types of Flow APIs support tensor offsets. Following image shows what is a ROI and what us a tensor offset. There are two ways of defining the tensor offset.
- By passing required starting indexes of batch, channel, row and column dimensions to the Flow API.
- By creating and passing a TensorOffset object to the Flow API.
File:
/src/core/flows/ddr_array_flow.hppLines 23–57 /**
* @brief High-level data access classes DdrArrayFlow and ArrayDdrFlow, manages flow between data array on cores and
* DDR, has underlying optimization via ping-pong buffers.
*
* @tparam flowType read/write
* @tparam Access the TensorAccessor
* @tparam DdrTensorShape The DDR tensor type.
* @tparam OcmAllocatorType The type of OCM allocator.
* @tparam RoiShape ROI, defaults to DDR
* @tparam side the stream side for the PLS
* @tparam numMinRoiPerBufferLimit max number of minROIs in each buffer, the ExtFlow class will determine the exact
* number to be used and the value is accessible via ElsFlowType::numMinRoiPerBuffer
* @tparam numBuffers number of buffers to keep in OCM, 2 will enable async buffering for better cycle utilization, 1
* will save OCM usage.
* @tparam doubleBufferLrm Whether or not to use double buffering in LRM. Default is `true`.
* @tparam internalLRMBuffer Whether or not to create internal LRM buffer in pls flow object.
*/
template <FlowType flowType,
typename Access,
typename DdrTensorShape,
typename OcmAllocatorType,
typename RoiShape = DdrTensorShape,
StreamSide side = flowType == FlowType::Read ? StreamSide::SOUTH : StreamSide::NORTH,
std::uint32_t numMinRoiPerBufferLimit = 2,
std::uint32_t numBuffers = 2,
bool doubleBufferLrm = true,
bool internalLRMBuffer = false>
struct DdrArrayFlowImpl : FlowSleepHandler<DdrArrayFlowImpl<flowType,
Access,
DdrTensorShape,
OcmAllocatorType,
RoiShape,
side,
numMinRoiPerBufferLimit,
numBuffers,
DDRArrayFlow
DdrArrayFlow<
typename Access,
typename DdrTensorShape,
typename RoiShape = DdrTensorShape,
StreamSide side = StreamSide::SOUTH,
std::uint32_t numMinRoiPerBufferLimit = 2,
std::uint32_t numBuffers = 2,
typename Buffer = OCMBufferShape<typename Access::MinRoi, RoiShape, typename Access::RoiAxisGroup, numMinRoiPerBufferLimit>,
>
(
DdrTensorShape& ddrTensor,
MemAllocator& allocator,
TensorOffset start = {}
)
Template parameters
Access: the TensorAccessor used to access the data array on cores.
DdrTensorShape: the shape of the DDR tensor.
RoiShape: the shape of the Region of Interest (ROI), which defaults to be the same as DdrTensorShape.
side: side the stream side for the PLS. This defaults to SOUTH for read operations
numMinRoiPerBufferLimit: the maximum number of minimum ROIs (minROIs) in each buffer. The ExtFlow class will determine the exact number to be used, and the value is accessible via ElsFlowType::numMinRoiPerBuffer.
numBuffers: the number of buffers to keep in on-chip memory (OCM). Setting this to 2 will enable asynchronous buffering for better cycle utilization, while setting it to 1 will save OCM usage.
Buffer: the type of the buffer to use.
Parameters
ddrTensor: Source DDR tensor
allocator: ocm allocator to be used locally to allocate ocm memory space
start: Offset for DDR tensor
Example: {bchOffset, chnOffset, rowOffset, colOffset}
ArrayDdrFlow
File:
/src/core/flows/ddr_array_flow.hppLines 23–57 /**
* @brief High-level data access classes DdrArrayFlow and ArrayDdrFlow, manages flow between data array on cores and
* DDR, has underlying optimization via ping-pong buffers.
*
* @tparam flowType read/write
* @tparam Access the TensorAccessor
* @tparam DdrTensorShape The DDR tensor type.
* @tparam OcmAllocatorType The type of OCM allocator.
* @tparam RoiShape ROI, defaults to DDR
* @tparam side the stream side for the PLS
* @tparam numMinRoiPerBufferLimit max number of minROIs in each buffer, the ExtFlow class will determine the exact
* number to be used and the value is accessible via ElsFlowType::numMinRoiPerBuffer
* @tparam numBuffers number of buffers to keep in OCM, 2 will enable async buffering for better cycle utilization, 1
* will save OCM usage.
* @tparam doubleBufferLrm Whether or not to use double buffering in LRM. Default is `true`.
* @tparam internalLRMBuffer Whether or not to create internal LRM buffer in pls flow object.
*/
template <FlowType flowType,
typename Access,
typename DdrTensorShape,
typename OcmAllocatorType,
typename RoiShape = DdrTensorShape,
StreamSide side = flowType == FlowType::Read ? StreamSide::SOUTH : StreamSide::NORTH,
std::uint32_t numMinRoiPerBufferLimit = 2,
std::uint32_t numBuffers = 2,
bool doubleBufferLrm = true,
bool internalLRMBuffer = false>
struct DdrArrayFlowImpl : FlowSleepHandler<DdrArrayFlowImpl<flowType,
Access,
DdrTensorShape,
OcmAllocatorType,
RoiShape,
side,
numMinRoiPerBufferLimit,
numBuffers,
Functions of DdrArray/ArrayDdr Flows
read() : Return an NDArray with fetched data
read(MinRoiArray& data) : Fetch data from DDR to “data“ NDArray
write(MinRoiArray& data) : Write “data” Array to DDR
numMROIs() : Return the number of min RoIs
Examples
File:
/tests/sweep_tests/tad_flow/ddr_array.cppLines 26–66using ReadAccess =
TensorAccessor<MinRoiDescriptor<sweeps::MinRoiOcm,
AxisGroup<sweeps::axis0, sweeps::axis1, sweeps::axis2>,
sweeps::granularity,
// For V1 ZY, only bordered version is available for flow-in
sweeps::granularity == Granularity::Square && sweeps::axis0 == Direction::Channel>,
AxisGroup<sweeps::axis3, sweeps::axis4, sweeps::axis5>>;
using WriteAccess = TensorAccessor<
MinRoiDescriptor<sweeps::MinRoiOcm, AxisGroup<sweeps::axis0, sweeps::axis1, sweeps::axis2>, sweeps::granularity>,
AxisGroup<sweeps::axis3, sweeps::axis4, sweeps::axis5>>;
EPU_ENTRY void testDdrArray(sweeps::InOutDdr::ptrType inputDdrPtr, sweeps::InOutDdr::ptrType outputDdrPtr) {
initCycleCount();
MemAllocator ocmMem;
sweeps::InOutDdr ddrInp(inputDdrPtr);
sweeps::InOutDdr ddrOut(outputDdrPtr);
DdrArrayFlow<ReadAccess,
sweeps::InOutDdr,
MemAllocator,
sweeps::InOutDdr,
StreamSide::SOUTH,
sweeps::minRoiInBuffer,
sweeps::numBuffers>
elsRead(ddrInp, ocmMem);
ArrayDdrFlow<WriteAccess,
sweeps::InOutDdr,
MemAllocator,
sweeps::InOutDdr,
StreamSide::NORTH,
sweeps::minRoiInBuffer,
sweeps::numBuffers>
elsWrite(ddrOut, ocmMem);
for(std::uint32_t i = 0; i < elsRead.numMROIs(); ++i) {
auto a = elsRead.read();
elsWrite.write(a);
}
}
ExtReadFlow / ExtWriteFlow
ExtReadFlow and ExtWriteFlow APIs provide double-buffered data movement between DDR and OCM memory. These flows handle the DDR-to-OCM transfer with ping-pong buffering for optimal performance, enabling concurrent data transfer and compute operations.
ExtReadFlow
File:
/src/core/flows/ext_flow.hppSymbol: ExtReadFlowExtReadFlowExtReadFlow<
typename Access, // TensorAccessor for data access patterns
typename DdrTensorShape, // DDR tensor type
typename OCMBufferShape = void, // OCM buffer shape (auto-determined if void)
typename OcmAllocatorType = void, // OCM allocator type
typename RoiShape = DdrTensorShape, // Region of Interest shape
std::uint32_t numMinRoiPerBufferLimit = 1, // Max minROIs per buffer
std::uint32_t numBuffers = 2 // Number of buffers (2 for double buffering)
>
(
DdrTensorShape& ddrTensor,
OcmAllocatorType& allocator,
TensorOffset start = {}
)
Key Features:
- Double buffering: Enables concurrent DDR-to-OCM transfers and compute operations
- Automatic buffer management: Handles OCM buffer allocation and deallocation
- Configurable buffer sizes: Supports customizable buffer shapes and minROI limits
ExtWriteFlow
File:
/src/core/flows/ext_flow.hppSymbol: ExtWriteFlowExtWriteFlownext() function to return while the outgoing flow for the previous buffer is still in process.ExtWriteFlow<
typename Access, // TensorAccessor for data access patterns
typename DdrTensorShape, // DDR tensor type
typename OCMBufferShape = void, // OCM buffer shape (auto-determined if void)
typename OcmAllocatorType = void, // OCM allocator type
typename RoiShape = DdrTensorShape, // Region of Interest shape
std::uint32_t numMinRoiPerBufferLimit = 1, // Max minROIs per buffer
std::uint32_t numBuffers = 2 // Number of buffers (2 for double buffering)
>
(
DdrTensorShape& ddrTensor,
OcmAllocatorType& allocator,
TensorOffset start = {}
)
Key Features:
- Asynchronous writes: Returns from write operations while data transfer continues
- Double buffering: Enables concurrent OCM-to-DDR transfers and compute operations
- Pipeline optimization: Reduces overall kernel execution time
External Flow Example
A complete working example demonstrating the definitive double buffering flow pattern using ExtReadFlow can be found in the test suite:
File:
/tests/static_data_tests/ext_read_flow_test.cppLines 49–106/**
* @brief The main entry point for the EPU kernel demonstrating ExtReadFlow usage with double buffering.
* @param ddrInPtr Pointer to the input tensor in DDR memory.
* @param ddrOutPtr Pointer to the output tensor in DDR memory.
*/
EPU_ENTRY void testExtReadFlow(DdrInShape::ptrType ddrInPtr, DdrInShape::ptrType ddrOutPtr) {
initCycleCount();
blocklist_allocator::MemAllocator mem_allocator;
DdrInShape ddrIn(ddrInPtr);
DdrInShape ddrOut(ddrOutPtr);
// 1. Define the access pattern for the EXTERNAL flow (DDR -> OCM).
using ExtMinRoiDesc = MinRoiDescriptor<OcmInShape>;
using ExtAccess = TensorAccessor<ExtMinRoiDesc, AxisGroup<Direction::Width, Direction::Height, Direction::Channel>>;
// 2. Instantiate the external flow manager. This handles the DDR->OCM double buffering.
ExtReadFlow<ExtAccess, DdrInShape, OcmInShape, MemAllocator> extReadFlow(ddrIn, mem_allocator);
// Allocate output OCM buffer for storing processed results
OcmInShape ocmOut;
mem_allocator.allocate(ocmOut);
// 3. Loop through all the chunks using ExtReadFlow.
for(std::uint32_t i = 0; i < extReadFlow.numMROIs; ++i) {
// This call gets the next ready OCM buffer from the double-buffered DDR stream.
auto currentOcmBuffer = extReadFlow.read</*returnRef=*/false>();
// 4. Define the access pattern for the INTERNAL flows (OCM <-> LRM).
using OcmReadMinRoiDesc = MinRoiDescriptor<OcmInShape, DefaultAxisGroup, Granularity::Row>;
using OcmReadAccess = TensorAccessor<OcmReadMinRoiDesc>;
using OcmWriteMinRoiDesc = MinRoiDescriptor<OcmInShape, DefaultAxisGroup, Granularity::Row>;
using OcmWriteAccess = TensorAccessor<OcmWriteMinRoiDesc>;
// 5. Create the internal flows to stream data from the ready OCM buffer into core registers.
UnbufferedReadFlow<OcmReadAccess, OcmInShape> ocm_to_lrm_flow(currentOcmBuffer);
UnbufferedWriteFlow<OcmWriteAccess, OcmInShape> lrm_to_ocm_flow(ocmOut);
// 6. Process data using MinROI iteration pattern
typename decltype(ocm_to_lrm_flow)::MinRoiArray minROI;
ocm_to_lrm_flow.read(minROI);
// 7. Process the data (simple increment operation) - iterate over MinROI elements
typename decltype(lrm_to_ocm_flow)::MinRoiArray qOut;
for(std::size_t j = 0; j < minROI.size(); ++j) {
qOut[j] = minROI[j] + 1; // Process each element: increment by 1
}
// 8. Write processed data back
lrm_to_ocm_flow.write(qOut);
// 9. Copy this processed chunk back to the correct location in DDR
TensorOffset srcOffset = {0, 0, 0, 0};
TensorOffset dstOffset = {0, 0, 0, static_cast<std::int32_t>(i * OcmInShape::NUM_COLS)};
memCpy(ocmOut, srcOffset, ddrOut, dstOffset, mem_allocator);
}
}
This test demonstrates:
- Complete ExtReadFlow setup with proper tensor definitions and memory allocation
- Double buffering in action with DDR-to-OCM transfers managed automatically
- Data validation ensuring processed results match expected output
- Realistic processing pattern showing how to combine external and internal flows
Flow Architecture:
- DDR → OCM: Managed by
ExtReadFlowwith double buffering - OCM → LRM: Managed by internal
UnbufferedReadFlow - Concurrent operation: While processing current buffer, next buffer is being prefetched
Performance Benefits
External Flow APIs provide several performance advantages:
- Overlapped Execution: Memory transfers and compute operations run concurrently
- Reduced Stalls: Double buffering minimizes wait times for data availability
- Automatic Management: Handles complex buffer management and synchronization
- Scalable Design: Works efficiently with different buffer sizes and access patterns
DDR ↔︎ Array flow API
The Chimera architecture allows the cores to work simultaneously with the memory transfers between the array and the L2 meory, and between the L2 memory and the DDR. The flow APIs
File:
/src/core/flows/ddr_array_flow.hppSymbol: DdrArrayFlowDdrArrayFlowFile:
/src/core/flows/ddr_array_flow.hppSymbol: ArrayDdrFlowArrayDdrFlowIt simplifies the code as the user can use the same code for memory transfers across the system.
It boosts performance as the API also handles the dual buffer needed for parallelization.
However, it is important to note that users still have access to L2 flow and DDR memcpy APIs.
ExtFlow
File:
/src/core/flows/ext_flow.hppLines 249–291 /**
* @brief An external flow class with optional buffering. Buffering is enabled for `num_buffers > 1`. In case it is
* enabled, for flowing in, it prefetches the next OCM buffer so that the stall for the next read access could
* be reduced; and for writing out, it returns from the `next` function while the outgoing flow for the
* previous buffer is in process, allowing other work to be processed without waiting for the external flow to
* complete.
*
* @detail `OcmAllocatorType` should be specified if the allocation is needed to be done by an OCM memory allocator.
* If `OcmAllocatorType` is void, it is assumed that the user has manually allocated all the requisite buffers
* and the constructor will need just the pointers of the buffers.
*
* > Note: If `OCMBufferShape` is void, the class will determine the number of min-ROIs per buffer based on the
* `numMinRoiPerBufferLimit` and the moving axes of the min-ROI. It will greedily put as many min-ROIs as
* possible on each axis, and the final number is stored as static member `numMinRoiPerBuffer`.
*
* @tparam flowType Whether to read (`FlowType::Read`) or write (`FlowType::Write`).
* @tparam Access The `TensorAccessor` type.
* @tparam DdrTensorShape The DDR tensor type.
* @tparam OcmBufferShape The shape of each OCM buffer. Default is `void`.
* @tparam OcmAllocatorType The OCM allocator type. Default is `void`, i.e. no allocation is needed.
* @tparam RoiShape The type and shape of the region of interest. Default is `DdrTensorShape`.
* @tparam numMinRoiPerBufferLimit Maximum number of minROIs in each buffer. Default is `2`.
* @tparam _numBuffers The number of buffers to keep in OCM: `2` will enable async buffering for better
* cycle utilization, `1` will save OCM usage.
* @tparam _hasAllocator Whether or not the flow already has a memory allocator.
*/
template <FlowType flowType,
typename Access,
typename DdrTensorShape,
typename OcmBufferShape = void,
typename OcmAllocatorType = void,
typename RoiShape = DdrTensorShape,
std::uint32_t numMinRoiPerBufferLimit = 2,
std::uint8_t _numBuffers = 2,
bool _hasAllocator = std::is_void<OcmAllocatorType>::value>
struct ExtFlow : detail::ext_flow::ExtFlowImpl<flowType,
Access,
DdrTensorShape,
OcmBufferShape,
OcmAllocatorType,
RoiShape,
numMinRoiPerBufferLimit,
_numBuffers> {};
