Sigmoids
/src/neuralNetBlocks/activations.hppLines 55–112This calculates the following sigmoid function for all elements in a single tile.
f(x) = 1/(1 + exp(-x))
template <FracRepType numFracBits, typename T>
qVar_t<FixedPoint<T, 31>> sigmoid(qVar_t<FixedPoint<T, numFracBits>> x)| Name | Type | Default | Description |
|---|---|---|---|
numFracBits | FracRepType | Number of fraction bits in the fixedpoint numbers. | |
T | typename | The type of the data. |
| Name | Type | Description |
|---|---|---|
x | qVar_t<FixedPoint<T, numFracBits>> | A single tile of data. |
qVar_t<FixedPoint<T, 31>> — result The sigmoid function outputs for all elements in a tile. / // clang-format on
TanH (Hyperbolic Tangents)
/src/neuralNetBlocks/activations.hppLines 214–251This function performs the tanh function on the input, which is a single tile.
tanh(x) = (1- exp(-2*x))/(1 + exp(-2*x))
template <FracRepType numFracBits, typename T>
qVar_t<FixedPoint32<31>> tanh(qVar_t<FixedPoint<T, numFracBits>> x)| Name | Type | Default | Description |
|---|---|---|---|
numFracBits | FracRepType | Number of fraction bits in the fixedpoint numbers. | |
T | typename | The type of the data. |
| Name | Type | Description |
|---|---|---|
x | qVar_t<FixedPoint<T, numFracBits>> | A single tile of data. |
qVar_t<FixedPoint32<31>> — result The tanh function outputs for all elements in a tile. / // clang-format on
ReLU (Rectified Linear Units)
/src/neuralNetBlocks/activations.hppLines 579–594This function performs the ReLU6 function on the input, which is a single tile: f(x) = min(max(0, x), 6).
template <ReluMethod reluMethod,
typename T,
typename std::enable_if_t<reluMethod == ReluMethod::SIX, std::int32_t> = 0>
qVar_t<T> relu(qVar_t<T> qData)| Name | Type | Default | Description |
|---|---|---|---|
reluMethod | ReluMethod | The Relu method chosen. | |
T | typename | The type of the data |
| Name | Type | Description |
|---|---|---|
qData | qVar_t<T> | A single tile of data |
qVar_t<T> — qOutout The ReLU6 function outputs for all elements in a tile. / // clang-format on
/src/neuralNetBlocks/activations.hppLines 579–594This function performs the ReLU6 function on the input, which is a single tile: f(x) = min(max(0, x), 6).
template <ReluMethod reluMethod,
typename T,
typename std::enable_if_t<reluMethod == ReluMethod::SIX, std::int32_t> = 0>
qVar_t<T> relu(qVar_t<T> qData)| Name | Type | Default | Description |
|---|---|---|---|
reluMethod | ReluMethod | The Relu method chosen. | |
T | typename | The type of the data |
| Name | Type | Description |
|---|---|---|
qData | qVar_t<T> | A single tile of data |
qVar_t<T> — qOutout The ReLU6 function outputs for all elements in a tile. / // clang-format on
Softmax
/src/neuralNetBlocks/activations.hppLines 862–1023This function performs a softmax, or normalized exponential function, on the input data buffer and stores the result in the output data buffer.
A softmax normalizes the input data to a probability distribution proportional to the exponentials of the inputs. In other words, softmax takes the exponential function of each input elements and then normalizes by dividing each of the elements by the sum of all the exponentials. The output has the same shape as the input but is bound to the interval (0, 1).
Additionally, there is a iteration that goes through the the scores and takes the maximum value for each channel. This maximum is subtracted from each score with a new range of (-inf, 0] so the output of exp is bound to the interval (0, 1]. This is done to ensure that the softmax has a countable sum (0, Height*Width) which is numerically stable while still ensuring that the maximum/largest scores are well represented in fixed-point.
The equation for softmax is:
softmax(x_i) = exp(x_i) / summation(exp(x_j) for all j)
template <typename OcmTensorShape,
bool masked = false,
SoftmaxReduction reduction = (OcmTensorShape::NUM_CHN == 1) ? SoftmaxReduction::Spatial : SoftmaxReduction::Channel,
typename NDArrayIn,
typename NDArrayOut,
typename Lambda = EmptyType,
typename SinkParam = EmptyType,
std::enable_if_t<reduction == SoftmaxReduction::Spatial ||
reduction == SoftmaxReduction::Width ||
reduction == SoftmaxReduction::Height, std::int32_t> = 0>
INLINE void softmax(NDArrayIn& qInBuffer,
NDArrayOut& qOutBuffer,
const std::int32_t& maskThresh = OcmTensorShape::NUM_ELEMS_PER_BCH,
const Lambda& postProc = EmptyType(),
const SinkParam& sink = EmptyType())| Name | Type | Default | Description |
|---|---|---|---|
OcmTensorShape | typename | The shape of the OCM that we get values from. | |
masked | bool | false | Whether or not to apply a causal mask to this softmax. |
reduction | SoftmaxReduction | (OcmTensorShape::NUM_CHN == 1) ? SoftmaxReduction::Spatial : SoftmaxReduction::Channel | Which axis the softmax reduces over (Spatial for single-channel input). |
NDArrayIn | typename | The input NDArray type. | |
NDArrayOut | typename | The output NDArray type. | |
Lambda | typename | EmptyType | The type of the lambda postprocessing function. |
SinkParam | typename | EmptyType |
| Name | Type | Description |
|---|---|---|
qInBuffer | NDArrayIn& | The input buffer. |
qOutBuffer | NDArrayOut& | The output buffer. |
maskThresh | const std::int32_t& | The mask threshold to be apply if masked=true. |
postProc | const Lambda& | The postprocessing function. |
sink | const SinkParam& | Optional extra logit that joins the denominator and is then dropped; EmptyType compiles the sink path out. Must carry the same frac bits as the input logits. |
INLINE void
Mish
/src/neuralNetBlocks/activations.hppLines 606–746This function calculates mish activation: f(x) = x * tanh(log(exp(x)+1)).
template <typename T>
qVar_t<T> mish(qVar_t<T> x)| Name | Type | Default | Description |
|---|---|---|---|
T | typename | The type of the data |
| Name | Type | Description |
|---|---|---|
x | qVar_t<T> | A single tile of data |
qVar_t<T> — result Mish activation function outputs. / // clang-format on
