Unlock this feature

This feature isn’t part of your plan yet

Contact sales to get upgraded to the full DevStudio experience.

Unlock this feature

This feature isn't part of your plan yet.

Activation Functions

Sigmoids

sigmoidfunction
File: /src/neuralNetBlocks/activations.hppLines 55–112

This calculates the following sigmoid function for all elements in a single tile.

f(x) = 1/(1 + exp(-x))

template <FracRepType numFracBits, typename T>
qVar_t<FixedPoint<T, 31>> sigmoid(qVar_t<FixedPoint<T, numFracBits>> x)
NameTypeDefaultDescription
numFracBitsFracRepTypeNumber of fraction bits in the fixedpoint numbers.
TtypenameThe type of the data.
NameTypeDescription
xqVar_t<FixedPoint<T, numFracBits>>A single tile of data.

qVar_t<FixedPoint<T, 31>> — result The sigmoid function outputs for all elements in a tile. / // clang-format on

TanH (Hyperbolic Tangents)

tanhfunction
File: /src/neuralNetBlocks/activations.hppLines 214–251

This function performs the tanh function on the input, which is a single tile.

tanh(x) = (1- exp(-2*x))/(1 + exp(-2*x))

template <FracRepType numFracBits, typename T>
qVar_t<FixedPoint32<31>> tanh(qVar_t<FixedPoint<T, numFracBits>> x)
NameTypeDefaultDescription
numFracBitsFracRepTypeNumber of fraction bits in the fixedpoint numbers.
TtypenameThe type of the data.
NameTypeDescription
xqVar_t<FixedPoint<T, numFracBits>>A single tile of data.

qVar_t<FixedPoint32<31>> — result The tanh function outputs for all elements in a tile. / // clang-format on

ReLU (Rectified Linear Units)

relufunction
File: /src/neuralNetBlocks/activations.hppLines 579–594

This function performs the ReLU6 function on the input, which is a single tile: f(x) = min(max(0, x), 6).

template <ReluMethod reluMethod,
          typename T,
          typename std::enable_if_t<reluMethod == ReluMethod::SIX, std::int32_t> = 0>
qVar_t<T> relu(qVar_t<T> qData)
NameTypeDefaultDescription
reluMethodReluMethodThe Relu method chosen.
TtypenameThe type of the data
NameTypeDescription
qDataqVar_t<T>A single tile of data

qVar_t<T> — qOutout The ReLU6 function outputs for all elements in a tile. / // clang-format on

relufunction
File: /src/neuralNetBlocks/activations.hppLines 579–594

This function performs the ReLU6 function on the input, which is a single tile: f(x) = min(max(0, x), 6).

template <ReluMethod reluMethod,
          typename T,
          typename std::enable_if_t<reluMethod == ReluMethod::SIX, std::int32_t> = 0>
qVar_t<T> relu(qVar_t<T> qData)
NameTypeDefaultDescription
reluMethodReluMethodThe Relu method chosen.
TtypenameThe type of the data
NameTypeDescription
qDataqVar_t<T>A single tile of data

qVar_t<T> — qOutout The ReLU6 function outputs for all elements in a tile. / // clang-format on

Softmax

softmaxfunction
File: /src/neuralNetBlocks/activations.hppLines 862–1023

This function performs a softmax, or normalized exponential function, on the input data buffer and stores the result in the output data buffer.

A softmax normalizes the input data to a probability distribution proportional to the exponentials of the inputs. In other words, softmax takes the exponential function of each input elements and then normalizes by dividing each of the elements by the sum of all the exponentials. The output has the same shape as the input but is bound to the interval (0, 1).

Additionally, there is a iteration that goes through the the scores and takes the maximum value for each channel. This maximum is subtracted from each score with a new range of (-inf, 0] so the output of exp is bound to the interval (0, 1]. This is done to ensure that the softmax has a countable sum (0, Height*Width) which is numerically stable while still ensuring that the maximum/largest scores are well represented in fixed-point.

The equation for softmax is:

  softmax(x_i) = exp(x_i) / summation(exp(x_j) for all j)
template <typename OcmTensorShape,
          bool masked = false,
          SoftmaxReduction reduction = (OcmTensorShape::NUM_CHN == 1) ? SoftmaxReduction::Spatial : SoftmaxReduction::Channel,
          typename NDArrayIn,
          typename NDArrayOut,
          typename Lambda = EmptyType,
          typename SinkParam = EmptyType,
          std::enable_if_t<reduction == SoftmaxReduction::Spatial ||
            reduction == SoftmaxReduction::Width ||
            reduction == SoftmaxReduction::Height, std::int32_t> = 0>
INLINE void softmax(NDArrayIn& qInBuffer,
                    NDArrayOut& qOutBuffer,
                    const std::int32_t& maskThresh = OcmTensorShape::NUM_ELEMS_PER_BCH,
                    const Lambda& postProc = EmptyType(),
                    const SinkParam& sink = EmptyType())
NameTypeDefaultDescription
OcmTensorShapetypenameThe shape of the OCM that we get values from.
maskedboolfalseWhether or not to apply a causal mask to this softmax.
reductionSoftmaxReduction(OcmTensorShape::NUM_CHN == 1) ? SoftmaxReduction::Spatial : SoftmaxReduction::ChannelWhich axis the softmax reduces over (Spatial for single-channel input).
NDArrayIntypenameThe input NDArray type.
NDArrayOuttypenameThe output NDArray type.
LambdatypenameEmptyTypeThe type of the lambda postprocessing function.
SinkParamtypenameEmptyType
NameTypeDescription
qInBufferNDArrayIn&The input buffer.
qOutBufferNDArrayOut&The output buffer.
maskThreshconst std::int32_t&The mask threshold to be apply if masked=true.
postProcconst Lambda&The postprocessing function.
sinkconst SinkParam&Optional extra logit that joins the denominator and is then dropped; EmptyType compiles the sink path out. Must carry the same frac bits as the input logits.

INLINE void

Mish

mishfunction
File: /src/neuralNetBlocks/activations.hppLines 606–746

This function calculates mish activation: f(x) = x * tanh(log(exp(x)+1)).

template <typename T>
qVar_t<T> mish(qVar_t<T> x)
NameTypeDefaultDescription
TtypenameThe type of the data
NameTypeDescription
xqVar_t<T>A single tile of data

qVar_t<T> — result Mish activation function outputs. / // clang-format on

Sign in to your account

Don't have an account? 
By signing in, you are agreeing to our Terms of Use and Privacy Policy.
Quadric // One architecture. Every algorithm.

Develop.

Simulate.

Profile.

Collaborate.