NOTE: The Jupyter Notebook below is included in the Chimera SDK and can be run interactively by running the following CLI command:
$ quadric sdk notebook
From the Jupyter Notebook window in your browser, select the notebook named /quadric/sdk-cli/examples/custom_op/tutorial_custom_op_without_onnx_equivalent.ipynb.
Tutorial: Custom Operator Insertion for Operators without an ONNX Equivalent
In this tutorial, we'll review the Open Neural Network Exchange (ONNX) graph representation framework, why Quadric selected it as its primary ingestion format for the Chimera Graph Compiler (CGC), and some situations when converting your entire model to ONNX format may be an inefficient process for porting your model to the Chimera Processor Architecture.
Lastly, we'll walk-through the custom operator flow for one of these situations, a YOLOv4 model's object-detector head, to demonstrate how a developer might port their model to the Chimera Processor architecture in the most efficient way.
Open Neural Network Exchange (ONNX)
The Chimera Graph Compiler (CGC) requires that Deep Neural Network (DNN) models be represented in the Open Neural Network Exchange (ONNX) format to run compilation.
ONNX is an open-source format for representing DNN models and is designed to enable AI developers to more easily use, compile, and deploy models trained with different Deep Learning (DL) frameworks, e.g. PyTorch, TensorFlow, TensorFlow Lite, Caffe, etc. ONNX accomplishes this by defining a common set of operators - the building blocks of deep learning models - and a conversion paradigm for representing these operators in each of the aforementioned DL frameworks.
Why Quadric chose ONNX
Quadric has elected to use ONNX as its initial ingestion format for the CGC so that the toolchain is accessible to all developers, regardless of their preferred DL framework for training.
What are ONNX OpSets?
The Deep Learning community and its techniques are rapidly changing as more efficient and effective methods are discovered.
ONNX adapts to these changes by periodically releasing updates to their common set of operators that are used to build and represent DNNs. These updates are called ONNX OpSets. A list of all ONNX operators, past and present, and the OpSet with which they were introduced or edited can be found here.
When converting to ONNX is a bad idea
ONNX is not a popular DL framework for model development and training. Instead, it excels at conversion between frameworks, operator canonicalization, and deployment to runtime environments. Thus, ONNX is often if not always behind in supporting the latest operators developer for the newest model architectures. It cannot support conversion of an operator between different DL frameworks if it only exists in one framework.
This process does have its advantages. ONNX may not be a first-mover in the DL space; but when it does add support for an operator, it does so because the operator has demonstrated widespread value across numerous DNN architectures. Since ONNX is also used to quickly port models to a vareity of supported runtime environments, ONNX has a vested interest in keeping the number of operators that it supports in each of its OpSets lean.
Despite this context, it may still surprise some users to learn that some very popular model architectures have custom operators defined in their native DL frameworks that have not been adopted by ONNX. For example, none of the popular YOLO object detection layers have been supported natively by ONNX.
ONNX is not wrong for refusing to canonicalize these operators. They are the algorithms that have changed the most between each new version of the YOLO architecture and supporting them in ONNX might quickly
Example: YOLOv4 Object-Detector Head
For the remainder of this tutorial, we'll focus on one model - the YOLOv4 Object Detector - to demonstrate when writing a custom operator for an algorothm without an equivalent in ONNX is simpler than attempting to convert that model to ONNX.
We'll start by looking at the PyTorch implementation of the YOLOv4 adapted from this open-source GitHub repository: https://github.com/Tianxiaomo/pytorch-YOLOv4
NOTE: It's not necessary for you to understand all of the code in the block below. You may skip ahead to the tutorials steps and refer back as needed.
import sys
import numpy as np
import torch
from torch import nn
import torch.nn.functional as F
## PyTorch Modules used to Build YOLOv4 Model - copied from: https://github.com/Tianxiaomo/pytorch-YOLOv4/blob/master/models.py
## ----------------------------------------------------------------------------------------------------------------------------
class Mish(torch.nn.Module):
def __init__(self):
super().__init__()
def forward(self, x):
x = x * (torch.tanh(torch.nn.functional.softplus(x)))
return x
class Upsample(nn.Module):
def __init__(self):
super(Upsample, self).__init__()
def forward(self, x, target_size, inference=False):
assert x.data.dim() == 4
# _, _, tH, tW = target_size
if inference:
# B = x.data.size(0)
# C = x.data.size(1)
# H = x.data.size(2)
# W = x.data.size(3)
return (
x.view(x.size(0), x.size(1), x.size(2), 1, x.size(3), 1)
.expand(
x.size(0),
x.size(1),
x.size(2),
target_size[2] // x.size(2),
x.size(3),
target_size[3] // x.size(3),
)
.contiguous()
.view(x.size(0), x.size(1), target_size[2], target_size[3])
)
else:
return F.interpolate(x, size=(target_size[2], target_size[3]), mode="nearest")
class Conv_Bn_Activation(nn.Module):
def __init__(
self,
in_channels,
out_channels,
kernel_size,
stride,
activation,
bn=True,
bias=False,
):
super().__init__()
pad = (kernel_size - 1) // 2
self.conv = nn.ModuleList()
if bias:
self.conv.append(nn.Conv2d(in_channels, out_channels, kernel_size, stride, pad))
else:
self.conv.append(
nn.Conv2d(in_channels, out_channels, kernel_size, stride, pad, bias=False)
)
if bn:
self.conv.append(nn.BatchNorm2d(out_channels))
if activation == "mish":
self.conv.append(Mish())
elif activation == "relu":
self.conv.append(nn.ReLU(inplace=True))
elif activation == "leaky":
self.conv.append(nn.LeakyReLU(0.1, inplace=True))
elif activation == "linear":
pass
else:
print(
"activate error !!! {} {} {}".format(
sys._getframe().f_code.co_filename,
sys._getframe().f_code.co_name,
sys._getframe().f_lineno,
)
)
def forward(self, x):
for l in self.conv:
x = l(x)
return x
class ResBlock(nn.Module):
"""
Sequential residual blocks each of which consists of \
two convolution layers.
Args:
ch (int): number of input and output channels.
nblocks (int): number of residual blocks.
shortcut (bool): if True, residual tensor addition is enabled.
"""
def __init__(self, ch, nblocks=1, shortcut=True):
super().__init__()
self.shortcut = shortcut
self.module_list = nn.ModuleList()
for i in range(nblocks):
resblock_one = nn.ModuleList()
resblock_one.append(Conv_Bn_Activation(ch, ch, 1, 1, "mish"))
resblock_one.append(Conv_Bn_Activation(ch, ch, 3, 1, "mish"))
self.module_list.append(resblock_one)
def forward(self, x):
for module in self.module_list:
h = x
for res in module:
h = res(h)
x = x + h if self.shortcut else h
return x
class DownSample1(nn.Module):
def __init__(self):
super().__init__()
self.conv1 = Conv_Bn_Activation(3, 32, 3, 1, "mish")
self.conv2 = Conv_Bn_Activation(32, 64, 3, 2, "mish")
self.conv3 = Conv_Bn_Activation(64, 64, 1, 1, "mish")
# [route]
# layers = -2
self.conv4 = Conv_Bn_Activation(64, 64, 1, 1, "mish")
self.conv5 = Conv_Bn_Activation(64, 32, 1, 1, "mish")
self.conv6 = Conv_Bn_Activation(32, 64, 3, 1, "mish")
# [shortcut]
# from=-3
# activation = linear
self.conv7 = Conv_Bn_Activation(64, 64, 1, 1, "mish")
# [route]
# layers = -1, -7
self.conv8 = Conv_Bn_Activation(128, 64, 1, 1, "mish")
def forward(self, input):
x1 = self.conv1(input)
x2 = self.conv2(x1)
x3 = self.conv3(x2)
# route -2
x4 = self.conv4(x2)
x5 = self.conv5(x4)
x6 = self.conv6(x5)
# shortcut -3
x6 = x6 + x4
x7 = self.conv7(x6)
# [route]
# layers = -1, -7
x7 = torch.cat([x7, x3], dim=1)
x8 = self.conv8(x7)
return x8
class DownSample2(nn.Module):
def __init__(self):
super().__init__()
self.conv1 = Conv_Bn_Activation(64, 128, 3, 2, "mish")
self.conv2 = Conv_Bn_Activation(128, 64, 1, 1, "mish")
# r -2
self.conv3 = Conv_Bn_Activation(128, 64, 1, 1, "mish")
self.resblock = ResBlock(ch=64, nblocks=2)
# s -3
self.conv4 = Conv_Bn_Activation(64, 64, 1, 1, "mish")
# r -1 -10
self.conv5 = Conv_Bn_Activation(128, 128, 1, 1, "mish")
def forward(self, input):
x1 = self.conv1(input)
x2 = self.conv2(x1)
x3 = self.conv3(x1)
r = self.resblock(x3)
x4 = self.conv4(r)
x4 = torch.cat([x4, x2], dim=1)
x5 = self.conv5(x4)
return x5
class DownSample3(nn.Module):
def __init__(self):
super().__init__()
self.conv1 = Conv_Bn_Activation(128, 256, 3, 2, "mish")
self.conv2 = Conv_Bn_Activation(256, 128, 1, 1, "mish")
self.conv3 = Conv_Bn_Activation(256, 128, 1, 1, "mish")
self.resblock = ResBlock(ch=128, nblocks=8)
self.conv4 = Conv_Bn_Activation(128, 128, 1, 1, "mish")
self.conv5 = Conv_Bn_Activation(256, 256, 1, 1, "mish")
def forward(self, input):
x1 = self.conv1(input)
x2 = self.conv2(x1)
x3 = self.conv3(x1)
r = self.resblock(x3)
x4 = self.conv4(r)
x4 = torch.cat([x4, x2], dim=1)
x5 = self.conv5(x4)
return x5
class DownSample4(nn.Module):
def __init__(self):
super().__init__()
self.conv1 = Conv_Bn_Activation(256, 512, 3, 2, "mish")
self.conv2 = Conv_Bn_Activation(512, 256, 1, 1, "mish")
self.conv3 = Conv_Bn_Activation(512, 256, 1, 1, "mish")
self.resblock = ResBlock(ch=256, nblocks=8)
self.conv4 = Conv_Bn_Activation(256, 256, 1, 1, "mish")
self.conv5 = Conv_Bn_Activation(512, 512, 1, 1, "mish")
def forward(self, input):
x1 = self.conv1(input)
x2 = self.conv2(x1)
x3 = self.conv3(x1)
r = self.resblock(x3)
x4 = self.conv4(r)
x4 = torch.cat([x4, x2], dim=1)
x5 = self.conv5(x4)
return x5
class DownSample5(nn.Module):
def __init__(self):
super().__init__()
self.conv1 = Conv_Bn_Activation(512, 1024, 3, 2, "mish")
self.conv2 = Conv_Bn_Activation(1024, 512, 1, 1, "mish")
self.conv3 = Conv_Bn_Activation(1024, 512, 1, 1, "mish")
self.resblock = ResBlock(ch=512, nblocks=4)
self.conv4 = Conv_Bn_Activation(512, 512, 1, 1, "mish")
self.conv5 = Conv_Bn_Activation(1024, 1024, 1, 1, "mish")
def forward(self, input):
x1 = self.conv1(input)
x2 = self.conv2(x1)
x3 = self.conv3(x1)
r = self.resblock(x3)
x4 = self.conv4(r)
x4 = torch.cat([x4, x2], dim=1)
x5 = self.conv5(x4)
return x5
class Neck(nn.Module):
def __init__(self, inference=False):
super().__init__()
self.inference = inference
self.conv1 = Conv_Bn_Activation(1024, 512, 1, 1, "leaky")
self.conv2 = Conv_Bn_Activation(512, 1024, 3, 1, "leaky")
self.conv3 = Conv_Bn_Activation(1024, 512, 1, 1, "leaky")
# SPP
self.maxpool1 = nn.MaxPool2d(kernel_size=5, stride=1, padding=5 // 2)
self.maxpool2 = nn.MaxPool2d(kernel_size=9, stride=1, padding=9 // 2)
self.maxpool3 = nn.MaxPool2d(kernel_size=13, stride=1, padding=13 // 2)
# R -1 -3 -5 -6
# SPP
self.conv4 = Conv_Bn_Activation(2048, 512, 1, 1, "leaky")
self.conv5 = Conv_Bn_Activation(512, 1024, 3, 1, "leaky")
self.conv6 = Conv_Bn_Activation(1024, 512, 1, 1, "leaky")
self.conv7 = Conv_Bn_Activation(512, 256, 1, 1, "leaky")
# UP
self.upsample1 = Upsample()
# R 85
self.conv8 = Conv_Bn_Activation(512, 256, 1, 1, "leaky")
# R -1 -3
self.conv9 = Conv_Bn_Activation(512, 256, 1, 1, "leaky")
self.conv10 = Conv_Bn_Activation(256, 512, 3, 1, "leaky")
self.conv11 = Conv_Bn_Activation(512, 256, 1, 1, "leaky")
self.conv12 = Conv_Bn_Activation(256, 512, 3, 1, "leaky")
self.conv13 = Conv_Bn_Activation(512, 256, 1, 1, "leaky")
self.conv14 = Conv_Bn_Activation(256, 128, 1, 1, "leaky")
# UP
self.upsample2 = Upsample()
# R 54
self.conv15 = Conv_Bn_Activation(256, 128, 1, 1, "leaky")
# R -1 -3
self.conv16 = Conv_Bn_Activation(256, 128, 1, 1, "leaky")
self.conv17 = Conv_Bn_Activation(128, 256, 3, 1, "leaky")
self.conv18 = Conv_Bn_Activation(256, 128, 1, 1, "leaky")
self.conv19 = Conv_Bn_Activation(128, 256, 3, 1, "leaky")
self.conv20 = Conv_Bn_Activation(256, 128, 1, 1, "leaky")
def forward(self, input, downsample4, downsample3, inference=False):
x1 = self.conv1(input)
x2 = self.conv2(x1)
x3 = self.conv3(x2)
# SPP
m1 = self.maxpool1(x3)
m2 = self.maxpool2(x3)
m3 = self.maxpool3(x3)
spp = torch.cat([m3, m2, m1, x3], dim=1)
# SPP end
x4 = self.conv4(spp)
x5 = self.conv5(x4)
x6 = self.conv6(x5)
x7 = self.conv7(x6)
# UP
up = self.upsample1(x7, downsample4.size(), self.inference)
# R 85
x8 = self.conv8(downsample4)
# R -1 -3
x8 = torch.cat([x8, up], dim=1)
x9 = self.conv9(x8)
x10 = self.conv10(x9)
x11 = self.conv11(x10)
x12 = self.conv12(x11)
x13 = self.conv13(x12)
x14 = self.conv14(x13)
# UP
up = self.upsample2(x14, downsample3.size(), self.inference)
# R 54
x15 = self.conv15(downsample3)
# R -1 -3
x15 = torch.cat([x15, up], dim=1)
x16 = self.conv16(x15)
x17 = self.conv17(x16)
x18 = self.conv18(x17)
x19 = self.conv19(x18)
x20 = self.conv20(x19)
return x20, x13, x6
class YoloLayer(nn.Module):
"""Yolo layer
model_out: while inference,is post-processing inside or outside the model
true:outside
"""
def __init__(
self,
anchor_mask=[],
num_classes=0,
anchors=[],
num_anchors=1,
stride=32,
model_out=False,
):
super(YoloLayer, self).__init__()
self.anchor_mask = anchor_mask
self.num_classes = num_classes
self.anchors = anchors
self.num_anchors = num_anchors
self.anchor_step = len(anchors) // num_anchors
self.coord_scale = 1
self.noobject_scale = 1
self.object_scale = 5
self.class_scale = 1
self.thresh = 0.6
self.stride = stride
self.seen = 0
self.scale_x_y = 1
self.model_out = model_out
def yolo_forward_dynamic(
self,
output,
conf_thresh,
num_classes,
anchors,
num_anchors,
only_objectness=1,
validation=False,
):
# Output would be invalid if it does not satisfy this assert
# assert (output.size(1) == (5 + num_classes) * num_anchors)
# print(output.size())
# Slice the second dimension (channel) of output into:
# [ 2, 2, 1, num_classes, 2, 2, 1, num_classes, 2, 2, 1, num_classes ]
# And then into
# bxy = [ 6 ] bwh = [ 6 ] det_conf = [ 3 ] cls_conf = [ num_classes * 3 ]
# batch = output.size(0)
# H = output.size(2)
# W = output.size(3)
bxy_list = []
bwh_list = []
det_confs_list = []
cls_confs_list = []
for i in range(num_anchors):
begin = i * (5 + num_classes)
end = (i + 1) * (5 + num_classes)
bxy_list.append(output[:, begin : begin + 2])
bwh_list.append(output[:, begin + 2 : begin + 4])
det_confs_list.append(output[:, begin + 4 : begin + 5])
cls_confs_list.append(output[:, begin + 5 : end])
# Shape: [batch, num_anchors * 2, H, W]
bxy = torch.cat(bxy_list, dim=1)
# Shape: [batch, num_anchors * 2, H, W]
bwh = torch.cat(bwh_list, dim=1)
# Shape: [batch, num_anchors, H, W]
det_confs = torch.cat(det_confs_list, dim=1)
# Shape: [batch, num_anchors * H * W]
det_confs = det_confs.view(output.size(0), num_anchors * output.size(2) * output.size(3))
# Shape: [batch, num_anchors * num_classes, H, W]
cls_confs = torch.cat(cls_confs_list, dim=1)
# Shape: [batch, num_anchors, num_classes, H * W]
cls_confs = cls_confs.view(
output.size(0), num_anchors, num_classes, output.size(2) * output.size(3)
)
# Shape: [batch, num_anchors, num_classes, H * W] --> [batch, num_anchors * H * W, num_classes]
cls_confs = cls_confs.permute(0, 1, 3, 2).reshape(
output.size(0), num_anchors * output.size(2) * output.size(3), num_classes
)
# Apply sigmoid(), exp() and softmax() to slices
#
bxy = torch.sigmoid(bxy) * self.scale_x_y - 0.5 * (self.scale_x_y - 1)
bwh = torch.exp(bwh)
det_confs = torch.sigmoid(det_confs)
cls_confs = torch.sigmoid(cls_confs)
# Prepare C-x, C-y, P-w, P-h (None of them are torch related)
grid_x = np.expand_dims(
np.expand_dims(
np.expand_dims(np.linspace(0, output.size(3) - 1, output.size(3)), axis=0).repeat(
output.size(2), 0
),
axis=0,
),
axis=0,
)
grid_y = np.expand_dims(
np.expand_dims(
np.expand_dims(np.linspace(0, output.size(2) - 1, output.size(2)), axis=1).repeat(
output.size(3), 1
),
axis=0,
),
axis=0,
)
# grid_x = torch.linspace(0, W - 1, W).reshape(1, 1, 1, W).repeat(1, 1, H, 1)
# grid_y = torch.linspace(0, H - 1, H).reshape(1, 1, H, 1).repeat(1, 1, 1, W)
anchor_w = []
anchor_h = []
for i in range(num_anchors):
anchor_w.append(anchors[i * 2])
anchor_h.append(anchors[i * 2 + 1])
device = None
cuda_check = output.is_cuda
if cuda_check:
device = output.get_device()
bx_list = []
by_list = []
bw_list = []
bh_list = []
# Apply C-x, C-y, P-w, P-h
for i in range(num_anchors):
ii = i * 2
# Shape: [batch, 1, H, W]
bx = bxy[:, ii : ii + 1] + torch.tensor(
grid_x, device=device, dtype=torch.float32
) # grid_x.to(device=device, dtype=torch.float32)
# Shape: [batch, 1, H, W]
by = bxy[:, ii + 1 : ii + 2] + torch.tensor(
grid_y, device=device, dtype=torch.float32
) # grid_y.to(device=device, dtype=torch.float32)
# Shape: [batch, 1, H, W]
bw = bwh[:, ii : ii + 1] * anchor_w[i]
# Shape: [batch, 1, H, W]
bh = bwh[:, ii + 1 : ii + 2] * anchor_h[i]
bx_list.append(bx)
by_list.append(by)
bw_list.append(bw)
bh_list.append(bh)
########################################
# Figure out bboxes from slices #
########################################
# Shape: [batch, num_anchors, H, W]
bx = torch.cat(bx_list, dim=1)
# Shape: [batch, num_anchors, H, W]
by = torch.cat(by_list, dim=1)
# Shape: [batch, num_anchors, H, W]
bw = torch.cat(bw_list, dim=1)
# Shape: [batch, num_anchors, H, W]
bh = torch.cat(bh_list, dim=1)
# Shape: [batch, 2 * num_anchors, H, W]
bx_bw = torch.cat((bx, bw), dim=1)
# Shape: [batch, 2 * num_anchors, H, W]
by_bh = torch.cat((by, bh), dim=1)
# normalize coordinates to [0, 1]
bx_bw /= output.size(3)
by_bh /= output.size(2)
# Shape: [batch, num_anchors * H * W, 1]
bx = bx_bw[:, :num_anchors].view(
output.size(0), num_anchors * output.size(2) * output.size(3), 1
)
by = by_bh[:, :num_anchors].view(
output.size(0), num_anchors * output.size(2) * output.size(3), 1
)
bw = bx_bw[:, num_anchors:].view(
output.size(0), num_anchors * output.size(2) * output.size(3), 1
)
bh = by_bh[:, num_anchors:].view(
output.size(0), num_anchors * output.size(2) * output.size(3), 1
)
bx1 = bx - bw * 0.5
by1 = by - bh * 0.5
bx2 = bx1 + bw
by2 = by1 + bh
# Shape: [batch, num_anchors * h * w, 4] -> [batch, num_anchors * h * w, 1, 4]
boxes = torch.cat((bx1, by1, bx2, by2), dim=2).view(
output.size(0), num_anchors * output.size(2) * output.size(3), 1, 4
)
# boxes = boxes.repeat(1, 1, num_classes, 1)
# boxes: [batch, num_anchors * H * W, 1, 4]
# cls_confs: [batch, num_anchors * H * W, num_classes]
# det_confs: [batch, num_anchors * H * W]
det_confs = det_confs.view(output.size(0), num_anchors * output.size(2) * output.size(3), 1)
confs = cls_confs * det_confs
# boxes: [batch, num_anchors * H * W, 1, 4]
# confs: [batch, num_anchors * H * W, num_classes]
return boxes, confs
def forward(self, output, target=None):
if self.training:
return output
masked_anchors = []
for m in self.anchor_mask:
masked_anchors += self.anchors[m * self.anchor_step : (m + 1) * self.anchor_step]
masked_anchors = [anchor / self.stride for anchor in masked_anchors]
return self.yolo_forward_dynamic(
output, self.thresh, self.num_classes, masked_anchors, len(self.anchor_mask)
)
class Yolov4Head(nn.Module):
def __init__(self, output_ch, n_classes, inference=False):
super().__init__()
self.inference = inference
self.conv1 = Conv_Bn_Activation(128, 256, 3, 1, "leaky")
self.conv2 = Conv_Bn_Activation(256, output_ch, 1, 1, "linear", bn=False, bias=True)
self.yolo1 = YoloLayer(
anchor_mask=[0, 1, 2],
num_classes=n_classes,
anchors=[
12,
16,
19,
36,
40,
28,
36,
75,
76,
55,
72,
146,
142,
110,
192,
243,
459,
401,
],
num_anchors=9,
stride=8,
)
# R -4
self.conv3 = Conv_Bn_Activation(128, 256, 3, 2, "leaky")
# R -1 -16
self.conv4 = Conv_Bn_Activation(512, 256, 1, 1, "leaky")
self.conv5 = Conv_Bn_Activation(256, 512, 3, 1, "leaky")
self.conv6 = Conv_Bn_Activation(512, 256, 1, 1, "leaky")
self.conv7 = Conv_Bn_Activation(256, 512, 3, 1, "leaky")
self.conv8 = Conv_Bn_Activation(512, 256, 1, 1, "leaky")
self.conv9 = Conv_Bn_Activation(256, 512, 3, 1, "leaky")
self.conv10 = Conv_Bn_Activation(512, output_ch, 1, 1, "linear", bn=False, bias=True)
self.yolo2 = YoloLayer(
anchor_mask=[3, 4, 5],
num_classes=n_classes,
anchors=[
12,
16,
19,
36,
40,
28,
36,
75,
76,
55,
72,
146,
142,
110,
192,
243,
459,
401,
],
num_anchors=9,
stride=16,
)
# R -4
self.conv11 = Conv_Bn_Activation(256, 512, 3, 2, "leaky")
# R -1 -37
self.conv12 = Conv_Bn_Activation(1024, 512, 1, 1, "leaky")
self.conv13 = Conv_Bn_Activation(512, 1024, 3, 1, "leaky")
self.conv14 = Conv_Bn_Activation(1024, 512, 1, 1, "leaky")
self.conv15 = Conv_Bn_Activation(512, 1024, 3, 1, "leaky")
self.conv16 = Conv_Bn_Activation(1024, 512, 1, 1, "leaky")
self.conv17 = Conv_Bn_Activation(512, 1024, 3, 1, "leaky")
self.conv18 = Conv_Bn_Activation(1024, output_ch, 1, 1, "linear", bn=False, bias=True)
self.yolo3 = YoloLayer(
anchor_mask=[6, 7, 8],
num_classes=n_classes,
anchors=[
12,
16,
19,
36,
40,
28,
36,
75,
76,
55,
72,
146,
142,
110,
192,
243,
459,
401,
],
num_anchors=9,
stride=32,
)
def forward(self, input1, input2, input3):
x1 = self.conv1(input1)
x2 = self.conv2(x1)
x3 = self.conv3(input1)
# R -1 -16
x3 = torch.cat([x3, input2], dim=1)
x4 = self.conv4(x3)
x5 = self.conv5(x4)
x6 = self.conv6(x5)
x7 = self.conv7(x6)
x8 = self.conv8(x7)
x9 = self.conv9(x8)
x10 = self.conv10(x9)
# R -4
x11 = self.conv11(x8)
# R -1 -37
x11 = torch.cat([x11, input3], dim=1)
x12 = self.conv12(x11)
x13 = self.conv13(x12)
x14 = self.conv14(x13)
x15 = self.conv15(x14)
x16 = self.conv16(x15)
x17 = self.conv17(x16)
x18 = self.conv18(x17)
if self.inference:
y1 = self.yolo1(x2)
y2 = self.yolo2(x10)
y3 = self.yolo3(x18)
return self.get_region_boxes([y1, y2, y3])
else:
return [x2, x10, x18]
def get_region_boxes(self, boxes_and_confs):
# print('Getting boxes from boxes and confs ...')
boxes_list = []
confs_list = []
for item in boxes_and_confs:
boxes_list.append(item[0])
confs_list.append(item[1])
# boxes: [batch, num1 + num2 + num3, 1, 4]
# confs: [batch, num1 + num2 + num3, num_classes]
boxes = torch.cat(boxes_list, dim=1)
confs = torch.cat(confs_list, dim=1)
return [boxes, confs]
class Yolov4(nn.Module):
def __init__(self, yolov4conv137weight=None, n_classes=80, inference=False):
super().__init__()
output_ch = (4 + 1 + n_classes) * 3
# backbone
self.down1 = DownSample1()
self.down2 = DownSample2()
self.down3 = DownSample3()
self.down4 = DownSample4()
self.down5 = DownSample5()
# neck
self.neck = Neck(inference)
# yolov4conv137
if yolov4conv137weight:
_model = nn.Sequential(
self.down1, self.down2, self.down3, self.down4, self.down5, self.neck
)
pretrained_dict = torch.load(yolov4conv137weight)
model_dict = _model.state_dict()
# 1. filter out unnecessary keys
pretrained_dict = {k1: v for (k, v), k1 in zip(pretrained_dict.items(), model_dict)}
# 2. overwrite entries in the existing state dict
model_dict.update(pretrained_dict)
_model.load_state_dict(model_dict)
# head
self.head = Yolov4Head(output_ch, n_classes, inference)
def forward(self, input):
d1 = self.down1(input)
d2 = self.down2(d1)
d3 = self.down3(d2)
d4 = self.down4(d3)
d5 = self.down5(d4)
x20, x13, x6 = self.neck(d5, d4, d3)
output = self.head(x20, x13, x6)
return output
1. Next, install the following PIP requirements from the pytorch-YOLOv4 repo in a Python environment outside of this notebook:
```sh
$ pip install numpy torch>=1.4.0 torchinfo netron
### 2. Optional: Follow the instructions in the `README.md` of the `pytorch-YOLOv4` repo for downloading pre-trained model weights from the author's Google Drive or Baidu Wangpan.
### 3. Change global variables, if needed:
```python
## Update this variable if you've downloaded pre-trained weights for the YOLOv4 Model
## E.g. 'yolov4.conv.137.pth'
PATH_TO_DOWNLOADED_WEIGHTS = None
## Below we set the number of classes to `80` because the COCO dataset that is often used to benchmark Object Detection models has 80 classes
## More on the COCO dataset: https://cocodataset.org
## NOTE: If the weights you download or train have a different number of classes, update this variable
NUM_MODEL_CLASSES = 80
## NOTE: Below defines the expected shape(s) of the YOLOv4 Model's Inputs
## E.g. (1, 3, 608, 608)
INPUT_BATCH_SIZE = 1
INPUT_IMAGE_CHANNELS = 3
INPUT_IMAGE_HEIGHT = 608
INPUT_IMAGE_WIDTH = 608
4. Print the YOLOv4 model summary:
from torchinfo import summary
model = Yolov4(n_classes=80, inference=True)
summary(model, input_size=(INPUT_BATCH_SIZE, 3, INPUT_IMAGE_HEIGHT, INPUT_IMAGE_WIDTH))
=========================================================================================================
Layer (type:depth-idx) Output Shape Param #
=========================================================================================================
Yolov4 [1, 22743, 1, 4] --
├─DownSample1: 1-1 [1, 64, 304, 304] --
│ └─Conv_Bn_Activation: 2-1 [1, 32, 608, 608] --
│ │ └─ModuleList: 3-1 -- 928
│ └─Conv_Bn_Activation: 2-2 [1, 64, 304, 304] --
│ │ └─ModuleList: 3-2 -- 18,560
│ └─Conv_Bn_Activation: 2-3 [1, 64, 304, 304] --
│ │ └─ModuleList: 3-3 -- 4,224
│ └─Conv_Bn_Activation: 2-4 [1, 64, 304, 304] --
│ │ └─ModuleList: 3-4 -- 4,224
│ └─Conv_Bn_Activation: 2-5 [1, 32, 304, 304] --
│ │ └─ModuleList: 3-5 -- 2,112
│ └─Conv_Bn_Activation: 2-6 [1, 64, 304, 304] --
│ │ └─ModuleList: 3-6 -- 18,560
│ └─Conv_Bn_Activation: 2-7 [1, 64, 304, 304] --
│ │ └─ModuleList: 3-7 -- 4,224
│ └─Conv_Bn_Activation: 2-8 [1, 64, 304, 304] --
│ │ └─ModuleList: 3-8 -- 8,320
├─DownSample2: 1-2 [1, 128, 152, 152] --
│ └─Conv_Bn_Activation: 2-9 [1, 128, 152, 152] --
│ │ └─ModuleList: 3-9 -- 73,984
│ └─Conv_Bn_Activation: 2-10 [1, 64, 152, 152] --
│ │ └─ModuleList: 3-10 -- 8,320
│ └─Conv_Bn_Activation: 2-11 [1, 64, 152, 152] --
│ │ └─ModuleList: 3-11 -- 8,320
│ └─ResBlock: 2-12 [1, 64, 152, 152] --
│ │ └─ModuleList: 3-12 -- 82,432
│ └─Conv_Bn_Activation: 2-13 [1, 64, 152, 152] --
│ │ └─ModuleList: 3-13 -- 4,224
│ └─Conv_Bn_Activation: 2-14 [1, 128, 152, 152] --
│ │ └─ModuleList: 3-14 -- 16,640
├─DownSample3: 1-3 [1, 256, 76, 76] --
│ └─Conv_Bn_Activation: 2-15 [1, 256, 76, 76] --
│ │ └─ModuleList: 3-15 -- 295,424
│ └─Conv_Bn_Activation: 2-16 [1, 128, 76, 76] --
│ │ └─ModuleList: 3-16 -- 33,024
│ └─Conv_Bn_Activation: 2-17 [1, 128, 76, 76] --
│ │ └─ModuleList: 3-17 -- 33,024
│ └─ResBlock: 2-18 [1, 128, 76, 76] --
│ │ └─ModuleList: 3-18 -- 1,314,816
│ └─Conv_Bn_Activation: 2-19 [1, 128, 76, 76] --
│ │ └─ModuleList: 3-19 -- 16,640
│ └─Conv_Bn_Activation: 2-20 [1, 256, 76, 76] --
│ │ └─ModuleList: 3-20 -- 66,048
├─DownSample4: 1-4 [1, 512, 38, 38] --
│ └─Conv_Bn_Activation: 2-21 [1, 512, 38, 38] --
│ │ └─ModuleList: 3-21 -- 1,180,672
│ └─Conv_Bn_Activation: 2-22 [1, 256, 38, 38] --
│ │ └─ModuleList: 3-22 -- 131,584
│ └─Conv_Bn_Activation: 2-23 [1, 256, 38, 38] --
│ │ └─ModuleList: 3-23 -- 131,584
│ └─ResBlock: 2-24 [1, 256, 38, 38] --
│ │ └─ModuleList: 3-24 -- 5,251,072
│ └─Conv_Bn_Activation: 2-25 [1, 256, 38, 38] --
│ │ └─ModuleList: 3-25 -- 66,048
│ └─Conv_Bn_Activation: 2-26 [1, 512, 38, 38] --
│ │ └─ModuleList: 3-26 -- 263,168
├─DownSample5: 1-5 [1, 1024, 19, 19] --
│ └─Conv_Bn_Activation: 2-27 [1, 1024, 19, 19] --
│ │ └─ModuleList: 3-27 -- 4,720,640
│ └─Conv_Bn_Activation: 2-28 [1, 512, 19, 19] --
│ │ └─ModuleList: 3-28 -- 525,312
│ └─Conv_Bn_Activation: 2-29 [1, 512, 19, 19] --
│ │ └─ModuleList: 3-29 -- 525,312
│ └─ResBlock: 2-30 [1, 512, 19, 19] --
│ │ └─ModuleList: 3-30 -- 10,493,952
│ └─Conv_Bn_Activation: 2-31 [1, 512, 19, 19] --
│ │ └─ModuleList: 3-31 -- 263,168
│ └─Conv_Bn_Activation: 2-32 [1, 1024, 19, 19] --
│ │ └─ModuleList: 3-32 -- 1,050,624
├─Neck: 1-6 [1, 128, 76, 76] --
│ └─Conv_Bn_Activation: 2-33 [1, 512, 19, 19] --
│ │ └─ModuleList: 3-33 -- 525,312
│ └─Conv_Bn_Activation: 2-34 [1, 1024, 19, 19] --
│ │ └─ModuleList: 3-34 -- 4,720,640
│ └─Conv_Bn_Activation: 2-35 [1, 512, 19, 19] --
│ │ └─ModuleList: 3-35 -- 525,312
│ └─MaxPool2d: 2-36 [1, 512, 19, 19] --
│ └─MaxPool2d: 2-37 [1, 512, 19, 19] --
│ └─MaxPool2d: 2-38 [1, 512, 19, 19] --
│ └─Conv_Bn_Activation: 2-39 [1, 512, 19, 19] --
│ │ └─ModuleList: 3-36 -- 1,049,600
│ └─Conv_Bn_Activation: 2-40 [1, 1024, 19, 19] --
│ │ └─ModuleList: 3-37 -- 4,720,640
│ └─Conv_Bn_Activation: 2-41 [1, 512, 19, 19] --
│ │ └─ModuleList: 3-38 -- 525,312
│ └─Conv_Bn_Activation: 2-42 [1, 256, 19, 19] --
│ │ └─ModuleList: 3-39 -- 131,584
│ └─Upsample: 2-43 [1, 256, 38, 38] --
│ └─Conv_Bn_Activation: 2-44 [1, 256, 38, 38] --
│ │ └─ModuleList: 3-40 -- 131,584
│ └─Conv_Bn_Activation: 2-45 [1, 256, 38, 38] --
│ │ └─ModuleList: 3-41 -- 131,584
│ └─Conv_Bn_Activation: 2-46 [1, 512, 38, 38] --
│ │ └─ModuleList: 3-42 -- 1,180,672
│ └─Conv_Bn_Activation: 2-47 [1, 256, 38, 38] --
│ │ └─ModuleList: 3-43 -- 131,584
│ └─Conv_Bn_Activation: 2-48 [1, 512, 38, 38] --
│ │ └─ModuleList: 3-44 -- 1,180,672
│ └─Conv_Bn_Activation: 2-49 [1, 256, 38, 38] --
│ │ └─ModuleList: 3-45 -- 131,584
│ └─Conv_Bn_Activation: 2-50 [1, 128, 38, 38] --
│ │ └─ModuleList: 3-46 -- 33,024
│ └─Upsample: 2-51 [1, 128, 76, 76] --
│ └─Conv_Bn_Activation: 2-52 [1, 128, 76, 76] --
│ │ └─ModuleList: 3-47 -- 33,024
│ └─Conv_Bn_Activation: 2-53 [1, 128, 76, 76] --
│ │ └─ModuleList: 3-48 -- 33,024
│ └─Conv_Bn_Activation: 2-54 [1, 256, 76, 76] --
│ │ └─ModuleList: 3-49 -- 295,424
│ └─Conv_Bn_Activation: 2-55 [1, 128, 76, 76] --
│ │ └─ModuleList: 3-50 -- 33,024
│ └─Conv_Bn_Activation: 2-56 [1, 256, 76, 76] --
│ │ └─ModuleList: 3-51 -- 295,424
│ └─Conv_Bn_Activation: 2-57 [1, 128, 76, 76] --
│ │ └─ModuleList: 3-52 -- 33,024
├─Yolov4Head: 1-7 [1, 22743, 1, 4] --
│ └─Conv_Bn_Activation: 2-58 [1, 256, 76, 76] --
│ │ └─ModuleList: 3-53 -- 295,424
│ └─Conv_Bn_Activation: 2-59 [1, 255, 76, 76] --
│ │ └─ModuleList: 3-54 -- 65,535
│ └─Conv_Bn_Activation: 2-60 [1, 256, 38, 38] --
│ │ └─ModuleList: 3-55 -- 295,424
│ └─Conv_Bn_Activation: 2-61 [1, 256, 38, 38] --
│ │ └─ModuleList: 3-56 -- 131,584
│ └─Conv_Bn_Activation: 2-62 [1, 512, 38, 38] --
│ │ └─ModuleList: 3-57 -- 1,180,672
│ └─Conv_Bn_Activation: 2-63 [1, 256, 38, 38] --
│ │ └─ModuleList: 3-58 -- 131,584
│ └─Conv_Bn_Activation: 2-64 [1, 512, 38, 38] --
│ │ └─ModuleList: 3-59 -- 1,180,672
│ └─Conv_Bn_Activation: 2-65 [1, 256, 38, 38] --
│ │ └─ModuleList: 3-60 -- 131,584
│ └─Conv_Bn_Activation: 2-66 [1, 512, 38, 38] --
│ │ └─ModuleList: 3-61 -- 1,180,672
│ └─Conv_Bn_Activation: 2-67 [1, 255, 38, 38] --
│ │ └─ModuleList: 3-62 -- 130,815
│ └─Conv_Bn_Activation: 2-68 [1, 512, 19, 19] --
│ │ └─ModuleList: 3-63 -- 1,180,672
│ └─Conv_Bn_Activation: 2-69 [1, 512, 19, 19] --
│ │ └─ModuleList: 3-64 -- 525,312
│ └─Conv_Bn_Activation: 2-70 [1, 1024, 19, 19] --
│ │ └─ModuleList: 3-65 -- 4,720,640
│ └─Conv_Bn_Activation: 2-71 [1, 512, 19, 19] --
│ │ └─ModuleList: 3-66 -- 525,312
│ └─Conv_Bn_Activation: 2-72 [1, 1024, 19, 19] --
│ │ └─ModuleList: 3-67 -- 4,720,640
│ └─Conv_Bn_Activation: 2-73 [1, 512, 19, 19] --
│ │ └─ModuleList: 3-68 -- 525,312
│ └─Conv_Bn_Activation: 2-74 [1, 1024, 19, 19] --
│ │ └─ModuleList: 3-69 -- 4,720,640
│ └─Conv_Bn_Activation: 2-75 [1, 255, 19, 19] --
│ │ └─ModuleList: 3-70 -- 261,375
│ └─YoloLayer: 2-76 [1, 17328, 1, 4] --
│ └─YoloLayer: 2-77 [1, 4332, 1, 4] --
│ └─YoloLayer: 2-78 [1, 1083, 1, 4] --
=========================================================================================================
Total params: 64,363,101
Trainable params: 64,363,101
Non-trainable params: 0
Total mult-adds (G): 64.20
=========================================================================================================
Input size (MB): 4.44
Forward/backward pass size (MB): 1814.99
Params size (MB): 257.45
Estimated Total Size (MB): 2076.88
=========================================================================================================
NOTE: The model output summary is long and may be truncated by your notebook with the default Jupyter notebook settings
Looking at the model summary and the model definition in PyTorch, it's important to note that the YoloLayer is an uncommon, custom nn.Module defined specifically for this model. This particular model architecture, has three YoloLayer layers at the very end of the model.
For the purposes of this tutorial, it's also important to examine the YoloLayer.forward function which calls the YoloLayer.yolo_forward_dynamic function. All torch.nn.Module functions must have a forward member function defined that overrides the virtual function in that parent class. This function is defines what the layer does during a forward-pass through the model, i.e. inference.
Below is a code snippet from the YoloLayer.yolo_forward_dynamic function:
bxy_list = []
bwh_list = []
det_confs_list = []
cls_confs_list = []
for i in range(num_anchors):
begin = i * (5 + num_classes)
end = (i + 1) * (5 + num_classes)
bxy_list.append(output[:, begin : begin + 2])
bwh_list.append(output[:, begin + 2 : begin + 4])
det_confs_list.append(output[:, begin + 4 : begin + 5])
cls_confs_list.append(output[:, begin + 5 : end])
## Shape: [batch, num_anchors * 2, H, W]
bxy = torch.cat(bxy_list, dim=1)
## Shape: [batch, num_anchors * 2, H, W]
bwh = torch.cat(bwh_list, dim=1)
## Shape: [batch, num_anchors, H, W]
det_confs = torch.cat(det_confs_list, dim=1)
## Shape: [batch, num_anchors * H * W]
det_confs = det_confs.view(output.size(0), num_anchors * output.size(2) * output.size(3))
## Shape: [batch, num_anchors * num_classes, H, W]
cls_confs = torch.cat(cls_confs_list, dim=1)
## Shape: [batch, num_anchors, num_classes, H * W]
cls_confs = cls_confs.view(output.size(0), num_anchors, num_classes, output.size(2) * output.size(3))
## Shape: [batch, num_anchors, num_classes, H * W] --> [batch, num_anchors * H * W, num_classes]
cls_confs = cls_confs.permute(0, 1, 3, 2).reshape(output.size(0), num_anchors * output.size(2) * output.size(3), num_classes)
## Apply sigmoid(), exp() and softmax() to slices
#
bxy = torch.sigmoid(bxy) * self.scale_x_y - 0.5 * (self.scale_x_y - 1)
bwh = torch.exp(bwh)
det_confs = torch.sigmoid(det_confs)
cls_confs = torch.sigmoid(cls_confs)
## Prepare C-x, C-y, P-w, P-h (None of them are torch related)
grid_x = np.expand_dims(np.expand_dims(np.expand_dims(np.linspace(0, output.size(3) - 1, output.size(3)), axis=0).repeat(output.size(2), 0), axis=0), axis=0)
grid_y = np.expand_dims(np.expand_dims(np.expand_dims(np.linspace(0, output.size(2) - 1, output.size(2)), axis=1).repeat(output.size(3), 1), axis=0), axis=0)
## grid_x = torch.linspace(0, W - 1, W).reshape(1, 1, 1, W).repeat(1, 1, H, 1)
## grid_y = torch.linspace(0, H - 1, H).reshape(1, 1, H, 1).repeat(1, 1, 1, W)
anchor_w = []
anchor_h = []
for i in range(num_anchors):
anchor_w.append(anchors[i * 2])
anchor_h.append(anchors[i * 2 + 1])
bx_list = []
by_list = []
bw_list = []
bh_list = []
## Apply C-x, C-y, P-w, P-h
for i in range(num_anchors):
ii = i * 2
# Shape: [batch, 1, H, W]
bx = bxy[:, ii : ii + 1] + torch.tensor(grid_x, device=device, dtype=torch.float32) # grid_x.to(device=device, dtype=torch.float32)
# Shape: [batch, 1, H, W]
by = bxy[:, ii + 1 : ii + 2] + torch.tensor(grid_y, device=device, dtype=torch.float32) # grid_y.to(device=device, dtype=torch.float32)
# Shape: [batch, 1, H, W]
bw = bwh[:, ii : ii + 1] * anchor_w[i]
# Shape: [batch, 1, H, W]
bh = bwh[:, ii + 1 : ii + 2] * anchor_h[i]
bx_list.append(bx)
by_list.append(by)
bw_list.append(bw)
bh_list.append(bh)
########################################
## Figure out bboxes from slices #
########################################
## Shape: [batch, num_anchors, H, W]
bx = torch.cat(bx_list, dim=1)
## Shape: [batch, num_anchors, H, W]
by = torch.cat(by_list, dim=1)
## Shape: [batch, num_anchors, H, W]
bw = torch.cat(bw_list, dim=1)
## Shape: [batch, num_anchors, H, W]
bh = torch.cat(bh_list, dim=1)
## Shape: [batch, 2 * num_anchors, H, W]
bx_bw = torch.cat((bx, bw), dim=1)
## Shape: [batch, 2 * num_anchors, H, W]
by_bh = torch.cat((by, bh), dim=1)
## normalize coordinates to [0, 1]
bx_bw /= output.size(3)
by_bh /= output.size(2)
## Shape: [batch, num_anchors * H * W, 1]
bx = bx_bw[:, :num_anchors].view(output.size(0), num_anchors * output.size(2) * output.size(3), 1)
by = by_bh[:, :num_anchors].view(output.size(0), num_anchors * output.size(2) * output.size(3), 1)
bw = bx_bw[:, num_anchors:].view(output.size(0), num_anchors * output.size(2) * output.size(3), 1)
bh = by_bh[:, num_anchors:].view(output.size(0), num_anchors * output.size(2) * output.size(3), 1)
bx1 = bx - bw * 0.5
by1 = by - bh * 0.5
bx2 = bx1 + bw
by2 = by1 + bh
## Shape: [batch, num_anchors * h * w, 4] -> [batch, num_anchors * h * w, 1, 4]
boxes = torch.cat((bx1, by1, bx2, by2), dim=2).view(output.size(0), num_anchors * output.size(2) * output.size(3), 1, 4)
## boxes = boxes.repeat(1, 1, num_classes, 1)
## boxes: [batch, num_anchors * H * W, 1, 4]
## cls_confs: [batch, num_anchors * H * W, num_classes]
## det_confs: [batch, num_anchors * H * W]
det_confs = det_confs.view(output.size(0), num_anchors * output.size(2) * output.size(3), 1)
confs = cls_confs * det_confs
## boxes: [batch, num_anchors * H * W, 1, 4]
## confs: [batch, num_anchors * H * W, num_classes]
Without going into too much detail, this function slices the concatenated tensor output into components:
bxy_list, the(x, y)coordinate-pairs of object bounding boxesbwh_list, the widths and heights of the object bounding boxesdet_confs_list, the 'objectiveness' scores for the objects found in the bounding boxescls_confs_list, the numerical class label of the object predicted to be in the bounding boxes
These components are the outputs of the YOLOv4 model's backbone and are object detections in the high-dimensional feature space. The feature space is a compressed grid of the original input image. The feature space makes a fixed-number of object detections for each grid space. This is why the model produces an 'objectiveness' score; the model must make a prediction even if it is confident that no object exists in the grid space.
This forward-pass function aggregates this component data, filters out the low-scoring predictions, and formats the high-scoring predictions into more easily understood bounding boxes and class confidence scores that can be rendered onto an image and visually verified by a human, like below:
This forward-pass function, as a developer, is easy enough to read, understand, and potentially port to a C++ representation. Let's continue to see how this algorithm looks after we've converted this model to ONNX format so that it can be compiled by the CGC.
5. Convert the model to ONNX
model = Yolov4(n_classes=80, inference=True)
if PATH_TO_DOWNLOADED_WEIGHTS:
pretrained_dict = torch.load(PATH_TO_DOWNLOADED_WEIGHTS, map_location=torch.device("cpu"))
model.load_state_dict(pretrained_dict)
input_names = ["input"]
output_names = ["boxes", "confs"]
x = torch.randn((INPUT_BATCH_SIZE, 3, INPUT_IMAGE_HEIGHT, INPUT_IMAGE_WIDTH), requires_grad=True)
onnx_file_name = "yolov4_{}_3_{}_{}_static.onnx".format(
INPUT_BATCH_SIZE, INPUT_IMAGE_HEIGHT, INPUT_IMAGE_WIDTH
)
## Export the model
print("Export the onnx model ...")
torch.onnx.export(
model,
x,
onnx_file_name,
export_params=True,
opset_version=14,
do_constant_folding=True,
input_names=input_names,
output_names=output_names,
dynamic_axes=None,
)
print("Onnx model exporting done")
Export the onnx model ...
/tmp/ipykernel_2686/2593715709.py:490: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
bx = bxy[:, ii : ii + 1] + torch.tensor(
/tmp/ipykernel_2686/2593715709.py:494: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
by = bxy[:, ii + 1 : ii + 2] + torch.tensor(
Onnx model exporting done
6. Visualize & Inspect the ONNX model using Netron
Netron is an open-source viewer for deep neural networks (DNN) models. Below we use Netron to visualize the YOLOv4 model we recently converted to ONNX format:
import IPython
import netron
netron.start(onnx_file_name, 6006, browse=False)
IPython.display.IFrame(f"http://localhost:6006", width=1000, height=1000)
Serving 'yolov4_1_3_608_608_static.onnx' at http://localhost:6006
<iframe width="1000" height="1000" src="http://localhost:6006" frameborder="0" allowfullscreen
</iframe>
Notice in the ONNX graph that the end of the network is particularly dense with operators:
If we zoom in a bit, we can see that these operators are mostly basic math operators, such as Exp, Mul, Div, Add, and Sub, alongside non-computational tensor-editing operators, such as Slice, Concat, and Reshape.
The reason that the tail-end of this ONNX representation of the YOLO model looks so chaotic is because ONNX has no better way of representing the custom YoloLayer defined by the authors in PyTorch. This custom layer only exists in the YOLOv4 model architecture and therefore will likely never be adopted or standardized by the ONNX community. Further, the ability to convert this more traditional algorithm into an ONNX representation using more basic operators, speaks to the robustness of ONNX as a conversion tool.
The main point of this tutorial is that, while ONNX is a robust tool and can represent more traditional algorithms in its graph representation framework if asked to do so, it is inherently counterintuitive to the value added by ONNX.
Similarly, these complex graph substructures of basic math and non-computational tensor-editing operators are challenging for the Chimera Graph Compiler. Simply put, it involves trying to reverse engineer the traditional algorithm from the mangled ONNX representation. As the CGC matures, we hope to be able to accomodate more of these complex substructures; however, in the near term you are likely to experience issues compiling a model with the CGC that contain traditional algorithms represented as ONNX graphs.
Quadric recommends that if you experience issues compiling your model, check the ONNX graph for structures like these. They will typically be found at the end of the network graphs. If you ecounter this situation, we recommend that you try to identify the traditional algorithm in the original model that resulted in the complext graph, such as the custom YoloLayer of the original PyTorch model in this example, and attempt to write it as a custom operator when porting your model to the Chimera platform.
To learn more about the basics of writing a custom operator, refer to the Tutorial: Custom Op Insertion into a Model.
'%3e%3crect%20width='512'%20height='512'%20rx='60'%3e%3c/rect%3e%3cpath%20fill='%23fff'%20d='M355.904%20100H408.832L293.2%20232.16L429.232%20412H322.72L239.296%20302.928L143.84%20412H90.8805L214.56%20270.64L84.0645%20100H193.28L268.688%20199.696L355.904%20100ZM337.328%20380.32H366.656L177.344%20130.016H145.872L337.328%20380.32Z'%3e%3c/path%3e%3c/g%3e%3cdefs%3e%3cclipPath%20id='clip0_84_15698'%3e%3crect%20width='512'%20height='512'%20fill='%23fff'%3e%3c/rect%3e%3c/clipPath%3e%3c/defs%3e%3c/svg%3e)
