Model Demo: Classifiers Zoo

NOTE: The Jupyter Notebook below is included in the Chimera SDK and can be run interactively by running the following CLI command:

$ quadric sdk notebook

From the Jupyter Notebook window in your browser, select the notebook named /quadric/sdk-cli/examples/models/zoo/classifiers_zoo/classifiers_zoo.ipynb.

Classifiers Zoo

Abstract

Various classifiers can be experimented in this notebook.

CSPNet

CSPDarknet53 is a convolutional neural network and backbone for object detection that uses DarkNet-53. It employs a CSPNet strategy to partition the feature map of the base layer into two parts and then merges them through a cross-stage hierarchy. The use of a split and merge strategy allows for more gradient flow through the network. This CNN is used as the backbone for YOLOv4.

• Chien-Yao Wang, et al. CSPNet: A New Backbone that can Enhance Learning Capability of CNN

MnasNet

MnasNet is a type of convolutional neural network optimized for mobile devices that is discovered through mobile neural architecture search, which explicitly incorporates model latency into the main objective so that the search can identify a model that achieves a good trade-off between accuracy and latency. The main building block is an inverted residual block (from MobileNetV2)

• Mingxing Tan, et al. MnasNet: Platform-Aware Neural Architecture Search for Mobile

MobileNetV2

MobileNetV2 is a convolutional neural network architecture that seeks to perform well on mobile devices. It is based on an inverted residual structure where the residual connections are between the bottleneck layers. The intermediate expansion layer uses lightweight depthwise convolutions to filter features as a source of non-linearity. As a whole, the architecture of MobileNetV2 contains the initial fully convolution layer with 32 filters, followed by 19 residual bottleneck layers.

• Mark Sandler, et al. MobileNetV2: Inverted Residuals and Linear Bottlenecks

ResNet

Residual Networks, or ResNets, learn residual functions with reference to the layer inputs, instead of learning unreferenced functions. Instead of hoping each few stacked layers directly fit a desired underlying mapping, residual nets let these layers fit a residual mapping. They stack residual blocks ontop of each other to form network: e.g. a ResNet-50 has fifty layers using these blocks.

• Kaiming He, et al. Deep Residual Learning for Image Recognition

SelecSLS

The network architecture performs comparable to ResNet-50 while being 1.4-1.8x faster, particularly with larger image sizes. The network architecture has a much smaller memory footprint, and can be used as a drop in replacement for ResNet-50 in various tasks.

• Dushyant Mehta, et al. XNect: Real-time Multi-Person 3D Motion Capture with a Single RGB Camera

SSL ResNet / SWSL ResNet

In order to improve the performance of the model, SSL ResNet utilizes semi-supervised learning and SWSL ResNet utilizes semi-weakly supervised learning respectively.

• I. Zeki Yalniz, et al. Billion-scale semi-supervised learning for image classification

VGG

VGG is a classical convolutional neural network architecture. It was based on an analysis of how to increase the depth of such networks. The network utilises small 3 x 3 filters. Otherwise the network is characterized by its simplicity: the only other components being pooling layers and a fully connected layer.

• Karen Simonyan, Andrew Zisserman. Very Deep Convolutional Networks for Large-Scale Image Recognition

WideResNet

Deep residual networks were shown to be able to scale up to thousands of layers and still have improving performance. However, each fraction of a percent of improved accuracy costs nearly doubling the number of layers, and so training very deep residual networks has a problem of diminishing feature reuse, which makes these networks very slow to train. To tackle these problems, wide residual networks was proposed to decrease depth and increase width of residual networks, showing superior over their commonly used thin and very deep counterparts.

• Sergey Zagoruyko, Nikos Komodakis. Wide Residual Networks

BarlowTwins

Barlow Twins does not require large batches nor asymmetry between the network twins such as a predictor network, gradient stopping, or a moving average on the weight updates. Intriguingly it benefits from very high-dimensional output vectors.

• Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, Stéphane Deny. Barlow Twins: Self-Supervised Learning via Redundancy Reduction

BYOL

Bootstrap Your Own Latent (BYOL) is a new approach to self-supervised image representation learning. BYOL relies on two neural networks, referred to as online and target networks, that interact and learn from each other.

• Jean-Bastien Grill, etal. Bootstrap your own latent: A new approach to self-supervised Learning

DenseCL

DenseCL designs an effective, dense self-supervised learning method that directly works at the level of pixels (or local features) by taking into account the correspondence between local features.

• Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, Lei Li. Dense contrastive learning for self-supervised visual pre-training

MoCo

With simple modifications to Momentum Contrast (MoCo) — namely, using an MLP projection head and more data augmentation — stronger baselines that outperform SimCLR and do not require large training batches is established.

• Xinlei Chen, Haoqi Fan, Ross Girshick, Kaiming He. Improved Baselines with Momentum Contrastive Learning
• Xinlei Chen, Saining Xie, Kaiming He. An Empirical Study of Training Self-Supervised Vision Transformers

SimCLR

SimCLR: a simple framework for contrastive learning of visual representations proposes contrastive self-supervised learning algorithms without requiring specialized architectures or a memory bank.

• Ting Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey Hinton. A simple framework for contrastive learning of visual representations

SimSiam

Simple Siamese (SimSiam) can learn meaningful representations even using none of the following: (i) negative sample pairs, (ii) large batches, (iii) momentum encoders.

• Xinlei Chen, Kaiming He. Exploring simple siamese representation learning

SparK

Sparse masKed modeling (SparK) is general: it can be used directly on any convolutional model without backbone modifications.

• Keyu Tian, Yi Jiang, Qishuai Diao, Chen Lin, Liwei Wang, Zehuan Yuan. Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling

SwAV

SwAV takes advantage of contrastive methods without requiring to compute pairwise comparisons. Specifically, it simultaneously clusters the data while enforcing consistency between cluster assignments produced for different augmentations (or “views”) of the same image, instead of comparing features directly as in contrastive learning.

• Jiahao Wang, Songyang Zhang, Yong Liu, Taiqiang Wu, Yujiu Yang, Xihui Liu, Kai Chen, Ping Luo, Dahua Lin. Unsupervised Learning of Visual Features by Contrasting Cluster Assignments

RIFormer

RIFormer is a way to keep a vision backbone effective while removing token mixers in its basic building blocks. Equipped with the proposed optimization strategy, the authors claim that they are able to build an extremely simple vision backbone with encouraging performance, while enjoying the high efficiency during inference. RIFormer shares nearly the same macro and micro design as MetaFormer, but safely removing all token mixers.

• Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, Armand Joulin. RIFormer: Keep Your Vision Backbone Effective While Removing Token Mixer

EfficientNet

EfficientNets, a family of models, are obtained with neural architecture search to design a new baseline network and scale it up. They achieve much better accuracy and efficiency than previous ConvNets.

• Mingxing Tan, Quoc V. Le. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks

FBNet

DNAS (differentiable neural architecture search) framework uses gradient-based methods to optimize ConvNet architectures, avoiding enumerating and training individual architectures separately as in previous methods. FBNets, a family of models discovered by DNAS surpass state-of-the-art models both designed manually and generated automatically.

• Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, Kurt Keutzer. FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search

Res2Net

Res2Net is a novel building block for CNNs, by constructing hierarchical residual-like connections within one single residual block. The Res2Net represents multi-scale features at a granular level and increases the range of receptive fields for each network layer. The proposed Res2Net block can be plugged into the state-of-the-art backbone CNN models, e.g., ResNet, ResNeXt, and DLA.

• Shang-Hua Gao, Ming-Ming Cheng, Kai Zhao, Xin-Yu Zhang, Ming-Hsuan Yang, and Philip Torr. Res2Net: A New Multi-scale Backbone Architecture

ResNeXt

A ResNeXt repeats a building block that aggregates a set of transformations with the same topology. This strategy exposes a new dimension, which we call “cardinality” (the size of the set of transformations), as an essential factor in addition to the dimensions of depth and width.

• Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, Kaiming He. Aggregated Residual Transformations for Deep Neural Networks

SPNASNet

Neural architecture search (NAS) has revolutionized the design of hardware-efficient ConvNets by automating this process. However, the NAS problem remains challenging due to the combinatorially large design space, causing a significant searching time (at least 200 GPU-hours). To alleviate this complexity, Single-Path NAS which is a novel differentiable NAS method for designing hardware-efficient ConvNets in less than 4 hours is proposed.

• Dimitrios Stamoulis, Ruizhou Ding, Di Wang, Dimitrios Lymberopoulos, Bodhi Priyantha, Jie Liu, Diana Marculescu. Single-Path NAS: Designing Hardware-Efficient ConvNets in less than 4 Hours

(Gluon) ResNet / ResNeXt

The weights from teses modesl were ported from Gluon5)

Set-Up

Install Package / Import Modules

!pip3 install -r ../../../requirements.txt -q
!mim install -r ../../../requirements_mim.txt -q

import gc
import matplotlib.pyplot as plt
import os
from pathlib import Path
from PIL import Image
import random
import shutil
import zipfile

import numpy as np
import onnx
from onnxruntime import InferenceSession
import onnxsim
import timm
import torch
from torch.utils.data import Subset
import torchvision
from torchvision.transforms import Compose, Normalize, ToTensor, Resize, CenterCrop

from sdk_cli.lib.inference import InferenceEngine, batch_inference
from sdk_cli.lib.quantize import (
    QuantizedONNXModel,
    quantize_onnx_model,
)
from sdk_cli.utils.datasets import ImageNet_Mini_Quadric
from sdk_cli.utils.datasets.ImageNet import (
    IMAGENET_1K_NORMALIZATION_PARAMETERS,
    OPTIMIZED_IMAGENET_1K_LABELS,
)
from sdk_cli.utils.models.classifier import (
    ClassifierDeployVariant,
    ClassifierVariant,
)
from sdk_cli.utils.models.mmdeploy import MMDeployVariant
from sdk_cli.visualizers.layouter import ClassifierLayouter
from tvm.contrib.epu.chimera_job.chimera_job import ChimeraJob
import tvm.contrib.epu.chimera_job.constants as sdk_constants
from tvm.contrib.epu.onnx_util import cut_onnx

from examples.models.zoo.zoo_utils import (
    download_file,
    onnx_check_and_simplify,
    get_mmdeploy_parameters,
    get_transforms_and_dataset,
    install_openmmlab_github,
)
from examples.models.zoo.classifiers_zoo.classifiers_utils import (
    ClassifierParameters,
    get_classifier_parameters,
    get_classifier_model,
)

CUR_DIR = os.path.abspath(os.getcwd())

Disable progress bars for Hugging Face download to avoid display error.

from huggingface_hub.utils import disable_progress_bars

disable_progress_bars()

Install MMPretrain and MMDeploy

github_names = [
    ("mmpretrain", "1.2.0", None),
    ("mmdeploy", "1.3.1", None),
    ("mmengine", "0.10.7", "mmengine-main"),
]
symlink_list = [
    ("configs", "mmpretrain"),
    ("mmengine", "mmengine-main"),
]
install_openmmlab_github(github_names, symlink_list)

!sed -i "347s/map_location=map_location)/map_location=map_location, weights_only=False)/" mmengine/runner/checkpoint.py

import examples.models.zoo.fix_registry

os.chdir(Path(CUR_DIR) / "mmdeploy")

from mmdeploy.apis import torch2onnx
import torch

os.chdir(CUR_DIR)

Model Select

This notebook can experiment the following models. As a default, we are going to experiment cspdarknet53.

## Select target model.
MODEL_NAME = ClassifierVariant.cspdarknet53

classifier_parameters = get_classifier_parameters(MODEL_NAME)
model_input_size = classifier_parameters.model_input_size

ONNX Export

onnx_file = f"{MODEL_NAME}.onnx"
IMAGENET_DATA_DIR = ImageNet_Mini_Quadric.ROOT / "val"

if classifier_parameters.deployer == ClassifierDeployVariant.mmpretrain:
    mmdeploy_parameters = get_mmdeploy_parameters(MMDeployVariant.mmpretrain, classifier_parameters)
    model_input_size = mmdeploy_parameters.model_input_size
    weights = Path(mmdeploy_parameters.weights_url).name

    if not Path(weights).exists():
        download_file(mmdeploy_parameters.weights_url, weights)
        print(f"{weights} is downloaded.")

    image_file = IMAGENET_DATA_DIR / "n03777568" / "ILSVRC2012_val_00049034.JPEG"
    Image.open(image_file).resize(model_input_size).save("image.jpg")

    # Convert the model to ONNX
    torch2onnx(
        "image.jpg",
        work_dir=".",
        save_file=onnx_file,
        deploy_cfg=mmdeploy_parameters.deploy_config,
        model_cfg=mmdeploy_parameters.config,
        model_checkpoint=weights,
        device="cpu",
    )
else:
    pytorch_model = get_classifier_model(classifier_parameters)
    input_tensor = torch.randn(1, 3, *model_input_size[::-1], requires_grad=False)

    # Export the model
    torch.onnx.export(
        pytorch_model,  # model being run
        input_tensor,  # model input (or a tuple for multiple inputs)
        onnx_file,  # where to save the model (can be a file or file-like object)
        export_params=True,  # store the trained parameter weights inside the model file
        do_constant_folding=True,  # whether to execute constant folding for optimization
        opset_version=sdk_constants.DEFAULT_ONNX_OPSET,
    )
    del input_tensor, pytorch_model

onnx_model = onnx_check_and_simplify(
    onnx.load(onnx_file), classifier_parameters.skip_shape_inference
)
onnx.save(onnx_model, onnx_file)

print(f"ONNX is exported and simplified as {onnx_file}")

gc.collect();

Split ONNX

We split onnx files of mnasnet0_75 and mnasnet1_0 before ReduceMean.

target_onnx_file = onnx_file
head_onnx_file = None

if classifier_parameters.split_edges:
    onnx_model = onnx.load(onnx_file)
    session = InferenceSession(onnx_model.SerializeToString())
    input_edges = [inp.name for inp in session.get_inputs()]
    output_edges = [oup.name for oup in session.get_outputs()]

    backbone_onnx_file = f"{MODEL_NAME}-backbone.onnx"
    head_onnx_file = f"{MODEL_NAME}-head.onnx"

    # Extract the backbone.
    sub_graph = cut_onnx(onnx_model, input_edges, classifier_parameters.split_edges)
    sub_graph = onnx_check_and_simplify(sub_graph, classifier_parameters.skip_shape_inference)
    onnx.save(sub_graph, backbone_onnx_file)
    target_onnx_file = backbone_onnx_file
    print(f"Backbone onnx is saved as {backbone_onnx_file}")

    # Extract the head.
    sub_graph = cut_onnx(onnx_model, classifier_parameters.split_edges, output_edges)
    onnx.save(sub_graph, head_onnx_file)
    print(f"Head onnx is saved as {head_onnx_file}")

    del sub_graph, session, onnx_model
    gc.collect();

Quantization

if classifier_parameters.deployer == ClassifierDeployVariant.mmpretrain:
    transforms, subset_of_dataset = get_transforms_and_dataset(mmdeploy_parameters)
else:
    dataset_mean = IMAGENET_1K_NORMALIZATION_PARAMETERS.channel_means
    dataset_std = IMAGENET_1K_NORMALIZATION_PARAMETERS.channel_standard_deviations
    transforms = Compose(
        [
            Resize(256),
            CenterCrop(model_input_size[0]),
            ToTensor(),
            Normalize(dataset_mean, dataset_std),
        ]
    )

    imagenet_dataset = ImageNet_Mini_Quadric.Dataset(transform=transforms)
    subset_of_dataset = Subset(imagenet_dataset, range(100))

quantized_onnx_model: QuantizedONNXModel = quantize_onnx_model(
    target_onnx_file,
    subset_of_dataset,
    asymmetric_activation=True,
)
print(
    f"quantized onnx: {quantized_onnx_model.model_path}, tranges file: {quantized_onnx_model.tensor_ranges_path}"
)

gc.collect();

Compilation

cgc_job = ChimeraJob(
    model_p=str(quantized_onnx_model.model_path),
    trange_file=str(quantized_onnx_model.tensor_ranges_path),
)
cgc_job.compile(quiet=True)
print(cgc_job)

Demo

Inference

all_images_dict = {
    str(IMAGENET_DATA_DIR / "n03777568" / "ILSVRC2012_val_00049034.JPEG"): 661,
    str(IMAGENET_DATA_DIR / "n02877765" / "ILSVRC2012_val_00049979.JPEG"): 455,
    str(IMAGENET_DATA_DIR / "n02125311" / "ILSVRC2012_val_00015558.JPEG"): 286,
}
all_image_paths = list(all_images_dict.keys())
NUM_IMAGES = len(all_image_paths)

engines = {
    InferenceEngine.CHIMERA_ORT_INT8: cgc_job,
    InferenceEngine.CHIMERA_ISS_INT8: cgc_job,
}

head_session = (
    InferenceSession(onnx.load(head_onnx_file).SerializeToString()) if head_onnx_file else None
)

all_images = []
for image_path in all_image_paths:
    all_images.append(np.expand_dims(transforms(Image.open(image_path)), axis=0))

outputs_per_inference_engine = {}
THREADS = min(NUM_IMAGES, 6)
for inference_engine, engine in engines.items():
    outputs_per_inference_engine[inference_engine] = batch_inference(
        inference_engine,
        engine,
        all_images,
        head_session=head_session,
        threads=THREADS,
    )

Run Statistics

print(cgc_job)
cgc_job.plot_run_statistics()

Display Inference Results

%matplotlib inline

for i, image_path in enumerate(all_image_paths):
    title = f"{all_images_dict[image_path]}: {OPTIMIZED_IMAGENET_1K_LABELS.class_map[all_images_dict[image_path]]}"
    classifier_layouter = ClassifierLayouter(
        image_path, title, conversion=classifier_parameters.conversion
    )

    for inference_engine, all_outputs in outputs_per_inference_engine.items():
        classifier_layouter.add_data(
            all_outputs[i][0], f"{MODEL_NAME} Top 5 (%): {str(inference_engine)}"
        )
    classifier_layouter.display()

Citation

@software{torchvision2016,
    title        = {TorchVision: PyTorch's Computer Vision library},
    author       = {TorchVision maintainers and contributors},
    year         = 2016,
    journal      = {GitHub repository},
    publisher    = {GitHub},
    howpublished = {\url{https://github.com/pytorch/vision}}
}

@misc{rw2019timm,
  author = {Ross Wightman},
  title = {PyTorch Image Models},
  year = {2019},
  publisher = {GitHub},
  journal = {GitHub repository},
  doi = {10.5281/zenodo.4414861},
  howpublished = {\url{https://github.com/rwightman/pytorch-image-models}}
}

@misc{2023mmpretrain,
    title={OpenMMLab's Pre-training Toolbox and Benchmark},
    author={MMPreTrain Contributors},
    howpublished = {\url{https://github.com/open-mmlab/mmpretrain}},
    year={2023}
}

@misc{=mmdeploy,
    title={OpenMMLab's Model Deployment Toolbox.},
    author={MMDeploy Contributors},
    howpublished = {\url{https://github.com/open-mmlab/mmdeploy}},
    year={2021}
}