Unlock this feature

This feature isn’t part of your plan yet

Contact sales to get upgraded to the full DevStudio experience.

Unlock this feature

This feature isn't part of your plan yet.

Model Demo: Classifiers Zoo


NOTE: The Jupyter Notebook below is included in the Chimera SDK and can be run interactively by running the following CLI command:

$ quadric sdk notebook

From the Jupyter Notebook window in your browser, select the notebook named /quadric/sdk-cli/examples/models/zoo/classifiers_zoo/classifiers_zoo.ipynb.


Classifiers Zoo

License Notice

The images this notebook uses as input data, for calibration, evaluation or training, come from the ImageNet database, whose terms of access state: "Researcher shall use the Database only for non-commercial research and educational purposes." https://www.image-net.org/download.php

Abstract

Various classifiers can be experimented in this notebook.

CSPNet

CSPDarknet53 is a convolutional neural network and backbone for object detection that uses DarkNet-53. It employs a CSPNet strategy to partition the feature map of the base layer into two parts and then merges them through a cross-stage hierarchy. The use of a split and merge strategy allows for more gradient flow through the network. This CNN is used as the backbone for YOLOv4.

MnasNet

MnasNet is a type of convolutional neural network optimized for mobile devices that is discovered through mobile neural architecture search, which explicitly incorporates model latency into the main objective so that the search can identify a model that achieves a good trade-off between accuracy and latency. The main building block is an inverted residual block (from MobileNetV2)

MobileNetV2

MobileNetV2 is a convolutional neural network architecture that seeks to perform well on mobile devices. It is based on an inverted residual structure where the residual connections are between the bottleneck layers. The intermediate expansion layer uses lightweight depthwise convolutions to filter features as a source of non-linearity. As a whole, the architecture of MobileNetV2 contains the initial fully convolution layer with 32 filters, followed by 19 residual bottleneck layers.

ResNet

Residual Networks, or ResNets, learn residual functions with reference to the layer inputs, instead of learning unreferenced functions. Instead of hoping each few stacked layers directly fit a desired underlying mapping, residual nets let these layers fit a residual mapping. They stack residual blocks ontop of each other to form network: e.g. a ResNet-50 has fifty layers using these blocks.

SelecSLS

The network architecture performs comparable to ResNet-50 while being 1.4-1.8x faster, particularly with larger image sizes. The network architecture has a much smaller memory footprint, and can be used as a drop in replacement for ResNet-50 in various tasks.

VGG

VGG is a classical convolutional neural network architecture. It was based on an analysis of how to increase the depth of such networks. The network utilises small 3 x 3 filters. Otherwise the network is characterized by its simplicity: the only other components being pooling layers and a fully connected layer.

WideResNet

Deep residual networks were shown to be able to scale up to thousands of layers and still have improving performance. However, each fraction of a percent of improved accuracy costs nearly doubling the number of layers, and so training very deep residual networks has a problem of diminishing feature reuse, which makes these networks very slow to train. To tackle these problems, wide residual networks was proposed to decrease depth and increase width of residual networks, showing superior over their commonly used thin and very deep counterparts.

BarlowTwins

Barlow Twins does not require large batches nor asymmetry between the network twins such as a predictor network, gradient stopping, or a moving average on the weight updates. Intriguingly it benefits from very high-dimensional output vectors.

BYOL

Bootstrap Your Own Latent (BYOL) is a new approach to self-supervised image representation learning. BYOL relies on two neural networks, referred to as online and target networks, that interact and learn from each other.

DenseCL

DenseCL designs an effective, dense self-supervised learning method that directly works at the level of pixels (or local features) by taking into account the correspondence between local features.

MoCo

With simple modifications to Momentum Contrast (MoCo) — namely, using an MLP projection head and more data augmentation — stronger baselines that outperform SimCLR and do not require large training batches is established.

SimCLR

SimCLR: a simple framework for contrastive learning of visual representations proposes contrastive self-supervised learning algorithms without requiring specialized architectures or a memory bank.

SimSiam

Simple Siamese (SimSiam) can learn meaningful representations even using none of the following: (i) negative sample pairs, (ii) large batches, (iii) momentum encoders.

SparK

Sparse masKed modeling (SparK) is general: it can be used directly on any convolutional model without backbone modifications.

SwAV

SwAV takes advantage of contrastive methods without requiring to compute pairwise comparisons. Specifically, it simultaneously clusters the data while enforcing consistency between cluster assignments produced for different augmentations (or “views”) of the same image, instead of comparing features directly as in contrastive learning.

RIFormer

RIFormer is a way to keep a vision backbone effective while removing token mixers in its basic building blocks. Equipped with the proposed optimization strategy, the authors claim that they are able to build an extremely simple vision backbone with encouraging performance, while enjoying the high efficiency during inference. RIFormer shares nearly the same macro and micro design as MetaFormer, but safely removing all token mixers.

EfficientNet

EfficientNets, a family of models, are obtained with neural architecture search to design a new baseline network and scale it up. They achieve much better accuracy and efficiency than previous ConvNets.

FBNet

DNAS (differentiable neural architecture search) framework uses gradient-based methods to optimize ConvNet architectures, avoiding enumerating and training individual architectures separately as in previous methods. FBNets, a family of models discovered by DNAS surpass state-of-the-art models both designed manually and generated automatically.

ResNeXt

A ResNeXt repeats a building block that aggregates a set of transformations with the same topology. This strategy exposes a new dimension, which we call “cardinality” (the size of the set of transformations), as an essential factor in addition to the dimensions of depth and width.

SPNASNet

Neural architecture search (NAS) has revolutionized the design of hardware-efficient ConvNets by automating this process. However, the NAS problem remains challenging due to the combinatorially large design space, causing a significant searching time (at least 200 GPU-hours). To alleviate this complexity, Single-Path NAS which is a novel differentiable NAS method for designing hardware-efficient ConvNets in less than 4 hours is proposed.

(Gluon) ResNet / ResNeXt

The weights from teses modesl were ported from Gluon5)

Set-Up

Install Package / Import Modules

!pip3 install -r ../../../requirements.txt -q
!mim install -r ../../../requirements_mim.txt -q
import gc
import matplotlib.pyplot as plt
import os
from pathlib import Path
from PIL import Image
import random
import shutil
import zipfile

import numpy as np
import onnx
from onnxruntime import InferenceSession
import onnxsim
import timm
import torch
from torch.utils.data import Subset
import torchvision
from torchvision.transforms import Compose, Normalize, ToTensor, Resize, CenterCrop

from sdk_cli.lib.inference import InferenceEngine, batch_inference
from sdk_cli.lib.quantize import (
    QuantizedONNXModel,
    quantize_onnx_model,
)
from sdk_cli.utils.datasets import ImageNet_Mini_Quadric
from sdk_cli.utils.datasets.ImageNet import (
    IMAGENET_1K_NORMALIZATION_PARAMETERS,
    OPTIMIZED_IMAGENET_1K_LABELS,
)
from sdk_cli.utils.models.classifier import (
    ClassifierDeployVariant,
    ClassifierVariant,
)
from sdk_cli.utils.models.mmdeploy import MMDeployVariant
from sdk_cli.visualizers.layouter import ClassifierLayouter
from tvm.contrib.epu.chimera_job.chimera_job import ChimeraJob
import tvm.contrib.epu.chimera_job.constants as sdk_constants
from tvm.contrib.epu.onnx_util import cut_onnx

from examples.models.zoo.zoo_utils import (
    download_file,
    onnx_check_and_simplify,
    get_mmdeploy_parameters,
    get_transforms_and_dataset,
    install_openmmlab_github,
)
from examples.models.zoo.classifiers_zoo.classifiers_utils import (
    ClassifierParameters,
    get_classifier_parameters,
    get_classifier_model,
)

CUR_DIR = os.path.abspath(os.getcwd())

Disable progress bars for Hugging Face download to avoid display error.

from huggingface_hub.utils import disable_progress_bars

disable_progress_bars()

Install MMPretrain and MMDeploy

github_names = [
    ("mmpretrain", "1.2.0", None),
    ("mmdeploy", "1.3.1", None),
    ("mmengine", "0.10.7", "mmengine-main"),
]
symlink_list = [
    ("configs", "mmpretrain"),
    ("mmengine", "mmengine-main"),
]
install_openmmlab_github(github_names, symlink_list)
!sed -i "347s/map_location=map_location)/map_location=map_location, weights_only=False)/" mmengine/runner/checkpoint.py
import examples.models.zoo.fix_registry

os.chdir(Path(CUR_DIR) / "mmdeploy")

from mmdeploy.apis import torch2onnx
import torch

os.chdir(CUR_DIR)

Model Select

This notebook can experiment the following models. As a default, we are going to experiment cspdarknet53.

model namemodel namemodel namemodel namemodel name
barlowtwinsgluon_resnext101resnet18_linearclsheadriformer_s24_384x384vgg13
byolmnasnet0_75resnet26selecsls42bvgg13_bn
cs3darknet_focus_lmnasnet1_0resnet32tsselecsls60vgg13_clshead
cs3darknet_focus_mmnasnet_100resnet33tsselecsls60bvgg13bn_clshead
cs3darknet_lmobilenet_v2resnet34simclrvgg16
cs3darknet_mmobilenetv2_110dresnet34_linearclsheadsimsiamvgg16_bn
cspdarknet50_256x256mobilenetv2_140resnet50sparkvgg16_clshead
cspdarknet53mocov2resnet50_gnspnasnet_100vgg16bn_clshead
darknet53mocov3resnet50_linearclsheadswavvgg19
denseclresnet101resnet50ctf_efficientnet_lite0vgg19_bn
efficientnet_elresnet101_linearclsheadresnet50stf_efficientnet_lite1vgg19_clshead
efficientnet_el_prunedresnet101cresnetblur50tf_efficientnet_lite2vgg19bn_clshead
efficientnet_emresnet101sresnext101_32x8dtf_efficientnet_lite3wide_resnet101
efficientnet_esresnet152resnext101_64x4dtf_efficientnet_lite4wide_resnet50
efficientnet_es_prunedresnet152_linearclsheadresnext26tsvgg11
efficientnet_lite0resnet152cresnext50_32x4dvgg11_bn
fbnetc_100resnet152sriformer_s12vgg11_clshead
gluon_resnet101resnet18riformer_s24vgg11bn_clshead
## Select target model.
MODEL_NAME = ClassifierVariant.cspdarknet53

classifier_parameters = get_classifier_parameters(MODEL_NAME)
model_input_size = classifier_parameters.model_input_size

ONNX Export

onnx_file = f"{MODEL_NAME}.onnx"
IMAGENET_DATA_DIR = ImageNet_Mini_Quadric.ROOT / "val"

if classifier_parameters.deployer == ClassifierDeployVariant.mmpretrain:
    mmdeploy_parameters = get_mmdeploy_parameters(MMDeployVariant.mmpretrain, classifier_parameters)
    model_input_size = mmdeploy_parameters.model_input_size
    weights = Path(mmdeploy_parameters.weights_url).name

    if not Path(weights).exists():
        download_file(mmdeploy_parameters.weights_url, weights)
        print(f"{weights} is downloaded.")

    image_file = IMAGENET_DATA_DIR / "n03777568" / "ILSVRC2012_val_00049034.JPEG"
    Image.open(image_file).resize(model_input_size).save("image.jpg")

    # Convert the model to ONNX
    torch2onnx(
        "image.jpg",
        work_dir=".",
        save_file=onnx_file,
        deploy_cfg=mmdeploy_parameters.deploy_config,
        model_cfg=mmdeploy_parameters.config,
        model_checkpoint=weights,
        device="cpu",
    )
else:
    pytorch_model = get_classifier_model(classifier_parameters)
    input_tensor = torch.randn(1, 3, *model_input_size[::-1], requires_grad=False)

    # Export the model
    torch.onnx.export(
        pytorch_model,  # model being run
        input_tensor,  # model input (or a tuple for multiple inputs)
        onnx_file,  # where to save the model (can be a file or file-like object)
        export_params=True,  # store the trained parameter weights inside the model file
        do_constant_folding=True,  # whether to execute constant folding for optimization
        opset_version=sdk_constants.DEFAULT_ONNX_OPSET,
    )
    del input_tensor, pytorch_model

onnx_model = onnx_check_and_simplify(
    onnx.load(onnx_file), classifier_parameters.skip_shape_inference
)
onnx.save(onnx_model, onnx_file)

print(f"ONNX is exported and simplified as {onnx_file}")

gc.collect();

Split ONNX

We split onnx files of mnasnet0_75 and mnasnet1_0 before ReduceMean.

target_onnx_file = onnx_file
head_onnx_file = None

if classifier_parameters.split_edges:
    onnx_model = onnx.load(onnx_file)
    session = InferenceSession(onnx_model.SerializeToString())
    input_edges = [inp.name for inp in session.get_inputs()]
    output_edges = [oup.name for oup in session.get_outputs()]

    backbone_onnx_file = f"{MODEL_NAME}-backbone.onnx"
    head_onnx_file = f"{MODEL_NAME}-head.onnx"

    # Extract the backbone.
    sub_graph = cut_onnx(onnx_model, input_edges, classifier_parameters.split_edges)
    sub_graph = onnx_check_and_simplify(sub_graph, classifier_parameters.skip_shape_inference)
    onnx.save(sub_graph, backbone_onnx_file)
    target_onnx_file = backbone_onnx_file
    print(f"Backbone onnx is saved as {backbone_onnx_file}")

    # Extract the head.
    sub_graph = cut_onnx(onnx_model, classifier_parameters.split_edges, output_edges)
    onnx.save(sub_graph, head_onnx_file)
    print(f"Head onnx is saved as {head_onnx_file}")

    del sub_graph, session, onnx_model
    gc.collect();

Quantization

if classifier_parameters.deployer == ClassifierDeployVariant.mmpretrain:
    transforms, subset_of_dataset = get_transforms_and_dataset(mmdeploy_parameters)
else:
    dataset_mean = IMAGENET_1K_NORMALIZATION_PARAMETERS.channel_means
    dataset_std = IMAGENET_1K_NORMALIZATION_PARAMETERS.channel_standard_deviations
    transforms = Compose(
        [
            Resize(256),
            CenterCrop(model_input_size[0]),
            ToTensor(),
            Normalize(dataset_mean, dataset_std),
        ]
    )

    imagenet_dataset = ImageNet_Mini_Quadric.Dataset(transform=transforms)
    subset_of_dataset = Subset(imagenet_dataset, range(100))

quantized_onnx_model: QuantizedONNXModel = quantize_onnx_model(
    target_onnx_file,
    subset_of_dataset,
    asymmetric_activation=True,
)
print(
    f"quantized onnx: {quantized_onnx_model.model_path}, tranges file: {quantized_onnx_model.tensor_ranges_path}"
)

gc.collect();

Compilation

cgc_job = ChimeraJob(
    model_p=str(quantized_onnx_model.model_path),
    trange_file=str(quantized_onnx_model.tensor_ranges_path),
)
cgc_job.compile(quiet=True)
print(cgc_job)

Demo

Inference

all_images_dict = {
    str(IMAGENET_DATA_DIR / "n03777568" / "ILSVRC2012_val_00049034.JPEG"): 661,
    str(IMAGENET_DATA_DIR / "n02877765" / "ILSVRC2012_val_00049979.JPEG"): 455,
    str(IMAGENET_DATA_DIR / "n02125311" / "ILSVRC2012_val_00015558.JPEG"): 286,
}
all_image_paths = list(all_images_dict.keys())
NUM_IMAGES = len(all_image_paths)

engines = {
    InferenceEngine.CHIMERA_ORT_INT8: cgc_job,
    InferenceEngine.CHIMERA_ISS_INT8: cgc_job,
}

head_session = (
    InferenceSession(onnx.load(head_onnx_file).SerializeToString()) if head_onnx_file else None
)

all_images = []
for image_path in all_image_paths:
    all_images.append(np.expand_dims(transforms(Image.open(image_path)), axis=0))

outputs_per_inference_engine = {}
THREADS = min(NUM_IMAGES, 6)
for inference_engine, engine in engines.items():
    outputs_per_inference_engine[inference_engine] = batch_inference(
        inference_engine,
        engine,
        all_images,
        head_session=head_session,
        threads=THREADS,
    )

Run Statistics

print(cgc_job)
cgc_job.plot_run_statistics()

Display Inference Results

%matplotlib inline

for i, image_path in enumerate(all_image_paths):
    title = f"{all_images_dict[image_path]}: {OPTIMIZED_IMAGENET_1K_LABELS.class_map[all_images_dict[image_path]]}"
    classifier_layouter = ClassifierLayouter(
        image_path, title, conversion=classifier_parameters.conversion
    )

    for inference_engine, all_outputs in outputs_per_inference_engine.items():
        classifier_layouter.add_data(
            all_outputs[i][0], f"{MODEL_NAME} Top 5 (%): {str(inference_engine)}"
        )
    classifier_layouter.display()

Citation

@software{torchvision2016,
    title        = {TorchVision: PyTorch's Computer Vision library},
    author       = {TorchVision maintainers and contributors},
    year         = 2016,
    journal      = {GitHub repository},
    publisher    = {GitHub},
    howpublished = {\url{https://github.com/pytorch/vision}}
}

@misc{rw2019timm,
  author = {Ross Wightman},
  title = {PyTorch Image Models},
  year = {2019},
  publisher = {GitHub},
  journal = {GitHub repository},
  doi = {10.5281/zenodo.4414861},
  howpublished = {\url{https://github.com/rwightman/pytorch-image-models}}
}

@misc{2023mmpretrain,
    title={OpenMMLab's Pre-training Toolbox and Benchmark},
    author={MMPreTrain Contributors},
    howpublished = {\url{https://github.com/open-mmlab/mmpretrain}},
    year={2023}
}

@misc{=mmdeploy,
    title={OpenMMLab's Model Deployment Toolbox.},
    author={MMDeploy Contributors},
    howpublished = {\url{https://github.com/open-mmlab/mmdeploy}},
    year={2021}
}

Sign in to your account

Don't have an account? 
By signing in, you are agreeing to our Terms of Use and Privacy Policy.
Quadric // One architecture. Every algorithm.

Develop.

Simulate.

Profile.

Collaborate.