Multicore Demo

NOTE: The Jupyter Notebook below is included in the Chimera SDK and can be run interactively by running the following CLI command:

$ quadric sdk notebook

From the Jupyter Notebook window in your browser, select the notebook named /quadric/sdk-cli/examples/multicore/multicore.ipynb.

Multicore Execution

Abstract

We support to perform inference on multiple gpnpus, which we call multicore. The multicore features:

• Independent per-core system AXI interfaces – for maximum performance
• Intra-cluster L2 -> L2 activation & intermediate data sharing – for reduced system bus traffic
• Weight Coalescer – single weight fetch / replication when splitting input image across cores

This notebook experiments multicore performance by using Mask RCNN.

Preparation

!pip3 install -r ../requirements.txt -q
!mim install -r ../requirements_mim.txt -q

from pathlib import Path

from examples.models.zoo.zoo_utils import (
    get_mmdeploy_parameters,
    get_transforms_and_dataset,
    install_openmmlab_github,
)
from tvm.contrib.epu.chimera_job.chimera_job import ChimeraJob
from tvm.contrib.epu.chimera_job.hw_config import HWConfig
from sdk_cli.utils.datasets.QuadricCalibration import COCOLikeDataset
from sdk_cli.utils.models.mmdeploy import MMDeployVariant
from sdk_cli.utils.models.mmdet import MMDetModelVariant
from sdk_cli.visualizers.layouter import (
    DetectorsLayouter,
    VisualizeHandle,
    VisualizeObject,
    get_visualize_handle,
)
import examples.models.zoo.fix_registry
from examples.models.zoo.detectors_zoo.detectors_utils import (
    get_mmdet_parameters,
)

from multicore_helpers import (
    export_and_quantize_onnx,
    run_inference,
    print_conclusion,
)

Install MMDtection and MMDeploy

github_names = [
    ("mmdetection", "3.2.0", None),
    ("mmdeploy", "1.3.1", None),
]
symlink_list = [
    ("configs", "mmdetection"),
]
install_openmmlab_github(github_names, symlink_list)

Specify Model

In this notebook, we use Mask RCNN with ResNet50 backbone.

## Select target model.
MODEL_NAME = MMDetModelVariant.mask_rcnn_r50_fpn_800x1344

mmdet_parameters = get_mmdet_parameters(MODEL_NAME)
mmdeploy_variant = MMDeployVariant.mmdetection
mmdeploy_parameters = get_mmdeploy_parameters(mmdeploy_variant, mmdet_parameters)

Export and Quantize ONNX

quantized_onnx_model, second_onnx_file, transforms = export_and_quantize_onnx(
    MODEL_NAME, mmdet_parameters, mmdeploy_parameters
)

Compare Performance between singlecore and multicore

Now, we have a quantized onnx for further experiment.

We are going to perform inferences on both single gpnpu and multiple gpnpus to compare performance.

coco_root = COCOLikeDataset.keywords["directory"]

## Specify target images
all_image_paths = [
    coco_root / "15673749081_767a7fa63a_k.jpeg",
    coco_root / "19064748793_bb942deea1_k.jpeg",
]

## Draw detected segmentation.
visualize_handle = get_visualize_handle(
    mmdet_parameters.visualize_object, mmdet_parameters.visual_threshold
)

Inference on Multicore

## Compilation
hw_config = HWConfig(ocm_size="8MB", num_cores=2)
cgc_multicore = ChimeraJob(
    model_p=str(quantized_onnx_model.model_path),
    hw_config=hw_config,
)
cgc_multicore.compile(quiet=True)

## Run inference.
detections_per_inference_engine = run_inference(
    cgc_multicore, all_image_paths, transforms, second_onnx_file, mmdet_parameters
)

for i in range(len(all_image_paths)):
    layouter = DetectorsLayouter(visualize_handle, image=str(all_image_paths[i]))
    for inference_engine, all_outputs in detections_per_inference_engine.items():
        layouter.add_data(
            bboxes=all_outputs[i][0],
            masks=all_outputs[i][1],
            title=f"{MODEL_NAME}: {str(inference_engine).upper()}",
        )
    layouter.display()

Run Statistics for Multicore

print(cgc_multicore)
cgc_multicore.plot_run_statistics()

Inference on Singlecore

In the above multicore config, the total size of L2M (OCM) is 16MB because each GPNPU has 8MB L2M (OCM).

In this experiment, we use 16MB L2M (OCM) to make the same size as the total L2M (OCM) size of the multicore.

## Compilation
hw_config = HWConfig(ocm_size="16MB", num_cores=1)
cgc_singlecore = ChimeraJob(
    model_p=str(quantized_onnx_model.model_path),
    hw_config=hw_config,
)
cgc_singlecore.compile(quiet=True)

## Run inference.
detections_per_inference_engine = run_inference(
    cgc_singlecore, all_image_paths, transforms, second_onnx_file, mmdet_parameters
)

for i in range(len(all_image_paths)):
    layouter = DetectorsLayouter(visualize_handle, image=str(all_image_paths[i]))
    for inference_engine, all_outputs in detections_per_inference_engine.items():
        layouter.add_data(
            bboxes=all_outputs[i][0],
            masks=all_outputs[i][1],
            title=f"{MODEL_NAME}: {str(inference_engine).upper()}",
        )
    layouter.display()

Run Statistics for Singlecore

print(cgc_singlecore)
cgc_singlecore.plot_run_statistics()

Conclusion

Note: Please perform the following cell after the execution of all the above command cells.

print_conclusion(cgc_singlecore, cgc_multicore)

Citation

@article{mmdetection,
  title   = {{MMDetection}: Open MMLab Detection Toolbox and Benchmark},
  author  = {Chen, Kai and Wang, Jiaqi and Pang, Jiangmiao and Cao, Yuhang and
             Xiong, Yu and Li, Xiaoxiao and Sun, Shuyang and Feng, Wansen and
             Liu, Ziwei and Xu, Jiarui and Zhang, Zheng and Cheng, Dazhi and
             Zhu, Chenchen and Cheng, Tianheng and Zhao, Qijie and Li, Buyu and
             Lu, Xin and Zhu, Rui and Wu, Yue and Dai, Jifeng and Wang, Jingdong
             and Shi, Jianping and Ouyang, Wanli and Loy, Chen Change and Lin, Dahua},
  journal= {arXiv preprint arXiv:1906.07155},
  year={2019}
}

@misc{=mmdeploy,
    title={OpenMMLab's Model Deployment Toolbox.},
    author={MMDeploy Contributors},
    howpublished = {\url{https://github.com/open-mmlab/mmdeploy}},
    year={2021}
}