NOTE: The Jupyter Notebook below is included in the Chimera SDK and can be run interactively by running the following CLI command:
$ quadric sdk notebook
From the Jupyter Notebook window in your browser, select the notebook named /quadric/sdk-cli/examples/doc_template/sdk-cli.ipynb.
The Quadric Chimera Software Development Kit (SDK) is provided as an environment within a container that has a collection of libraries and tools installed for the development of complex application code targeting the Chimera General-Prupose Neural Processing Unit (GPNPU).
The Chimera SDK environment includes libraries and pre-installed tools, such as the Chimera Graph Compiler (CGC) and the Chimera Compute Library (CCL) APIs, to enable developers to compile deep neural network (DNN) graphs and write performant AI algorithms.
For convenience, Quadric has also created a simple Command Line Interface (CLI) for some of the tools included in the Chimera SDK. This SDK CLI is accessible via quadric sdk <subcommand>.
$ quadric sdk <SDK CLI Command Here>
Getting Started
If you're new to the Chimera SDK, we recommend getting started by running the --help command and familiarizing yourself with the commands.
!sdk --help
Usage: sdk [OPTIONS] COMMAND [ARGS]...
Run Neural Networks and C++ Kernels using the quadric sdk.
Run this command with --help for more information.
Options:
--help Show this message and exit.
Commands:
graph Validate, Profile, Compile and/or Execute quantized Neural...
notebook Start Jupyter Notebook.
source Run quadric C++ kernels using the quadric sdk.
[0m
In the rest of this document, we'll review a few of the most important CLI commands of which you should be aware.
Run a Quadric Tutorial: notebook
Some of the tutorials in this guide are also made available as interactive Python, i.e. Jupyter Notebook, files (*.ipynb). The notebook CLI command starts a Jupyter notebook environment that is tunneled to your web browser from inside the SDK container.
This command makes it easy to walk through these tutorials step-by-step and examine the results of intermediate commands without needing to install anything or set up a Python environment.
!sdk notebook --help
Usage: sdk notebook [OPTIONS]
Start Jupyter Notebook.
Options:
--help Show this message and exit.
[0m
Check if Model is Formatted Properly for Compilation: graph analyze
The graph analyze CLI command can be used to check if an ONNX model is formatted properly for compilation by the Chimera Graph Compiler (CGC). Quadric provides this functionality as an assurance that a model has been converted to a supported ONNX format and is quantized properly before attempting to lower with CGC.
To learn more about the steps you need to take prior to compiling your model, refer to the Preparing a Model for Compilation section of the Chimera Software Users's Guide.
!sdk graph analyze --help
Usage: sdk graph analyze [OPTIONS] QUANTIZED_MODEL
Analyze an ONNX graph and check that model is ready for compilation by CGC.
Options:
-v, --verbose Turn on debug mode (verbose logging)
--config FILE Path to a JSON configuration file
--target [QB16|QB4|QB1|QC-N|QC-P|QC-U]
Target architecture or device
--num-cores [1|2|4|8] Number of cores in multicore configuration
--num-clusters [1|2|4|8] Number of clusters in multicluster
configuration
--ocm-size [1MB|2MB|4MB|8MB|16MB|32MB]
On Chip Memory size. (MB, Megabytes).
--lrm-size [4kB|2kB] Local register memory size. (kB, Kilobytes).
--macs-per-pe [8|16] Number of MACs per PE
--clock-freq-ghz FLOAT RANGE Clock frequency of iss in GHz [0.9<=x<=2]
--ext-read-bw BANDWIDTH The read bandwidth from the remote device to
the GPNPU. ie 128GBps (Unit: GBps, Gigabytes
per second).
--ext-write-bw BANDWIDTH The write bandwidth from the remote device
to the GPNPU. ie 128GBps (Unit: GBps,
Gigabytes per second).
--mean [<float>, <float>, <float>]
Mean of the input normalization that the
model was trained with. [default: [0, 0,
0]]
--std [<float>, <float>, <float>]
Standard Deviation of the input
normalization that the model was trained
with. [default: [1, 1, 1]]
--custom-operations FILENAME Custom operations written in quadric C++ to
embed in graph.
--image FILENAME An image that can be used as input to the
network. If not set, random noise will be
used.
--help Show this message and exit.
[0m
Naive Quantization: graph quantize
Quantization in the context of DL, is the process of approximating a DNN that uses floating-point numbers (typically 32 or 64-bit) for inference with a DNN that uses lower-precision numbers.
Naive quantization is a quantization strategy in which all of a DNN's operators are quantized to 8-bit integer, i.e. INT8, precision and are calibrated using the same method. Naive quantization is the simplest quantization methodology to implement, but often results in a dramatic drop in model accuracy compared to the original floating-point model. This is because the same quantization method is applied to all operators, regardless of their sensitivity to quantization.
The graph quantize CLI command can be used to perform naive quantization. Quadric provides this functionality to make it easier for users to quickly quantize their model and benchmark performance on the Chimera processor architecture.
If you choose to use this built-in naive quantization, please be aware that your model's accuracy will likely drop significantly and that this is not indicative of the potential accuracy you could achieve if using a more robust quantization methodology.
To learn more about quantization and specifically naiveqQuantization, refer to Accepted Model Formats - Quantized ONNX.
!sdk graph quantize --help
Usage: sdk graph quantize [OPTIONS] FLOATING_POINT_MODEL
Quantize a floating point model in ONNX format to int8 using onnxruntime.
Options:
--synthetic-input Generate synthetic data at the inputs. This
is useful to quickly quantize a graph.
Yields correct performance estimates after
lowering the result in CGC. Will yield valid
outputs against any validation images.
--calibration-folder PATH Folder containing .jpeg or .png images
--num-images INTEGER Total maximum number of images to use
--exclude-nodes-by-type <str>,<str>,<str>,...
Additional ONNX nodes to exclude from
quantization. Example: "Add,Mul,Relu6"
--exclude-nodes-by-name <str>,<str>,<str>,...
Specific ONNX nodes names to exclude from
quantization. Example: "854,867"
--force-nodes-by-type <str>,<str>,<str>,...
Specific ONNX nodes types to force
quantization. Example: "Softmax,Mul"
--asymmetric-activation Quantize activations with asymmetric
quantization. Default is Symmetric
quantization. NOTE: Experimental will result
in better performance. Use at your own risk.
--mean [<float>, <float>, <float>]
Mean of the input normalization that the
model was trained with. [default: [0, 0,
0]]
--std [<float>, <float>, <float>]
Standard Deviation of the input
normalization that the model was trained
with. [default: [1, 1, 1]]
--help Show this message and exit.
[0m
Compile a DNN with the Chimera Graph Compiler (CGC): graph compile
The graph compile CLI command is an easy way to invoke the Chimera Graph Compiler (CGC).
The CGC is a powerful conversion and code optimization tool that accepts a quantized Deep Neural Network (DNN) model as input, performs optimizations, and outputs an optimized C++ code representation of the graph utilizing the Chimera Compute Library (CCL) APIs.
If the command is invoked with the --run parameter, output performance metrics for the compiled model will be generated based on the target Chimera Processor architecture, LRM memory, and read/write bandwidth.
If your model contains custom operators or graph structures that are not natively supported by the CGC, you may also implement them yourselves and provide them to the CGC at compile-time by using the --custom-operations parameter.
!sdk graph compile --help
Usage: sdk graph compile [OPTIONS] QUANTIZED_MODEL
Compile an ONNX graph using CGC.
Options:
-v, --verbose Turn on debug mode (verbose logging)
--config FILE Path to a JSON configuration file
--target [QB16|QB4|QB1|QC-N|QC-P|QC-U]
Target architecture or device
--num-cores [1|2|4|8] Number of cores in multicore configuration
--num-clusters [1|2|4|8] Number of clusters in multicluster
configuration
--ocm-size [1MB|2MB|4MB|8MB|16MB|32MB]
On Chip Memory size. (MB, Megabytes).
--lrm-size [4kB|2kB] Local register memory size. (kB, Kilobytes).
--macs-per-pe [8|16] Number of MACs per PE
--clock-freq-ghz FLOAT RANGE Clock frequency of iss in GHz [0.9<=x<=2]
--ext-read-bw BANDWIDTH The read bandwidth from the remote device to
the GPNPU. ie 128GBps (Unit: GBps, Gigabytes
per second).
--ext-write-bw BANDWIDTH The write bandwidth from the remote device
to the GPNPU. ie 128GBps (Unit: GBps,
Gigabytes per second).
--custom-operations FILENAME Custom operations written in quadric C++ to
embed in graph.
--run Run the network after compilation and obtain
performance information.
--validate-iss Validate the iss network run against ort.
Prints a table containing diverging regions
--trange-file FILENAME
--studio-upload Upon compile success, the user will be
prompted to provide upload information for
Dev Studio Upload.
--package-kernel Package a kernel using parameters.json.
Required for executing RTL with cbench. Must
be used with --run.
--help Show this message and exit.
[0m
Compile a C++ Kernel with the Chimera LLVM C++ Compiler: source
The source CLI command is a simple way to invoke the Chimera LLVM C++ Compiler.
Use this command to compile a C++ kernel and output performance metrics for the kernel based on the target Chimera Processor architecture, LRM memory, and read/write bandwidth.
!sdk source --help
Usage: sdk source [OPTIONS] SOURCE_FILES...
Run quadric C++ kernels using the quadric sdk.
This comamnd is able to take in one or two CPP files, given the first file
contains an EPU Kernel, and the second a HOST only executable.
ex: $ sdk source epu_kernel.cpp host_tensor_setup.cpp
In the instance that a single CPP file is used, the CPP file must contain
both the EPU kernel and the HOST related tensor setup.
ex: $ sdk source myAllInOneKernel.cpp
Options:
--check-results Check results against expected tensor
generated in kernel.
--include-cgc-headers Include CGC headers in the source
compilation.
--ddr-axi-width INTEGER RANGE DDR AXI width in bits (default: 128)
[64<=x<=512]
--quiet Suppress stdout/stderr (including warnings)
and write to compile_log.txt and
iss_run_log.txt.
-v, --verbose Turn on debug mode (verbose logging)
--config FILE Path to a JSON configuration file
--target [QB16|QB4|QB1|QC-N|QC-P|QC-U]
Target architecture or device
--num-cores [1|2|4|8] Number of cores in multicore configuration
--num-clusters [1|2|4|8] Number of clusters in multicluster
configuration
--ocm-size [1MB|2MB|4MB|8MB|16MB|32MB]
On Chip Memory size. (MB, Megabytes).
--lrm-size [4kB|2kB] Local register memory size. (kB, Kilobytes).
--macs-per-pe [8|16] Number of MACs per PE
--clock-freq-ghz FLOAT RANGE Clock frequency of iss in GHz [0.9<=x<=2]
--ext-read-bw BANDWIDTH The read bandwidth from the remote device to
the GPNPU. ie 128GBps (Unit: GBps, Gigabytes
per second).
--ext-write-bw BANDWIDTH The write bandwidth from the remote device
to the GPNPU. ie 128GBps (Unit: GBps,
Gigabytes per second).
--help Show this message and exit.
[0m
Enter the SDK Container in Interactive Mode
In some situations, there may not be a CLI command for your use-case. If you'd like to enter the Chimera SDK Container in an interactive shell, run the following command:
$ quadric sdk interactive
