ChimeraJob API Doc

chimera_job

Class: ChimeraJob

Store all the information needed to run a Chimera Graph Compiler (CGC) job.

This includes the model path, the chimera profile and the result.

Methods Overview

• __init__(model_p, hw_config, product, ocm_size, lrm_size, ext_rd_bw, ext_wr_bw, macs_per_pe, clock_freq_ghz, num_cores, hw_version, custom_ops, quiet_iss, ccl_op_nodes, verbose, onnx_ort_override_p, disable_prefetch, validate_iss, trange_file, package_kernel, region_max_cycles, heartbeat_period, io_to_ignore, target_lang, bypass_quantization_checks, batch_size, attn_stub_src_path, directives) - Construct a ChimeraJob object.
• from_hw_config(cls, model_p, hw_config) - Create a new ChimeraJob instance from a hardware configuration object.
• __str__() - Generate a human-readable string summarizing compile and run statistics.
• has_custom_op() - Check if the model has any nodes with "QuadricCustomOp" in their name.
• check_quantized_nodes(graph, quantized_keywords) - Recursively traverse an ONNX graph and checks for quantized nodes. This function also checks for subgraph of custom quadric ops and checks for quantized nodes.
• ingest_onnx() - Load the ONNX model and perm pre-compilation checks.
• analyze_network(print_ccl) - Analyze a network and print out the custom op stubs.
• generate_fully_quantized_relay() - Generate fully quantized relay intermediate representation (IR).
• compile(upload_on_success, quiet) - Compile the ONNX model for the target hardware configuration.
• convert_to_fxpt_onnx(fxpt_model_path) - Convert an ONNX model to a fixed-point quantized version using CGC under the hood to convert onnx to relay and back to onnx.
• run_onnx_inf_session(inputs, enable_gpnpu_emulation) - Run an ONNX Runtime inference session on the provided inputs.
• run_relay_inf_session(inputs) - Run inference on a quantized Relay graph.
• run_inference_harness(inputs, compare_ort, quiet, additional_archsim_options) - Run a single inference harness on the specified inputs.
• run_batch_inference_harness(inputs, threads) - Run a batch inference harness on ISS backend using a specified number of threads.
• run_batch_ort_harness(inputs, threads) - Run a batch inference harness on ort backend using a specified number of threads.
• parse_profile_results() - Process profiling results and generate performance statistics.
• plot_run_statistics(per_core, close) - Plot and save run statistics, returning the data path.
• validate_ort_iss(inputs, quiet) - Validate ONNX Runtime outputs against ISS (Instruction Set Simulator) results.

Methods Details

__init__(model_p: str, hw_config: HWConfig, product: str, ocm_size: str, lrm_size: str, ext_rd_bw: str, ext_wr_bw: str, macs_per_pe: int, clock_freq_ghz: float, num_cores: int, hw_version: str, custom_ops: str, quiet_iss: bool, ccl_op_nodes: list, verbose: bool, onnx_ort_override_p: str, disable_prefetch: bool, validate_iss: bool, trange_file: str, package_kernel: bool, region_max_cycles: int, heartbeat_period: int, io_to_ignore: list, target_lang: str, bypass_quantization_checks, batch_size: int, attn_stub_src_path: str, directives: str, **kwargs)

Construct a ChimeraJob object.

Parameters:

• model_p : str Path to the model
• hw_config : HWConfig A HWConfig object that defines the hardware configuration. If provided, this will override individual hardware parameters.
• product : str Product name (deprecated, use hw_config instead)
• ocm_size : str OCM (L2M) size in MB (deprecated, use hw_config instead)
• lrm_size : str LRM size in kB (deprecated, use hw_config instead)
• ext_rd_bw : str External read bandwidth in GBps (deprecated, use hw_config instead)
• ext_wr_bw : str External write bandwidth in GBps (deprecated, use hw_config instead)
• hw_version : str HW version string. Ex: "0x0C010000" (deprecated, use hw_config instead)
• macs_per_pe : int Number of int8 MACs per PE (16, 8) (deprecated, use hw_config instead)
• clock_freq_ghz : float Clock speed of iss (default 1.7GHz) (deprecated, use hw_config instead)
• num_cores : int Total number of cores in GPNPU complex (default 1) (deprecated, use hw_config instead)
• custom_ops : str Path to the custom ops
• quiet_iss : bool Quiet ISS
• ccl_op_nodes : list CCL operator nodes
• verbose : bool Set to True to increase the verbosity of the CGC job. Default is False.
• onnx_ort_override_p : str ONNX model that will be overridden for ONNX runtime execution. Default is "".
• disable_prefetch : bool Set to True to disable predictive weight prefetching in CGC. Default is False.
• validate_iss : bool Validate compiled model with ISS after compilation. Default is True.
• trange_file : str File where trange data from onnxruntime quantization run can be provided to CGC for fixed point range hinting.
• package_kernel : bool Package a kernel using parameters.json. Required for executing RTL using cbench. Default is False
• region_max_cycles : int Maximum number of cycles for a FILM region Default is DEFAULT_MAX_REGION_CYCLES
• heartbeat_period : int Default is DEFAULT_HEARTBEAT_PERIOD
• io_to_ignore : list ONNX inputs and outputs to ignore when compiling
• target_lang : str The target language to lower to. Can either be QIL or ASM. Default is ASM
• batch_size : int Batch size for batch parallelism. When batch_size > 1, it must equal num_cores. Each core processes one batch instance in parallel. Default is 1
• attn_stub_src_path : str Path to implemented general attention helper functions Default is ""
• directives : str Path to a JSON file using directive_builder.py that contains a list of directives for ONNX Ops. Default is ""

Examples:

Basic instantiation:
>>> c_job = ChimeraJob("mymodel.onnx")

Using a HWConfig object (recommended):
>>> hw_cfg = HWConfig(product="QC-U", ocm_size="16MB")
>>> c_job = ChimeraJob("mymodel.onnx", hw_config=hw_cfg)

from_hw_config(cls, model_p: str, hw_config: HWConfig, **kwargs)

Create a new ChimeraJob instance from a hardware configuration object.

This is a convenience method that creates a ChimeraJob instance using a pre-configured HWConfig object, which makes it easy to reuse hardware configurations across multiple jobs.

Parameters:

• model_p : str Path to the model
• hw_config : HWConfig The hardware configuration object
• **kwargs : Additional keyword arguments to pass to the ChimeraJob constructor

Returns:

• ChimeraJob A new ChimeraJob instance configured with the provided hardware configuration

Examples:

>>> from tvm.contrib.epu.chimera_job.hw_config import HWConfig
>>> hw_cfg = HWConfig(product="QC-U", ocm_size="16MB", num_cores=2)
>>> job1 = ChimeraJob.from_hw_config("model1.onnx", hw_cfg)
>>> job2 = ChimeraJob.from_hw_config("model2.onnx", hw_cfg)

__str__() -> str

Generate a human-readable string summarizing compile and run statistics.

This method collates information from various internal data structures (such as compile settings, I/O tables, and post-ISS run statistics) into a single formatted string. It includes details about the compile configuration, gate simulation results, and references to the experiment path where additional data can be found.

Returns:

• str A multi-line string containing:

  • Compile summary data, displayed as a transposed table.
  • The estimator target message indicating hardware or product targets.
  • I/O table details, if available.
  • Post-ISS performance and gating statistics for the relevant hardware configuration, including clock frequency.
  • References to the run directory for deeper inspection.

Examples:

>>> c_job = ChimeraJob("mymodel.onnx")
>>> c_job.compile()
>>> print(c_job)

╒═════════════════════╤══════════════════════════════════════════════════════════════╕
│ Module Name         │ resnet18_float32_opt_QC_U_1d7_16MB_4kB_128GBps_128GBps_16_x1 │
├─────────────────────┼──────────────────────────────────────────────────────────────┤
│ ONNX File           │ /Users/daniel/bucket_of_onnx/resnet18_float32_opt.onnx       │
├─────────────────────┼──────────────────────────────────────────────────────────────┤
│ Product Target      │ QC-U                                                         │
├─────────────────────┼──────────────────────────────────────────────────────────────┤
│ Number of Cores     │ 1                                                            │
├─────────────────────┼──────────────────────────────────────────────────────────────┤
│ ISS Clock Frequency │ 1.700                                                        │
├─────────────────────┼──────────────────────────────────────────────────────────────┤
│ L2M Size            │ 16MB                                                         │
├─────────────────────┼──────────────────────────────────────────────────────────────┤
│ External Read BW    │ 128GBps                                                      │
├─────────────────────┼──────────────────────────────────────────────────────────────┤
│ External Write BW   │ 128GBps                                                      │
├─────────────────────┼──────────────────────────────────────────────────────────────┤
│ MACS per PE         │ 16                                                           │
├─────────────────────┼──────────────────────────────────────────────────────────────┤
│ Max L2M             │ 9.749MB                                                      │
├─────────────────────┼──────────────────────────────────────────────────────────────┤
│ Max LRM             │ 2.438kB                                                      │
├─────────────────────┼──────────────────────────────────────────────────────────────┤
│ Max Temp Ext Bytes  │ 0.000MB                                                      │
├─────────────────────┼──────────────────────────────────────────────────────────────┤
│ Network GMACs       │ 1.814                                                        │
╘═════════════════════╧══════════════════════════════════════════════════════════════╛

╒════╤════════╤══════════════════════╤══════════════════╤══════════════════════════╤═══════╕
│    │ Type   │ Name                 │ shape            │ type                     │ mse   │
╞════╪════════╪══════════════════════╪══════════════════╪══════════════════════════╪═══════╡
│  0 │ Input  │ data                 │ [1, 3, 224, 224] │ tensor[FixedPoint32<27>] │ n/a   │
├────┼────────┼──────────────────────┼──────────────────┼──────────────────────────┼───────┤
│  1 │ Output │ resnetv15_dense0_fwd │ [1, 1000]        │ tensor[FixedPoint32<28>] │ n/a   │
╘════╧════════╧══════════════════════╧══════════════════╧══════════════════════════╧═══════╛

has_custom_op() -> bool

Check if the model has any nodes with "QuadricCustomOp" in their name.

Returns:

• bool True if at least one node in the model contains "QuadricCustomOp" in its name, False otherwise.

Examples:

>>> c_job = ChimeraJob("mymodel.onnx")
>>> c_job.has_custom_op()
True

check_quantized_nodes(graph, quantized_keywords) -> bool

Recursively traverse an ONNX graph and checks for quantized nodes. This function also checks for subgraph of custom quadric ops and checks for quantized nodes.

Args: graph: onnx.GraphProto, the graph to traverse quantized_keywords: list of strings, keywords to look for in node op_type Returns: boolean: True if quantized node present

Returns:

• bool

ingest_onnx() -> None

Load the ONNX model and perm pre-compilation checks.

Returns:

• None

analyze_network(print_ccl: bool) -> str

Analyze a network and print out the custom op stubs.

Returns:

• str

generate_fully_quantized_relay() -> None

Generate fully quantized relay intermediate representation (IR).

Returns:

• None

compile(upload_on_success: bool, quiet: bool) -> None

Compile the ONNX model for the target hardware configuration.

This method orchestrates the steps required to compile an ONNX model, including preparing the model for compilation, performing the actual compilation process, and finalizing it (e.g., by generating and storing artifacts). If specified, it can also upload the successfully compiled model to the Quadric Studio. Logging verbosity can be reduced during the compilation process, and is restored afterward.

Parameters:

• upload_on_success : bool If True, uploads the compiled model to Quadric Studio upon successful compilation. Defaults to False.
• quiet : bool If True, sets the logger to output errors only during compilation and restores the original logging level at the end. Defaults to False.

Returns:

• None

Examples:

>>> c_job = ChimeraJob("mymodel.onnx")
>>> c_job.compile()
>>> print(c_job)

Using a custom hardware configuration:
>>> from tvm.contrib.epu.chimera_job.hw_config import HWConfig
>>> hw_cfg = HWConfig(product="QC-U", ocm_size="16MB", num_cores=2)
>>> c_job = ChimeraJob("mymodel.onnx", hw_config=hw_cfg)
>>> c_job.compile()

convert_to_fxpt_onnx(fxpt_model_path)

Convert an ONNX model to a fixed-point quantized version using CGC under the hood to convert onnx to relay and back to onnx.

Parameters:

• fxpt_model_path : str, optional Path where the fixed-point ONNX model will be saved. If not provided, automatically generates a path by adding "_fxpt" suffix to the original model path.

Returns:

• named tuple A FixedPointModel instance containing fields: .onnx, .relay, .io_info, .path

Raises:

Exception If any step in the fixed-point conversion pipeline fails

Notes:

The conversion process involves several steps:

1. Convert ONNX model to Relay intermediate representation
2. Optimize the Relay model
3. Converting floating point to fixed-point tensors
4. Convert back to ONNX format

Examples:

>>> chimera_job = ChimeraJob()
>>> fxpt_model = chimera_job.convert_to_fxpt_onnx()
>>> print(f"Fixed-point model saved to: {fxpt_model.path}")

>>> # Or specify custom path
>>> fxpt_model = chimera_job.convert_to_fxpt_onnx("/path/to/custom_fxpt.onnx")

run_onnx_inf_session(inputs: dict, enable_gpnpu_emulation: bool)

Run an ONNX Runtime inference session on the provided inputs.

This method ensures that the ONNX model is loaded (via ingest_onnx) if it has not yet been initialized. It then prepares and validates the inputs before invoking the ONNX Runtime execution. If onnx_ort_override_p is set, a custom ONNX model is used instead of the default one. The inference execution is timed with elapsed_timer, and the resulting outputs are returned as a dictionary.

For batch_size > 1, ORT is run multiple times (once per batch instance) and outputs are aggregated along the batch dimension.

Parameters:

• inputs : dict A dictionary of input tensors keyed by input node names. Defaults to an empty dictionary. This dictionary is internally adapted for ONNX Runtime during processing.

Returns:

• dict A dictionary containing the outputs from the ONNX Runtime session. The keys correspond to output node names, and the values are the resulting tensors. If validate_iss is True, additional intermediate outputs may also be included.

Notes:

• If model is not already loaded, ingest_onnx() is invoked automatically to load and parse the ONNX graph.
• If onnx_ort_override_p is provided, it will override the standard ONNX model file, and a message is logged to notify about the usage of an alternate model.

Examples:

>>> c_job = ChimeraJob("mymodel.onnx")
>>> # Prepare an example input
>>> input_data = {"input": np.random.rand(1, 3, 224, 224)}
>>> # Execute the ONNX Runtime session
>>> outputs = c_job.run_onnx_inf_session(inputs=input_data)

run_relay_inf_session(inputs: dict)

Run inference on a quantized Relay graph.

This method prepares the given inputs for an ISS (Instruction Set Simulator) context, invokes the Relay execution on a quantized version of the model, and applies float-casting to the resulting outputs. By using the elapsed_timer context manager, the execution time is also measured and logged for performance analysis.

Parameters:

• inputs : dict A dictionary of input data to be used for inference. The specific keys and shapes depend on the model’s expected inputs. This dictionary is internally adapted for ISS execution.

Returns:

• dict A dictionary containing the inference results after float casting has been applied to the raw quantized outputs.

Examples:

>>> c_job = ChimeraJob("mymodel.onnx")
>>> test_inputs = {"input": np.random.rand(1, 3, 224, 224)}
>>> # Run the relay inference session
>>> relay_outputs = c_job.run_relay_inf_session(inputs=test_inputs)

run_inference_harness(inputs: dict, compare_ort: bool, quiet, additional_archsim_options) -> dict[str, Any]

Run a single inference harness on the specified inputs.

This method ensures the network is lowered and compiled if necessary, sets up input data for the harness, and invokes the underlying archsim harness. Optionally, it compares outputs against ONNX Runtime for validation. Post-run metadata (event stream, statistics) and kernel packaging steps are also handled here.

Parameters:

• inputs : dict Key-value pairs representing the input data for inference. Defaults to an empty dictionary.
• compare_ort : bool Whether to compare results with an ONNX Runtime reference, storing mean squared error (MSE) in the output dictionary. Defaults to True.
• quiet : bool, optional If True, reduces logging verbosity during compilation and harness execution. Defaults to False.
• additional_archsim_options : str, optional Additional command-line parameters passed to the archsim harness. Defaults to an empty string.

Returns:

• dict[str, Any] A dictionary containing the inference outputs. If compare_ort is True, the dictionary may include keys for MSE comparisons.

Notes:

• If the network has not yet been lowered or compiled, this method will automatically trigger a compile step.
• If ORT comparison is enabled and the validate_iss flag is not set, the ONNX outputs are compared for a rough MSE validation.
• This method sets and updates various internal attributes, including experiment_path, and parses post-run logs/statistics.

Examples:

>>> c_job = ChimeraJob("mymodel.onnx")
>>> input = {"input": numpytensor2}
>>> result = c_job.run_inference_harness(inputs=input)
>>> print(c_job)

With custom hardware configuration:
>>> from tvm.contrib.epu.chimera_job.hw_config import HWConfig
>>> hw_cfg = HWConfig(product="QC-P", ocm_size="8MB", macs_per_pe=16)
>>> c_job = ChimeraJob("mymodel.onnx", hw_config=hw_cfg)
>>> result = c_job.run_inference_harness(inputs=input)

run_batch_inference_harness(inputs: list, threads: int) -> list

Run a batch inference harness on ISS backend using a specified number of threads.

This method processes multiple input items in batches, invoking the underlying inference harness function for each batch.

Parameters:

• inputs : list A list of inputs to be processed by the inference harness.
• threads : int The number of threads to use for parallel processing. Default is 0, which will use as many physical threads as possible.

Returns:

• list The outputs from running the inference harness on all the provided inputs. The structure of each item in the returned list will match the output expected from the underlying model.

Examples:

>>> c_job = ChimeraJob("mymodel.onnx")
>>> inputs = [{"input"}: numpytensor1, {"input": numpytensor2}]
>>> results = c_job.run_batch_inference_harness(inputs, threads=2)
>>> print(results)
[result_tensor1, result_tensor2]
>>> print(c_job)

run_batch_ort_harness(inputs: list, threads: int) -> list

Run a batch inference harness on ort backend using a specified number of threads.

This method processes multiple input items in batches, invoking the underlying inference harness function for each batch.

Parameters:

• inputs : list A list of inputs to be processed by the inference harness.
• threads : int The number of threads to use for parallel processing. Default is 0, which will use as many physical threads as possible.

Returns:

• list The outputs from running the inference harness on all the provided inputs. The structure of each item in the returned list will match the output expected from the underlying model.

Examples:

>>> c_job = ChimeraJob("mymodel.onnx")
>>> inputs = [{"input"}: numpytensor1, {"input": numpytensor2}]
>>> results = c_job.run_batch_ort_harness(inputs, threads=2)
>>> print(results)
[result_tensor1, result_tensor2]
>>> print(c_job)

parse_profile_results() -> Dict[str, Any]

Process profiling results and generate performance statistics.

This method analyzes profile data from ISS (Instruction Set Simulator) runs, calculates various performance metrics, and returns them as a dictionary. It processes execution cycle counts, power consumption data, and memory access patterns to provide comprehensive performance statistics.

The method requires profile data to be available from a previous execution run. It calculates metrics such as latency, frames per second (FPS), power consumption, memory bandwidth, and MAC utilization.

Returns:

• Dict[str, Any] A dictionary containing performance statistics with the following keys:

  • "Latency (ms)": Execution latency in milliseconds
  • "FPS": Frames per second
  • "Average Power @ 3nm SSGNP (mW)": Average power consumption in milliwatts
  • "FPS per Watt @ 3nm SSGNP (FPS/W)": Energy efficiency metric
  • "Ext Rd Bytes (MB)": External read memory traffic in megabytes
  • "Ext Wr Bytes (MB)": External write memory traffic in megabytes
  • "Avg Ext Rd BW (GBps)": Average external read bandwidth in gigabytes per second
  • "Avg Ext Wr BW (GBps)": Average external write bandwidth in gigabytes per second
  • "MAC Utilization": Percentage of MAC units utilized (if network GMACs are available)

  Returns an empty dictionary if no profile data is found.

Notes:

• This method depends on prior execution of ISS simulation and profile data collection.
• Power consumption metrics are based on simulation models and may be scaled to represent different process technologies.
• MAC utilization is only calculated if the total_network_gmacs attribute is set.
• All numeric values in the returned dictionary are formatted as strings with appropriate units and precision.

plot_run_statistics(per_core: bool, close: bool) -> str

Plot and save run statistics, returning the data path.

This method parses post-run statistics for the current experiment, generates plots of combined performance and power data, and saves them as PNG files. It also writes various JSON files that summarize activity and power data, along with compile/run I/O settings.

Parameters:

• per_core : bool If True, generate per-core statistics and plots. Otherwise, generate aggregated (combined) statistics. Defaults to False.
• close : bool If True, closes the generated figure after saving. Defaults to False.

Returns:

• str The path to the directory where the plots and JSON files are stored.

Raises:

RuntimeError If the parsed run statistics do not contain a 'region_name' of 'default', indicating missing or incomplete performance data.

Notes:

• The final combined figure is saved as <name>.combined.png in the data directory, where <name> is the experiment or model name.

Examples:

>>> c_job = ChimeraJob("mymodel.onnx")
>>> c_job.run_inference_harness() #run iss with random inputs
>>> c_job.plot_run_statistics()
>>> print(c_job)

With a custom hardware configuration:
>>> from tvm.contrib.epu.chimera_job.hw_config import HWConfig
>>> hw_cfg = HWConfig(product="QC-U", ocm_size="16MB", ext_rd_bw="64GBps")
>>> c_job = ChimeraJob("mymodel.onnx", hw_config=hw_cfg)
>>> c_job.run_inference_harness()
>>> c_job.plot_run_statistics()

validate_ort_iss(inputs: dict, quiet: bool) -> TableType

Validate ONNX Runtime outputs against ISS (Instruction Set Simulator) results.

This method compares the outputs of an ONNX Runtime inference session with the results of an ISS-based inference harness to ensure they match within specified tolerances. If any discrepancies are detected, a validation report is generated and stored as JSON.

Parameters:

• inputs : dict A dictionary of inputs for both ONNX and ISS inference. Defaults to an empty dict, which may trigger internal defaults.
• quiet : bool If True, temporarily suppresses logging messages during the validation process, restoring the previous logging level afterward. Defaults to False.

Returns:

• TableType A list (table) of validation summary entries. Each entry typically contains details of the comparison between ONNX and ISS outputs per FILM region, along with any errors or differences encountered.

Raises:

ValueError If validate_iss is not set before calling this method. The validation requires ISS comparisons to be enabled at initialization. ChimeraValidationDifference Propagated internally by validate_golden_outputs_vs_iss if differences in the results exceed the allowed tolerances. However, the exception is caught here, so this is primarily relevant if you modify the exception-handling logic.

Notes:

• The validation results are written to <experiment_path>/<DATA_PATH>/validation_report.json. <experiment_path>/<DATA_PATH>/validation_report.xlsx.
• The method uses a relaxed tolerance (rtol=1e-1, atol=-1)