ONNXRuntime Extensions
Introduction: ONNXRuntime-Extensions is a library that extends the capability of the ONNX models and inference with ONNX Runtime, via ONNX Runtime Custom Operator ABIs. It includes a set of ONNX Runtime Custom Operator to support the common pre- and post-processing operators for vision, text, and nlp models. And it supports multiple languages and platforms, like Python on Windows/Linux/macOS, some mobile platforms like Android and iOS, and Web-Assembly etc. The basic workflow is to enhance a ONNX model firstly and then do the model inference with ONNX Runtime and ONNXRuntime-Extensions package.
pip install onnxruntime-extensions
pip install --index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ORT-Nightly/pypi/simple/ onnxruntime-extensions
Please ensure that you have met the prerequisites of onnxruntime-extensions (e.g., onnx and onnxruntime) in your Python environment.
Please make sure the compiler toolkit like gcc(later than g++ 8.0) or clang are installed before the following command
python -m pip install git+https://github.com/microsoft/onnxruntime-extensions.git
With onnxruntime-extensions Python package, you can easily get the ONNX processing graph by converting them from Huggingface transformer data processing classes, check the following API for details.
help(onnxruntime_extensions.gen_processing_models)
There are individual packages for the following languages, please install it for the build.
import onnxruntime as _ort
from onnxruntime_extensions import get_library_path as _lib_path
so = _ort.SessionOptions()
so.register_custom_ops_library(_lib_path())
# Run the ONNXRuntime Session, as ONNXRuntime docs suggested.
# sess = _ort.InferenceSession(model, so)
# sess.run (...)
// The line loads the customop library into ONNXRuntime engine to load the ONNX model with the custom op
Ort::ThrowOnError(Ort::GetApi().RegisterCustomOpsLibrary((OrtSessionOptions*)session_options, custom_op_library_filename, &handle));
// The regular ONNXRuntime invoking to run the model.
Ort::Session session(env, model_uri, session_options);
RunSession(session, inputs, outputs);
var env = OrtEnvironment.getEnvironment();
var sess_opt = new OrtSession.SessionOptions();
/* Register the custom ops from onnxruntime-extensions */
sess_opt.registerCustomOpLibrary(OrtxPackage.getLibraryPath());
SessionOptions options = new SessionOptions()
options.RegisterOrtExtensions()
session = new InferenceSession(model, options)