Model Compilers & Optimizers

Tool	Description
Apache TVM	a compiler stack for deep learning systems. It is designed to close the gap between the productivity-focused deep learning frameworks, and the performance- and efficiency-focused hardware backends
Hidet	An open-source deep learning compiler, written in Python. It supports end-to-end compilation of DNN models from PyTorch and ONNX to efficient cuda kernels.
OpenVINO™	an open-source toolkit for optimizing and deploying deep learning models. It provides boosted deep learning performance for vision, audio, and language models from popular frameworks like TensorFlow, PyTorch, and more.
Speedster	automatically applies the best set of SOTA optimization techniques to achieve the maximum inference speed-up (latency, throughput, model size) physically possible on your hardware (single machine)
Neural Magic SparseML	An open-source model optimization toolkit that enables you to create inference-optimized sparse models using pruning, quantization, and distillation algorithms
Nvidia TensorRT	SDK for high-performance deep learning inference, includes a deep learning inference optimizer and runtime that delivers low latency and high throughput for inference applications.
XLA	takes models from popular ML frameworks such as PyTorch, TensorFlow, and JAX, and optimizes them for high-performance execution across different hardware platforms including GPUs, CPUs, and ML accelerators

Model Zoo

Zoo	Description
Hailo Model Zoo	Provides deep learning models for various computer vision tasks. The pre-trained models can be used to create fast prototypes on Hailo devices
Nvidia Pretrained AI Models	A collection of 600+ highly accurate models built by NVIDIA researchers and engineers using representative public and proprietary datasets for domain-specific tasks.
OpenVINO Model Zoo	Browse through over 200 neural network models, both public and from Intel, and pick the right one for your solution. Types include object detection, classification, image segmentation, handwriting recognition, text to speech, pose estimation, and others.
Pytorch Hub	Discover and publish models to a pre-trained model repository designed for research exploration.
Torch Serve Model Zoo	Pre-trained and pre-packaged, ready to be served for inference with TorchServe
Torch Vision Models	Contains definitions of models for addressing different tasks, including: image classification, pixelwise semantic segmentation, object detection, instance segmentation, person keypoint detection, video classification, and optical flow

Runtimes

Technology	Description
WONNX	A GPU-accelerated ONNX inference run-time written 100% in Rust, ready for the web.

Silicon & Chips

Technology	Description
Axelera AI	Metis AI Platform – a holistic hardware and software solution for AI inference at the Edge – empowers computer vision applications to become more accessible and powerful than ever before. At its core is our new Metis AI Processing Unit (AIPU), which establishes new price/performance and performance/Watt standards.
Blaize	Blaize Pathfinder and Xplorer AI Edge Platforms that are more efficient, more flexible, more accurate and more cost effective. Now you can deploy AI on the edge without sacrifice.
DeGirium	DeGirum ORCA™ is flexible, efficient, and affordable AI accelerator IC. ORCA provides application developers the ability to create rich, sophisticated, and highly functional products at the power and price suitable for the edge. ORCA™ is powered by a very efficient compute architecture, with support for pruned models.
Hailo	Hailo-8™ edge AI processor, featuring up to 26 tera-operations per second (TOPS), significantly outperforms all other edge processors. Its area and power efficiency are far superior to other leading solutions by a considerable order of magnitude – at a size smaller than a penny even including the required memory.
Sapeon	SAPEON has an optimal architecture for low-latency, large-scale inference of deep neural networks. Our products are optimally designed to process artificial intelligence tasks faster, using less power by efficiently processing large amounts of data simultaneously.

Tools

Tool	Description
UpTrain	An open-source, data-secure tool for ML practitioners to observe and refine their ML models by monitoring their performance, checking for (data) distribution shifts, and collecting edge cases to retrain them upon.
MLC LLM	MLC LLM is a universal solution that allows any language models to be deployed natively on a diverse set of hardware backends and native applications, plus a productive framework for everyone to further optimize model performance for their own use cases.