VIP 👤
🏠 Início
Avaliações
📊 Todos os Benchmarks 🦖 Dinossauro v1 🦖 Dinossauro v2 ✅ Aplicações To-Do List 🎨 Páginas Livres Criativas 🎯 FSACB - Showcase Definitivo 🌍 Benchmark de Tradução
Modelos
🏆 Top 10 Modelos 🆓 Modelos Gratuitos 📋 Todos os Modelos ⚙️ Kilo Code
Recursos
💬 Biblioteca de Prompts 📖 Glossário de IA 🔗 Links Úteis 🔌 APIs e roteadores
Advanced

Optimize a Machine Learning Inference Pipeline

#machine-learning #optimization #mlops #latency

Design an efficient pipeline for serving deep learning models in real-time with strict resource constraints.

You are responsible for optimizing the inference pipeline of a large language model (LLM) running on edge devices with limited RAM and compute power. Describe techniques you would employ to reduce the model size and inference latency, such as quantization, pruning, knowledge distillation, and operator fusion. Further, explain how you would implement a batching strategy to maximize throughput without violating real-time constraints. Provide a comparative analysis of ONNX Runtime versus TensorRT for this specific use case.