Optimize Deep Learning
Models for Cost-effective
Cloud Inference at Scale

The Challenge

Operating your deep learning models in cloud environments is a costly matter and high inference costs can dramatically cut down your product’s profitability. Companies want to offer end-users a high-performance deep learning model, without compromising on the model’s accuracy and SLA. However, customers are also challenged with finding the appropriate cloud-based compute configuration required for their scalable model inference tasks. For example, in today’s general-purpose cloud environments, customers may need to choose and trade-off between using a lower-cost cluster of CPU-based compute instances and a higher-cost, top-performing single dedicated GPU-based compute instance. At the end of the day, companies face the dilemma of executing model inference models at a lower operating margin but with high performance or sacrificing the user experience with poorly-performing deep learning models.