Optimize Deep Learning
Models for Cost-effective
Cloud Inference at Scale

The Challenge

Operating your deep learning models in cloud environments is a costly matter and high inference costs may dramatically cut down your product’s profitability. Companies' clear goal is to offer their end-users a high performing deep learning model, without compromising on the model’s accuracy and SLA. However, customers are also challenged with the appropriate cloud-based compute configuration required for their scalable model inference tasks. For example, in today’s general-purpose cloud environments, customers may need to select and trade-off between using a lower-cost cluster of CPU-based compute instances to the higher-cost, top-performing single dedicated GPU-based compute instance. At the end of the day, companies face the dilemma of executing model inference models at a lower operating margin but with high performance or sacrificing the user experience with poorly-performing deep learning models.