SageMaker Inference adds NVIDIA MIG to split GPUs across models
What is SageMaker Inference now supports Multi-Instance GPU to maximize GPU utilization for model endpoints?
Amazon SageMaker now lets you partition a single GPU into isolated slices using NVIDIA’s Multi-Instance GPU (MIG). You can run several smaller models on one card or allocate just the fraction of power each model needs.
Who it is for
ML engineers deploying inference at scale can cut cloud costs and improve latency by better packing workloads onto existing GPU instances.