wowsuch ai
Live

AWSDeveloper platform

SageMaker AI endpoints

SageMaker AI endpoints are AWS-managed inference endpoints for deploying generative AI models where AWS handles GPU provisioning, autoscaling, and operational monitoring. In 2026 AWS introduced multiple enhancements including automated inference recommendations, multi-instance capacity-aware pools, an OpenAI-compatible API surface, container caching to reduce cold-starts, token-level observability, inline async payloads, and prefix-aware routing for KV cache reuse.

What it does

You can deploy and serve large generative models with managed scaling, performance tuning, OpenAI-compatible APIs, and production telemetry for low-latency, high-throughput inference.

Availability
Available across all AWS Regions
Pricing model
Only pay for what you use
Deployment
Amazon ECR private repositories only
Privacy
Customize models with proprietary data
Licence
Not established
Capability
text generationanalysisretrievalembeddingsagentsautomationevaluationsafetyfine-tuningtraining
Modality
textmultimodalcode
Access
APISDKCLIweb
Deployment
hosted
Pricing
usage-basedenterpriseunknown
Target user
enterprisedeveloperresearcher

Evidence

23 claims
  • SageMaker endpoints expose an /openai/v1 path supporting Chat Completions with streaming to enable drop-in compatibility with OpenAI SDKs. [1]
    SageMaker endpoints now expose an /openai/v1 path supporting Chat Completions with streaming.
    AWS. Quoted text is outside CC BY 4.0 and keeps its original rights.
  • Inference recommendations automate instance selection, optimization (speculative decoding, kernel tuning, tensor parallelism), and benchmarking to produce a deployment-ready SageMaker Model Package at no extra cost for recommendation runs. [1]
  • Container caching pre-pulls serving container images on supported accelerator instance types and reduced end-to-end startup latency by 51% in a demonstrated example. [1]
  • Async InvokeEndpointAsync now accepts inline Body payloads up to 128,000 bytes and is available in 31 AWS Regions. [1]
    The InvokeEndpointAsync API now accepts a Body parameter with payloads up to 128,000 bytes directly in the request, removing the S3 pre-staging step for the vast majority of async workloads.
    AWS. Quoted text is outside CC BY 4.0 and keeps its original rights.
  • SageMaker emits 100+ detailed inference metrics via OpenTelemetry and ships a pre-built CloudWatch Insights dashboard with token-level metrics enabled by default for new endpoints. [1]
    SageMaker now emits 100+ detailed inference metrics via native OpenTelemetry, paired with a pre-built Insights dashboard in Amazon CloudWatch.
    AWS. Quoted text is outside CC BY 4.0 and keeps its original rights.
  • Prefix-aware routing consistently routes requests with shared prompt prefixes to the same instance, improving KV cache hit rates (example: from ~25% to 82%) and reducing P50/P90 time-to-first-token in benchmarks. [1]
  • You can specify a prioritized list of instance types and counts when submitting training and processing jobs [2]
    To use this feature, you specify your instance type and count preferences in priority order when submitting the training or processing job.
    AWS. Quoted text is outside CC BY 4.0 and keeps its original rights.
  • SageMaker will run the job on the first configuration in the list where capacity is available [2]
    SageMaker works through the list and launches your job on the first configuration where capacity is available.
    AWS. Quoted text is outside CC BY 4.0 and keeps its original rights.
  • You can configure capacity sourcing from on‑demand capacity or from reserved SageMaker Flexible Training Plans within the same job submission [2]
    You can also configure the capacity sourcing from on-demand sources or from your reserved SageMaker Flexible Training Plans within the same job submission.
    AWS. Quoted text is outside CC BY 4.0 and keeps its original rights.
  • Feature is available in all AWS Regions where SageMaker is available [2]
    Instance preference lists for SageMaker training and processing jobs is available today in all AWS Regions where SageMaker is available through the SageMaker CLIs, APIs, SDKs and Console UI.
    AWS. Quoted text is outside CC BY 4.0 and keeps its original rights.
  • Accessible through the SageMaker CLIs, APIs, SDKs, and Console UI [2]
    Instance preference lists for SageMaker training and processing jobs is available today in all AWS Regions where SageMaker is available through the SageMaker CLIs, APIs, SDKs and Console UI.
    AWS. Quoted text is outside CC BY 4.0 and keeps its original rights.
  • Customization is available for Muse Glimmer, Qwen3.8 27B, and Ministral 3 3B, 8B, and 14B. [3]
    Amazon SageMaker AI now supports serverless model customization for Muse Glimmer, Qwen3.8 27B, and Ministral 3 3B, 8B, and 14B models.
    AWS. Quoted text is outside CC BY 4.0 and keeps its original rights.
  • Supported techniques include SFT, DPO, RFT, LoRA, and full fine-tuning. [3]
    In addition to deploying these models on SageMaker AI, you can now adapt them to your specific domains and workflows, using supervised fine-tuning (SFT), direct preference optimization (DPO), and reinforcement fine-tuning (RFT) techniques.
    AWS. Quoted text is outside CC BY 4.0 and keeps its original rights.
  • SageMaker AI handles infrastructure provisioning and training orchestration for serverless customization. [3]
    With serverless customization, SageMaker AI handles all infrastructure provisioning and training orchestration, so you can focus on your data and evaluation rather than cluster management, and only pay for what you use.
    AWS. Quoted text is outside CC BY 4.0 and keeps its original rights.
  • Serverless model customization is available in US East (N. Virginia), US West (Oregon), Asia Pacific (Tokyo), and Europe (Ireland). [3]
    Virginia), US West (Oregon), Asia Pacific (Tokyo), and Europe (Ireland).
    AWS. Quoted text is outside CC BY 4.0 and keeps its original rights.
  • Customers only pay for what they use. [3]
    With serverless customization, SageMaker AI handles all infrastructure provisioning and training orchestration, so you can focus on your data and evaluation rather than cluster management, and only pay for what you use.
    AWS. Quoted text is outside CC BY 4.0 and keeps its original rights.
  • Users can start a customization job from SageMaker Studio, the SageMaker Python SDK, or Agent skills with a coding assistant. [3]
    To get started, navigate to the Models page in Amazon SageMaker Studio to launch a customization job, use the SageMaker Python SDK or use our Agent skills with your choice of coding assistant.
    AWS. Quoted text is outside CC BY 4.0 and keeps its original rights.
  • SageMaker AI lazy-loads SOCI-indexed container images for training and processing jobs. [4]
    Today, we’re excited to announce Seekable OCI (SOCI) indexing for Amazon SageMaker AI Training and Amazon SageMaker AI processing jobs , which takes full container image download off the critical path of every job.
    AWS. Quoted text is outside CC BY 4.0 and keeps its original rights.
  • The service fetches files needed to start the container and streams the remaining files in the background. [4]
    It fetches only the files needed to start the container and streams the rest in the background as your code runs, so your job begins in seconds instead of minutes.
    AWS. Quoted text is outside CC BY 4.0 and keeps its original rights.
  • SOCI indexing for SageMaker AI jobs is available across all AWS Regions. [4]
    SOCI indexing for SageMaker AI jobs is available across all AWS Regions.
    AWS. Quoted text is outside CC BY 4.0 and keeps its original rights.
  • Lazy loading is limited to images in Amazon ECR private repositories. [4]
  • Jobs fall back to a standard download if the index cannot be read or lazy loading does not succeed. [4]
    Because jobs always fall back to a standard download when needed, adding a SOCI index never prevents your job from running.
    AWS. Quoted text is outside CC BY 4.0 and keeps its original rights.
  • AWS reports that GPU idle time during image pulls was reduced by up to 90 percent in its benchmarks. [4]
    As we show in the What we measured section, this reduced GPU idle time during image pulls by up to 90 percent in our benchmarks.
    AWS. Quoted text is outside CC BY 4.0 and keeps its original rights.

Sources

4
3
AWSOwner-controlled
https://aws.amazon.com/about-aws/whats-new/2026/10/amazon-sagemaker-muse-qwen-ministral/ Retrieved 2026-10-25 05:55 UTC
Verified

Versions

6
VersionEffectiveChangeChanged fieldsState
62026-10-28 00:19 UTCmajor releasemodality, deployment, availability, releaseCurrent
52026-10-25 05:55 UTCmajor releasecapability, privacy, deployment, availability, pricing model, releaseCurrent
42026-10-19 17:30 UTCevidence updatescoreCurrent
32026-10-15 20:56 UTCevidence updatescoreCurrent
22026-09-23 19:31 UTCmajor releasepricing, deployment, availability, releaseCurrent
12026-09-23 17:30 UTCpublicationNoneCurrent

In briefs

2

Open in the catalog

Original catalog content is licensed under CC BY 4.0, attributed to wowsuchai. Source quotations and linked content keep their original rights.