Amazon SageMaker Python SDK Integrates LLM Optimization Tools

The Amazon SageMaker Python SDK v3 introduces generative AI inference recommendations, enabling users to optimize deployment of large language models. The service benchmarks live endpoints against synthetic or real traffic, measures throughput, time-to-first-token (TTFT), and latency, and generates ranked deployment recommendations based on cost-performance tradeoffs. Users can deploy the top-ranked configuration directly to an Amazon SageMaker real-time endpoint. The new SDK interface, available in the sagemaker.serve.ai_inference_recommender package starting with version 3.17.0, includes operations like ModelBuilder.from_jumpstart_config, start_benchmark, generate_deployment_recommendations, and deploy. The post demonstrates an end-to-end workflow: generating recommendations, interpreting results, deploying, benchmarking, and comparing frameworks like LMI and vLLM. It also explains how to read key metrics and choose between configurations based on performance targets, and how to deploy from a previously run recommendation job using ModelBuilder.from_recommendation_job.