Back to Articles
Cut GPU inference cold start from 8 minutes to less than a minute
AI & Machine Learning

Cut GPU inference cold start from 8 minutes to less than a minute

Ali NematiAli NematiSep 3
1m & 1 s read31 views0 listens

The article discusses a comprehensive approach to optimizing the Time To First Token Sent (TTFTS) in large language model (LLM) inference, focusing on reducing cold-node startup times for models running on Kubernetes with NVIDIA GPUs. The optimization strategy is divided into six layers:

  1. Platform Fixes:

    • Use pre-compiled drivers and libraries instead of compiling them at runtime.
    • Optimize Docker image transfer using parallel pull to reduce the time from minutes to seconds.

  2. Kubernetes Configuration:

    • Utilize Karpenter for efficient node provisioning, ensuring nodes are provisioned with exactly the required hardware resources without unnecessary overhead.

  3. Image Optimization:

    • Use large container images that include necessary dependencies and configurations pre-installed, reducing the need for runtime installations or compilations.

  4. Model Weights Download:

    • Optimize S3 downloads by splitting model files into small chunks (e.g., 4 GB) and downloading them concurrently to saturate network bandwidth.
    • Implement aggressive timeouts and retries for slow requests to avoid bottlenecks caused by stalled connections.

  5. GPU Kernel Compilation Cache:

    • Leverage the torch.compile cache, storing compiled CUDA kernels on

Read the full article at The New Stack

Want to create content about this topic? Use Nemati AI tools to generate articles, social posts, and more.

Views31
Tags
#AIComputer VisionLLM
Ali Nemati
Ali NematiBlogUI.author_articles
BlogUI.view_profile

You Might Like

Autoregressive vs Diffusion: A Different Way AI Could Generate Text
AI & Machine LearningSep 1622 sec read

Autoregressive vs Diffusion: A Different Way AI Could Generate Text

Two distinct approaches for AI text generation are autoregressive and diffusion. Autoregressive models generate text token by token sequentially, a method used by most current language models. Diffusion models, however, operate more like sculpting, i...

Ali NematiAli Nemati
Read More
DeepSeek's first vision model vs. Gemini 3.7 Flash: It comes down to spend vs. speed
AI & Machine LearningAug 3128 sec read

DeepSeek's first vision model vs. Gemini 3.7 Flash: It comes down to spend vs. speed

DeepSeek released its first image-processing model, V4 Flash Vision Exp, which competes with Google’s Gemini 3.7 Flash in terms of cost and performance for developers. While both models showed equal accuracy in understanding charts, invoices, and pro...

Ali NematiAli Nemati
Read More
What Happens When You Use LoRA On a CNN
AI & Machine LearningMay 431 sec read

What Happens When You Use LoRA On a CNN

A researcher tested Low-Rank Adaptation (LoRA) technique on a Convolutional Neural Network (CNN) trained on MNIST for adaptation to EMNIST. The experiment aimed to determine if LoRA could efficiently fine-tune CNNs without retraining the entire model...

Ali NematiAli Nemati
Read More