Why LLMs Are Slow
Executive Summary
Executive Summary
LLMs are slow primarily because autoregressive token generation requires sequential computation, memory bandwidth constraints dominate GPU utilization, and attention mechanisms scale quadratically with sequence length.
Autoregressive generation forces sequential token-by-token output where each token depends on all previous tokens, fundamentally limiting parallelization and creating an inherent latency floor that scales linearly with output length.
Memory bandwidth, not compute capacity, is the primary bottleneck for LLM inference because the model weights must be loaded from GPU memory for each token generated, and modern GPUs have compute-to-memory ratios that leave arithmetic units idle waiting for data.
The KV cache required for efficient attention grows linearly with sequence length and batch size, consuming GPU memory that could otherwise enable larger batches, creating a tradeoff between latency, throughput, and maximum context length.
The Bottom Line
LLM slowness is an architectural consequence of the transformer design and autoregressive generation paradigm, not merely an engineering oversight. Understanding these fundamental constraints enables practitioners to make informed tradeoffs between latency, throughput, cost, and quality rather than expecting silver-bullet solutions.
Definition
Definition
LLM slowness refers to the inherent latency characteristics of large language model inference, where generating text requires sequential token-by-token computation with each token depending on the complete preceding context.
This slowness manifests as time-to-first-token (TTFT) delays during prompt processing and inter-token latency during generation, both constrained by memory bandwidth limitations, attention computation costs, and the fundamental sequential nature of autoregressive decoding.
Extended Definition
The performance characteristics of LLM inference differ fundamentally from traditional software systems because the computation is dominated by memory-bound operations rather than compute-bound operations. While modern GPUs offer massive parallel compute capacity measured in teraflops, LLM inference typically achieves only a fraction of theoretical throughput because the model weights must be repeatedly transferred from high-bandwidth memory to compute units. Additionally, the autoregressive generation paradigm means that generating N tokens requires N sequential forward passes through the model, with each pass dependent on the output of the previous pass. This creates an inherent serialization that cannot be parallelized away, establishing a fundamental latency floor proportional to output length regardless of available compute resources.
Etymology & Origins
The term 'LLM slowness' emerged organically in the AI engineering community around 2022-2023 as practitioners began deploying large language models in production systems and encountered latency characteristics that differed dramatically from traditional web services. The underlying concepts draw from decades of computer architecture research on memory hierarchy bottlenecks, parallel computing limitations, and the distinction between latency-bound and throughput-bound workloads. The specific framing around 'why LLMs are slow' gained prominence as users of ChatGPT and similar systems observed the characteristic typing-like output pattern and engineers sought to understand and optimize these systems.
Also Known As
Not To Be Confused With
LLM training time
Training involves processing massive datasets to learn model weights over days or weeks, while inference slowness refers to the latency of generating outputs from a trained model, which occurs in seconds to minutes. Training is compute-bound and parallelizable; inference is memory-bound and sequential.
Network latency
Network latency refers to communication delays between client and server, which adds to but is separate from the fundamental model inference time. A locally-run LLM still exhibits generation slowness even with zero network overhead.
API rate limiting
Rate limiting is an artificial constraint imposed by service providers to manage capacity, distinct from the inherent computational latency of model inference. Rate limits affect request frequency; inference slowness affects individual request duration.
Model loading time
Model loading refers to the one-time cost of transferring model weights from disk to GPU memory when starting a service, while inference slowness refers to the per-request computation time after the model is loaded and ready.
Prompt processing time
While prompt processing (prefill) is part of overall latency, it is parallelizable and compute-bound, differing from the sequential, memory-bound token generation phase that dominates total response time for longer outputs.
Quantization artifacts
Quantization reduces model precision to improve speed but may affect output quality. The fundamental slowness exists regardless of quantization level; quantization is one optimization approach, not the cause of slowness.
Conceptual Foundation
Conceptual Foundation
Core Principles
(7 principles)Mental Models
(6 models)The Assembly Line Analogy
Think of LLM generation as an assembly line where each station (token) must complete before the next can begin, and each station requires fetching heavy components (model weights) from a distant warehouse (GPU memory). The line speed is limited by warehouse delivery time, not assembly speed.
The Typing Speed Ceiling
Imagine the LLM as a typist who must think carefully about each word before typing it, and whose thinking time is fixed regardless of how fast their fingers can move. The typing speed has a ceiling determined by thinking time, not motor skills.
The Library Book Retrieval Model
Consider each token generation as requiring the librarian to retrieve and consult every book in a massive library (model weights). The time is dominated by walking to shelves and carrying books, not by reading them. A bigger library means more walking time.
The Expanding Conversation Memory
Visualize the KV cache as a growing stack of notes the LLM must review before speaking each word. As the conversation lengthens, the stack grows, requiring more time to review and more desk space (memory) to store.
The Batch Processing Queue
Think of batched inference as a bus that waits to fill up before departing. Larger buses (batches) are more fuel-efficient (better throughput) but passengers (requests) wait longer for departure and arrival depends on the longest trip in the batch.
The Precision-Speed Dial
Imagine a dial that trades precision for speed: turning it toward speed (quantization) makes the model faster but slightly less accurate, while turning it toward precision maintains quality but keeps the model slow.
Key Insights
(10 insights)The memory bandwidth bottleneck means that a GPU running LLM inference typically achieves less than 1% of its theoretical compute capacity, making raw FLOPS comparisons misleading for inference performance prediction.
Time-to-first-token (TTFT) and inter-token latency are governed by different bottlenecks: TTFT is compute-bound during prompt processing, while inter-token latency is memory-bound during generation, requiring different optimization strategies.
The KV cache for a single long-context request can consume more GPU memory than the model weights themselves, making memory management the primary constraint for long-context applications.
Continuous batching techniques can improve throughput by 2-10x over static batching by allowing new requests to join and completed requests to leave mid-generation, but implementation complexity is significant.
Model parallelism across multiple GPUs can reduce per-token latency by distributing weight loading, but introduces communication overhead that provides diminishing returns beyond 4-8 GPUs for inference.
Speculative decoding with draft models provides consistent speedups only when the draft model accurately predicts the target model's outputs, which varies significantly by domain and prompt type.
Flash Attention and similar algorithmic optimizations reduce memory traffic for attention computation but do not address the fundamental memory bandwidth bottleneck for weight loading during generation.
The optimal batch size for inference depends on sequence length, model size, and GPU memory capacity, and must be dynamically adjusted rather than statically configured for production workloads.
Quantization to INT8 or INT4 can provide 2-4x speedups with minimal quality degradation for many tasks, but the speedup varies by hardware support and may require calibration for specific use cases.
Prefix caching can dramatically reduce TTFT for applications with common prompt prefixes (like system prompts), but provides no benefit for the generation phase that typically dominates total latency.
When to Use
When to Use
Ideal Scenarios
(12)When diagnosing production LLM systems exhibiting higher-than-expected latency to identify whether the bottleneck is memory bandwidth, attention computation, batching configuration, or infrastructure issues.
When setting realistic performance expectations with product stakeholders who may expect LLM responses to be as fast as traditional database queries or API calls.
When designing system architectures that must meet specific latency SLAs, requiring understanding of fundamental limits to avoid over-promising or under-engineering.
When evaluating hardware procurement decisions between different GPU models, understanding that memory bandwidth and capacity may matter more than raw compute for inference workloads.
When implementing optimization strategies like quantization, speculative decoding, or batching, requiring understanding of which bottlenecks each technique addresses.
When debugging user experience issues in conversational AI applications where perceived slowness impacts engagement and satisfaction.
When capacity planning for LLM inference infrastructure, requiring accurate models of latency and throughput characteristics under various load conditions.
When comparing different model sizes and architectures for deployment, understanding the latency implications of parameter count and attention mechanism design.
When designing streaming interfaces that mask latency through progressive output delivery, requiring understanding of token generation timing characteristics.
When implementing timeout and retry logic for LLM API calls, requiring understanding of expected latency distributions and failure modes.
When evaluating managed LLM services versus self-hosted deployment, understanding the performance characteristics that affect cost and user experience.
When training teams on LLM system behavior to build accurate mental models for debugging and optimization.
Prerequisites
(8)Basic understanding of neural network inference and the distinction between training and inference phases.
Familiarity with GPU architecture concepts including memory hierarchy, compute units, and memory bandwidth.
Understanding of the transformer architecture and self-attention mechanism at a conceptual level.
Knowledge of autoregressive language modeling and how tokens are generated sequentially.
Experience with production system performance analysis including latency measurement and profiling.
Understanding of batching concepts and the relationship between batch size, throughput, and latency.
Familiarity with common LLM deployment patterns including API services, streaming, and edge deployment.
Basic knowledge of model quantization and precision formats (FP16, INT8, INT4).
Signals You Need This
(10)Users are complaining about slow response times in your LLM-powered application despite adequate infrastructure.
Your LLM inference costs are higher than expected because you're over-provisioning hardware to meet latency targets.
You're observing high GPU utilization during prompt processing but low utilization during token generation.
Long conversations or documents cause disproportionate slowdowns compared to short interactions.
Batching requests improves throughput but individual request latency becomes unacceptable.
Your team is debating hardware choices without clear understanding of which specifications matter for inference.
Optimization attempts are not yielding expected improvements because they're targeting the wrong bottleneck.
You're evaluating different model sizes and need to understand the latency-quality tradeoff quantitatively.
Streaming implementation isn't improving perceived performance as much as expected.
Your capacity planning models are inaccurate, leading to either over-provisioning or SLA violations.
Organizational Readiness
(7)Engineering team has access to GPU profiling tools and can interpret memory bandwidth utilization metrics.
Organization has established latency SLAs and measurement infrastructure for LLM services.
Product team understands that LLM latency characteristics differ fundamentally from traditional services.
Budget exists for potential hardware upgrades or architectural changes based on optimization findings.
Team has authority to make tradeoff decisions between latency, throughput, cost, and quality.
Infrastructure supports experimentation with different batching strategies and model configurations.
Monitoring systems can capture detailed latency breakdowns including TTFT and inter-token latency.
When NOT to Use
When NOT to Use
Anti-Patterns
(12)Attempting to achieve sub-100ms total response times for multi-sentence LLM outputs without understanding that this violates fundamental sequential generation constraints.
Scaling horizontally with more GPUs expecting linear latency reduction, when the bottleneck is sequential token generation that cannot be parallelized.
Implementing complex optimization techniques before measuring to identify the actual bottleneck, potentially optimizing the wrong component.
Choosing the largest available model assuming quality is the only consideration, without accounting for the latency implications of model size.
Disabling batching entirely to minimize latency without understanding the cost implications of reduced throughput.
Expecting speculative decoding to provide order-of-magnitude speedups when typical improvements are 2-3x under favorable conditions.
Implementing aggressive quantization without validating quality impact on your specific use case and task distribution.
Caching LLM outputs for dynamic queries where the cache hit rate will be negligible, adding complexity without benefit.
Using synchronous LLM calls in latency-critical paths without implementing streaming or asynchronous patterns.
Comparing LLM latency to traditional database or API latency without adjusting expectations for the fundamentally different computation model.
Over-engineering for worst-case latency scenarios that occur rarely, when p50 or p95 latency is the appropriate optimization target.
Implementing custom inference engines before exhausting optimization opportunities in existing frameworks and configurations.
Red Flags
(10)Team believes adding more GPUs will proportionally reduce per-request latency for single requests.
Latency targets are set without understanding the relationship between output length and generation time.
Optimization efforts focus exclusively on compute efficiency while ignoring memory bandwidth constraints.
Hardware selection prioritizes FLOPS over memory bandwidth for inference-focused workloads.
Batching configuration is static rather than dynamically adjusted based on load and request characteristics.
Quality degradation from optimization is not being measured or is being ignored in pursuit of speed.
Team is implementing multiple optimization techniques simultaneously without isolating their individual effects.
Latency measurements don't distinguish between TTFT and generation time, masking the true bottleneck.
Cost analysis doesn't account for the throughput implications of latency optimization choices.
User experience design doesn't incorporate streaming or progressive disclosure to manage perceived latency.
Better Alternatives
(8)Application requires sub-100ms response times for simple classification or extraction tasks
Fine-tuned smaller models or traditional ML classifiers
Smaller specialized models can achieve adequate quality for narrow tasks with 10-100x lower latency than general-purpose LLMs, and traditional ML models can be even faster.
High-volume, low-latency requirements with predictable query patterns
Semantic caching with embedding similarity matching
Caching responses for semantically similar queries can provide sub-10ms responses for cache hits, dramatically reducing average latency for applications with query clustering.
Real-time conversational interface where any perceptible delay is unacceptable
Hybrid architecture with fast retrieval and optional LLM enhancement
Showing retrieved results immediately while LLM processing continues in background provides instant feedback while preserving LLM quality for complex queries.
Batch processing workload where individual request latency doesn't matter
Throughput-optimized configuration with large batch sizes
Maximizing throughput rather than minimizing latency can reduce cost per request by 5-10x, appropriate when users aren't waiting for real-time responses.
Edge deployment with strict latency requirements and limited connectivity
On-device small language models (SLMs)
Models under 3B parameters can run on mobile devices and edge hardware with sub-second latency, eliminating network round-trip and providing offline capability.
Application where output quality matters more than generation speed
Larger models with streaming UI and appropriate user expectations
Rather than compromising quality for speed, designing the user experience around streaming output and setting appropriate expectations may better serve user needs.
High-frequency, low-complexity queries that don't require reasoning
Retrieval-augmented generation with minimal LLM involvement
Using LLMs only for query understanding and response synthesis while retrieving pre-computed content can reduce LLM token generation requirements by 80-90%.
Development and testing environments where production latency isn't required
Shared inference endpoints with higher latency tolerance
Accepting higher latency in non-production environments enables significant cost savings through better resource utilization and smaller infrastructure.
Common Mistakes
(10)Measuring only end-to-end latency without breaking down TTFT, generation time, and network latency, making it impossible to identify the actual bottleneck.
Assuming that GPU utilization percentage indicates efficient inference, when low utilization during generation is expected due to memory bandwidth constraints.
Implementing streaming without understanding that it improves perceived latency but not total response time, leading to incorrect capacity planning.
Using average latency as the primary metric when p95 or p99 latency often determines user experience and SLA compliance.
Optimizing for single-request latency in high-throughput scenarios where batching efficiency matters more for overall system performance.
Applying optimizations developed for one model architecture to different architectures without validating effectiveness.
Ignoring the memory capacity constraints when implementing optimizations that trade memory for compute.
Failing to account for KV cache growth when testing with short sequences and then experiencing memory issues with production-length contexts.
Comparing latency across different output lengths without normalizing for tokens generated.
Implementing complex custom solutions before fully utilizing built-in optimizations in inference frameworks.
Core Taxonomy
Core Taxonomy
Primary Types
(8 types)The dominant constraint for most LLM inference workloads, where the time to load model weights from GPU memory exceeds the time to perform computations on those weights.
Characteristics
- GPU compute utilization below 50% during token generation
- Latency scales linearly with model parameter count
- Minimal improvement from faster compute hardware
- Significant improvement from higher memory bandwidth GPUs
Use Cases
Tradeoffs
Addressing memory bandwidth constraints typically requires hardware upgrades, model compression, or architectural changes that may affect quality or increase complexity.
Classification Dimensions
Latency Phase
Different phases of LLM response generation have different bottlenecks and optimization strategies. TTFT is compute-bound during prompt processing, while ITL is memory-bound during generation.
Bottleneck Type
The primary resource constraint limiting performance determines which optimizations will be effective. Most LLM inference is memory bandwidth-bound during generation.
Optimization Target
Different optimization goals require different strategies and involve different tradeoffs. Optimizing for one dimension typically compromises others.
Deployment Context
The acceptable latency range varies dramatically by use case, from sub-second for chat to hours for batch processing, affecting optimization priorities.
Hardware Configuration
Hardware topology affects both achievable latency and the dominant bottlenecks. Multi-GPU configurations introduce communication overhead but can reduce per-token latency.
Evolutionary Stages
Naive Implementation
Initial prototype phase, typically 1-2 weeks of developmentSingle-request processing without batching, no KV caching, naive attention implementation. Characterized by very low throughput and high per-token latency, but simple to implement and debug.
Basic Optimization
Early production deployment, 2-4 weeks of optimizationKV caching enabled, static batching implemented, basic quantization applied. Provides 5-10x throughput improvement over naive implementation with moderate implementation effort.
Production Optimization
Mature production system, 1-3 months of optimizationContinuous batching, Flash Attention, optimized memory management, dynamic batching based on load. Achieves near-optimal throughput for given hardware with significant engineering investment.
Advanced Optimization
High-scale production requiring dedicated ML infrastructure team, 3-6 monthsSpeculative decoding, custom CUDA kernels, multi-GPU tensor parallelism, KV cache compression. Pushes hardware utilization to practical limits with substantial engineering and maintenance overhead.
Frontier Optimization
Cutting-edge deployments at major AI companies, 6-12+ months with dedicated research teamCustom hardware integration, novel attention mechanisms, research-grade optimizations. Achieves performance beyond standard frameworks but requires deep expertise and ongoing research investment.
Architecture Patterns
Architecture Patterns
Architecture Patterns
(7 patterns)Streaming Response Architecture
Architecture that delivers tokens to users as they are generated rather than waiting for complete response, improving perceived latency without reducing actual generation time.
Components
- Token generation engine with streaming output
- Server-sent events (SSE) or WebSocket connection
- Client-side incremental rendering
- Backpressure handling for slow clients
Data Flow
User request → Load balancer → Inference server → Token-by-token streaming → SSE/WebSocket → Client incremental display
Best For
- Interactive chat applications
- Real-time content generation
- Applications where perceived latency matters more than total time
Limitations
- Doesn't reduce actual generation time
- Requires client-side handling of partial responses
- Connection management overhead at scale
- Complicates error handling and retry logic
Scaling Characteristics
Scales horizontally with inference servers. Connection management becomes bottleneck at high concurrency. Requires sticky sessions or connection pooling.
Integration Points
Load Balancer
Distributes requests across inference servers, manages connection pooling for streaming, and implements health checking for GPU instances.
Must handle long-lived connections for streaming. Health checks should verify GPU availability, not just process liveness. Session affinity may be needed for stateful conversations.
Request Queue
Buffers incoming requests, implements priority scheduling, and manages backpressure when inference capacity is exceeded.
Queue depth directly affects latency. Priority implementation affects fairness. Must handle request timeouts and cancellations gracefully.
KV Cache Manager
Allocates and manages GPU memory for KV caches, implements eviction policies, and handles memory fragmentation.
Memory fragmentation can significantly reduce effective capacity. Eviction policy affects both latency and throughput. Must coordinate with batch scheduler.
Monitoring System
Collects latency metrics, GPU utilization, memory usage, and queue depths for observability and alerting.
Must capture per-phase latency (TTFT, ITL) not just total latency. GPU metrics require NVML or similar. High-cardinality metrics for per-request tracking.
Model Registry
Manages model versions, handles model loading and unloading, and coordinates model updates across inference servers.
Model loading time affects cold start latency. Version rollback must be fast. A/B testing requires routing coordination.
Rate Limiter
Enforces usage quotas, prevents resource exhaustion, and implements fair scheduling across users or tenants.
Rate limiting at token level more accurate than request level. Must account for variable request costs. Graceful degradation preferred over hard rejection.
Caching Layer
Stores computed results for semantic caching, manages prefix caches, and handles cache invalidation.
Semantic similarity threshold affects cache hit rate and quality. Cache size must balance hit rate against memory cost. Invalidation strategy for model updates.
Autoscaler
Adjusts inference capacity based on load, manages GPU instance lifecycle, and optimizes cost-performance tradeoffs.
GPU instance startup time (minutes) affects scaling responsiveness. Scaling metrics should include queue depth, not just utilization. Cost optimization requires predictive scaling.
Decision Framework
Decision Framework
Focus on latency optimization: smaller models, tensor parallelism, speculative decoding
Focus on throughput optimization: larger batches, continuous batching, cost-efficient hardware
Most production systems need to balance both, but understanding the primary constraint guides initial optimization focus.
Technical Deep Dive
Technical Deep Dive
Overview
LLM inference consists of two distinct phases with different computational characteristics. The prefill phase processes the input prompt in parallel, computing attention over all input tokens simultaneously. This phase is compute-bound and benefits from GPU parallelism. The decode phase generates output tokens one at a time, with each token requiring a full forward pass through the model. This phase is memory-bound because the model weights must be loaded from GPU memory for each token, and the computation per weight load is minimal. The fundamental bottleneck during decoding is the ratio between memory bandwidth and compute capacity. Modern GPUs like the A100 have approximately 2 TB/s of memory bandwidth and 312 TFLOPS of compute capacity. For a 70B parameter model in FP16 (140GB), loading all weights takes approximately 70ms. The actual computation on those weights takes only a few milliseconds. This means the GPU spends most of its time waiting for data, achieving less than 5% of theoretical compute utilization during token generation. The attention mechanism adds additional complexity. For each generated token, the model must attend to all previous tokens in the sequence. Without optimization, this requires O(n²) computation and memory for a sequence of length n. The KV cache optimization stores the key and value projections from previous tokens, reducing per-token attention computation to O(n), but the cache itself grows linearly with sequence length, consuming GPU memory that could otherwise be used for batching. Batching multiple requests together amortizes the memory bandwidth cost across requests, improving throughput. However, batching increases latency for individual requests because each request must wait for the slowest request in the batch. Continuous batching mitigates this by allowing requests to join and leave the batch dynamically, but adds significant implementation complexity.
Step-by-Step Process
The input text is received and converted to token IDs using the model's tokenizer. This includes handling special tokens, applying chat templates if applicable, and validating input length against model limits.
Tokenization can be slow for very long inputs. Different tokenizers produce different token counts for the same text, affecting latency predictions. Unicode handling edge cases can cause unexpected behavior.
Under The Hood
At the hardware level, LLM inference is dominated by matrix-vector multiplications during the decode phase. Each layer of the transformer requires multiplying the model weights (a large matrix) by the current token's activations (a vector). For a 70B parameter model, this involves moving approximately 140GB of weight data through the memory hierarchy for each token generated. The GPU's memory bandwidth of 2-3 TB/s means this takes 50-70ms per token, regardless of the GPU's compute capability. The memory hierarchy plays a crucial role in performance. GPU HBM (High Bandwidth Memory) provides the bulk storage for model weights and KV cache. L2 cache (40-50MB on modern GPUs) can hold frequently accessed data. Registers and shared memory provide the fastest access but are limited in size. Optimized inference engines carefully manage data placement to maximize cache utilization, but the fundamental bandwidth limitation remains. The attention mechanism's memory access pattern is particularly challenging. During decode, the query vector for the new token must be compared against all cached key vectors. This requires reading the entire key cache, which grows linearly with sequence length. For a 100K token context with a typical model, the key cache alone can be several gigabytes, requiring significant memory bandwidth just for attention computation. Quantization reduces memory bandwidth requirements by storing weights in lower precision formats. INT8 quantization halves the memory traffic compared to FP16, while INT4 quarters it. However, quantization introduces approximation errors that accumulate through the network. Modern quantization techniques like GPTQ and AWQ use calibration data to minimize quality impact, but some degradation is unavoidable. The speedup from quantization depends on hardware support; GPUs with dedicated INT8 or INT4 tensor cores see larger benefits. Speculative decoding attempts to break the sequential dependency by using a smaller draft model to propose multiple tokens, then verifying them in parallel with the target model. When the draft model's predictions match the target model's, multiple tokens are accepted in a single target model call. The effectiveness depends on the draft model's accuracy, which varies by domain and prompt. Typical speedups are 2-3x, with diminishing returns for longer speculation depths due to decreasing acceptance probability.
Failure Modes
Failure Modes
KV cache growth exceeds available GPU memory as sequence length increases or batch size is too large for the context length.
- CUDA OOM errors mid-generation
- Requests failing after partial completion
- Memory usage climbing steadily during long conversations
Request failure, potential service restart, lost generation progress, user-visible errors.
Implement memory budgeting that reserves space for maximum expected sequence length. Use paged attention to reduce fragmentation. Set conservative batch size limits based on worst-case context length.
Implement graceful degradation that reduces batch size or truncates context when memory pressure is detected. Use memory monitoring to trigger preemptive actions before OOM.
Operational Considerations
Operational Considerations
Key Metrics (15)
Latency from request receipt to first token generation, measuring prompt processing time
Dashboard Panels
Alerting Strategy
Implement tiered alerting with different severity levels. P1 alerts for error rate spikes and complete outages require immediate response. P2 alerts for SLA violations and capacity warnings require response within 15 minutes. P3 alerts for efficiency degradation and trend anomalies require investigation within 4 hours. Use anomaly detection for metrics with variable baselines. Implement alert correlation to reduce noise from cascading issues.
Cost Analysis
Cost Analysis
Cost Drivers
(10)GPU Instance Hours
Primary cost driver, typically 70-90% of total inference cost. High-end GPUs (A100, H100) cost $2-8/hour.
Maximize GPU utilization through batching. Use spot instances for fault-tolerant workloads. Right-size instances for workload. Implement autoscaling to avoid idle capacity.
Model Size
Larger models require more expensive GPUs with more memory. 70B models require multiple GPUs, multiplying costs.
Use smallest model that meets quality requirements. Implement model routing to use smaller models for simple queries. Consider fine-tuned smaller models for specific tasks.
Output Token Count
Each output token requires a forward pass. Longer outputs directly increase compute time and cost.
Implement output length limits. Use concise prompting techniques. Consider summarization for verbose outputs.
Input Token Count
Longer inputs increase prefill time and KV cache memory. Very long contexts may require more expensive configurations.
Implement prompt compression. Use retrieval to include only relevant context. Summarize conversation history.
Batch Efficiency
Low batch utilization wastes GPU capacity. Padding in static batching wastes compute.
Implement continuous batching. Use request bucketing by length. Optimize batch size for workload.
Latency Requirements
Stricter latency requirements limit batching, reducing throughput and increasing cost per token.
Set realistic latency SLAs. Use streaming to relax total latency requirements. Implement tiered latency for different use cases.
Redundancy and Availability
High availability requires redundant capacity, potentially doubling or tripling costs.
Right-size redundancy for actual availability requirements. Use spot instances for non-critical redundancy. Implement graceful degradation.
Data Transfer
Network egress costs for streaming responses and cross-region traffic can be significant at scale.
Deploy in regions close to users. Use compression for responses. Implement CDN for cacheable content.
Storage
Model storage, logging, and caching require persistent storage. Large models require fast storage for acceptable load times.
Use tiered storage. Implement log rotation. Share model storage across instances.
Monitoring and Observability
Detailed metrics and logging generate significant data volumes with associated storage and processing costs.
Sample metrics appropriately. Implement log levels. Use aggregation to reduce cardinality.
Cost Models
Token-Based Pricing
Cost = (input_tokens × input_price) + (output_tokens × output_price)1000 input tokens at $0.003/1K + 500 output tokens at $0.006/1K = $0.003 + $0.003 = $0.006 per request
Compute-Time Pricing
Cost = GPU_hours × hourly_rate / requests_per_hour1 A100 hour at $4/hour processing 1000 requests = $0.004 per request
Throughput-Based Cost
Cost_per_token = hourly_rate / (tokens_per_second × 3600)A100 at $4/hour achieving 2000 tokens/s = $4 / 7,200,000 = $0.00000056 per token
Total Cost of Ownership
TCO = compute_cost + storage_cost + network_cost + operations_cost + redundancy_overheadMonthly: $10K compute + $500 storage + $200 network + $2K operations + $5K redundancy = $17.7K
Optimization Strategies
- 1Implement continuous batching to maximize GPU utilization and reduce cost per token
- 2Use quantization (INT8/INT4) to enable larger batch sizes or smaller GPUs
- 3Deploy smaller models for simple queries with routing based on complexity
- 4Implement semantic caching to avoid redundant generation for similar queries
- 5Use spot instances for batch workloads and fault-tolerant applications
- 6Right-size GPU instances based on actual memory and compute requirements
- 7Implement autoscaling to match capacity to demand and avoid idle resources
- 8Use prefix caching to reduce prefill costs for repeated prompt patterns
- 9Optimize prompt engineering to reduce input token counts
- 10Implement output length limits to control generation costs
- 11Use reserved instances or committed use discounts for baseline capacity
- 12Deploy in regions with lower GPU pricing when latency permits
Hidden Costs
- 💰Engineering time for optimization and maintenance, often exceeding infrastructure costs for small deployments
- 💰Quality assurance and testing costs for model updates and configuration changes
- 💰Incident response and on-call costs for production LLM services
- 💰Training and knowledge transfer costs as LLM infrastructure evolves rapidly
- 💰Technical debt from quick optimizations that complicate future changes
- 💰Opportunity cost of engineering resources focused on optimization rather than features
- 💰Compliance and security audit costs for regulated industries
- 💰Vendor lock-in costs if using proprietary optimization tools or frameworks
ROI Considerations
ROI analysis for LLM latency optimization must consider both direct cost savings and indirect benefits. Direct savings come from reduced infrastructure costs through better utilization and right-sizing. Indirect benefits include improved user experience leading to higher engagement and retention, reduced timeout-related errors improving reliability, and competitive advantage from faster response times. The investment required for optimization varies significantly by approach. Framework-level optimizations (batching configuration, quantization) typically require days to weeks of engineering time with low risk. Custom optimizations (speculative decoding, custom kernels) require months of specialized engineering with higher risk but potentially larger benefits. Hardware upgrades require capital investment but provide immediate, predictable improvements. Break-even analysis should consider the scale of deployment. Optimizations that save 10% on a $100K/month deployment save $10K/month, potentially justifying significant engineering investment. The same optimization on a $1K/month deployment saves only $100/month, rarely justifying more than configuration tuning. Risk factors include the rapid evolution of LLM infrastructure, where today's optimizations may be obsoleted by framework improvements or new hardware. Organizations should balance optimization investment against the likelihood of fundamental changes in the near term.
Security Considerations
Security Considerations
Threat Model
(10 threats)Denial of Service via Long Inputs
Attacker sends extremely long prompts to consume GPU memory and compute, preventing legitimate requests from being processed.
Service degradation or outage, increased costs, poor experience for legitimate users.
Implement strict input length limits. Use request validation before GPU processing. Implement per-user rate limiting. Monitor for anomalous input patterns.
Resource Exhaustion via Output Length
Crafted prompts that cause model to generate extremely long outputs, consuming resources disproportionate to input.
Increased costs, resource exhaustion, potential service degradation.
Implement output length limits. Use token budgets per request. Monitor output length distribution for anomalies.
Cache Poisoning
Attacker crafts inputs to populate cache with malicious or incorrect responses that are served to other users.
Incorrect or harmful responses served to legitimate users, reputation damage.
Implement cache isolation between users where appropriate. Use cryptographic cache keys. Validate cached responses before serving.
Model Extraction via Timing
Attacker uses latency measurements to infer model architecture or parameters through timing side channels.
Intellectual property theft, competitive disadvantage.
Add random delays to responses. Use constant-time operations where feasible. Monitor for systematic probing patterns.
Prompt Injection via Cached Context
Attacker injects malicious instructions into shared prefix cache that affect subsequent users.
Unauthorized actions, data leakage, harmful outputs.
Isolate cache by user or tenant. Validate cache contents. Implement strict prompt boundaries.
GPU Memory Snooping
In multi-tenant GPU environments, attacker attempts to read other users' data from GPU memory.
Data leakage, privacy violations.
Use dedicated GPU instances for sensitive workloads. Clear GPU memory between requests. Use secure multi-tenancy features where available.
Model Weight Theft
Attacker gains access to model weights through compromised infrastructure or insider threat.
Intellectual property theft, competitive disadvantage, potential misuse of model.
Encrypt model weights at rest and in transit. Implement strict access controls. Monitor for unauthorized access attempts.
Inference Log Exposure
Attacker accesses inference logs containing user prompts and model outputs.
Privacy violations, data leakage, compliance violations.
Minimize logging of sensitive content. Encrypt logs. Implement strict access controls. Use log anonymization.
Supply Chain Attack on Model
Compromised model weights or inference framework introduces backdoors or vulnerabilities.
Arbitrary code execution, data theft, model manipulation.
Verify model checksums from trusted sources. Audit inference framework dependencies. Use reproducible builds.
API Key Theft and Abuse
Stolen API credentials used to access service, potentially incurring costs or accessing data.
Financial loss, data access, service abuse.
Implement key rotation. Use short-lived tokens. Monitor for anomalous usage patterns. Implement IP allowlisting where appropriate.
Security Best Practices
- ✓Implement input validation and sanitization before GPU processing
- ✓Use strict rate limiting per user, IP, and API key
- ✓Encrypt all data in transit using TLS 1.3
- ✓Encrypt model weights and sensitive data at rest
- ✓Implement comprehensive audit logging for security events
- ✓Use dedicated GPU instances for sensitive workloads
- ✓Implement network segmentation for inference infrastructure
- ✓Regularly update inference frameworks and dependencies
- ✓Conduct security audits and penetration testing
- ✓Implement incident response procedures for security events
- ✓Use secrets management for API keys and credentials
- ✓Implement least-privilege access controls
- ✓Monitor for anomalous usage patterns
- ✓Implement request signing for API authentication
- ✓Use secure multi-tenancy practices for shared infrastructure
Data Protection
- 🔒Implement data classification to identify sensitive content in prompts and outputs
- 🔒Use tokenization or masking for sensitive data before LLM processing
- 🔒Implement retention policies that minimize storage of sensitive data
- 🔒Provide mechanisms for users to delete their data
- 🔒Use encryption for all stored prompts and outputs
- 🔒Implement access controls limiting who can view inference logs
- 🔒Consider on-premises deployment for highly sensitive workloads
- 🔒Implement data loss prevention monitoring for outputs
- 🔒Use anonymization techniques for analytics and debugging
- 🔒Document data flows and processing activities
Compliance Implications
GDPR
Personal data in prompts and outputs must be handled according to GDPR requirements including right to deletion and data minimization.
Implement data retention policies. Provide mechanisms for data deletion. Minimize logging of personal data. Implement consent mechanisms.
HIPAA
Protected health information in medical LLM applications must meet HIPAA security and privacy requirements.
Use HIPAA-compliant infrastructure. Implement BAAs with cloud providers. Encrypt PHI at rest and in transit. Implement audit logging.
SOC 2
Service organization controls for security, availability, processing integrity, confidentiality, and privacy.
Implement security controls. Document procedures. Conduct regular audits. Maintain evidence of compliance.
PCI DSS
Payment card data must not be processed or stored in LLM systems without PCI compliance.
Exclude payment data from prompts. Implement data masking. Use PCI-compliant infrastructure if payment data is necessary.
CCPA
California consumer privacy rights including right to know, delete, and opt-out of data sale.
Implement data inventory. Provide consumer rights mechanisms. Document data practices.
AI Act (EU)
High-risk AI systems must meet requirements for transparency, human oversight, and risk management.
Document model capabilities and limitations. Implement human oversight mechanisms. Conduct risk assessments.
Export Controls
Advanced AI models may be subject to export control regulations limiting deployment in certain jurisdictions.
Understand applicable export controls. Implement geographic restrictions where required. Document compliance.
Industry-Specific Regulations
Financial services, healthcare, and other regulated industries have specific requirements for AI systems.
Identify applicable regulations. Implement required controls. Engage compliance and legal teams.
Scaling Guide
Scaling Guide
Scaling Dimensions
Request Volume
Horizontal scaling with additional inference instances behind load balancer. Use autoscaling based on queue depth or latency metrics.
Limited by available GPU capacity and budget. Network and load balancer can become bottlenecks at very high scale.
Ensure load balancer can handle connection volume for streaming. Consider regional deployment for geographic distribution.
Throughput (Tokens/Second)
Optimize batch size and implement continuous batching. Use multiple GPUs per instance for larger batches.
Memory bandwidth limits per-GPU throughput. Diminishing returns from batching beyond memory capacity.
Throughput optimization often conflicts with latency optimization. Choose based on workload requirements.
Latency
Use tensor parallelism across GPUs to reduce per-token latency. Implement speculative decoding. Use smaller models.
Fundamental limit from autoregressive generation. Communication overhead limits multi-GPU scaling.
Latency optimization increases cost per token. Streaming can improve perceived latency without reducing actual latency.
Context Length
Use efficient attention implementations (Flash Attention). Implement KV cache compression. Use models designed for long context.
Memory grows linearly with context. Attention computation grows quadratically without optimization.
Long context support requires careful memory management. Quality may degrade for very long contexts.
Model Size
Use tensor or pipeline parallelism across multiple GPUs. Implement quantization to reduce memory requirements.
Communication overhead for distributed inference. Cost scales with GPU count.
Larger models generally provide better quality but higher latency and cost.
Concurrent Users
Scale inference capacity and connection handling. Implement efficient queuing and batching.
Connection limits, memory for per-user state, queue management overhead.
Concurrent users with long conversations require significant KV cache memory.
Geographic Distribution
Deploy inference capacity in multiple regions close to users. Use global load balancing.
Model synchronization across regions. Operational complexity of multi-region deployment.
Reduces network latency but increases operational complexity and cost.
Multi-Tenancy
Implement tenant isolation at request level. Use resource quotas and fair scheduling.
Isolation overhead. Noisy neighbor effects in shared infrastructure.
Balance isolation requirements against efficiency of shared infrastructure.
Capacity Planning
Required_GPUs = (requests_per_second × avg_tokens_per_request) / (tokens_per_second_per_GPU × target_utilization) × redundancy_factorPlan for 30-50% headroom above expected peak load. Include redundancy for availability requirements (typically N+1 or 2N for high availability). Account for traffic growth with 3-6 month projection.
Scaling Milestones
- Basic batching configuration
- Initial monitoring setup
- Manual scaling sufficient
Single GPU instance with basic batching. Simple load balancer. Manual deployment.
- Efficient batching becomes critical
- Need for continuous batching
- Autoscaling required
Multiple GPU instances with continuous batching. Autoscaling based on queue depth. Comprehensive monitoring.
- Memory management at scale
- Request routing complexity
- Operational overhead
Multi-region deployment. Sophisticated load balancing. Dedicated operations team. Advanced caching strategies.
- Infrastructure cost optimization critical
- Complex multi-region architecture
- Custom optimizations may be needed
Global deployment with regional routing. Custom inference optimizations. Dedicated infrastructure team. Sophisticated cost management.
- Frontier-scale infrastructure
- Custom hardware considerations
- Research-level optimizations
Custom inference infrastructure. Potential custom hardware. Dedicated ML infrastructure organization. Industry-leading optimization techniques.
Benchmarks
Benchmarks
Industry Benchmarks
| Metric | P50 | P95 | P99 | World Class |
|---|---|---|---|---|
| Time to First Token (7B model) | 150ms | 300ms | 500ms | <100ms |
| Time to First Token (70B model) | 500ms | 1000ms | 2000ms | <300ms |
| Inter-Token Latency (7B model) | 25ms | 40ms | 60ms | <20ms |
| Inter-Token Latency (70B model) | 50ms | 80ms | 120ms | <40ms |
| Throughput (7B model, A100) | 2000 tokens/s | 1500 tokens/s | 1000 tokens/s | >3000 tokens/s |
| Throughput (70B model, 8xA100) | 1000 tokens/s | 750 tokens/s | 500 tokens/s | >1500 tokens/s |
| GPU Memory Utilization | 75% | 85% | 90% | >85% sustained |
| Batch Efficiency | 60% | 75% | 85% | >80% |
| Cost per 1M tokens (7B model) | $0.50 | $0.30 | $0.20 | <$0.15 |
| Cost per 1M tokens (70B model) | $2.00 | $1.50 | $1.00 | <$0.75 |
| Speculative Decoding Speedup | 1.5x | 2.0x | 2.5x | >2.5x |
| Prefix Cache Hit Rate | 30% | 50% | 70% | >60% |
Comparison Matrix
| Framework | Throughput | Latency | Ease of Use | Features | Maturity | Community |
|---|---|---|---|---|---|---|
| vLLM | Excellent | Excellent | Good | Excellent | Good | Excellent |
| TensorRT-LLM | Excellent | Excellent | Moderate | Excellent | Good | Good |
| Text Generation Inference | Good | Good | Excellent | Good | Excellent | Good |
| llama.cpp | Moderate | Good | Good | Moderate | Good | Excellent |
| DeepSpeed-Inference | Good | Good | Moderate | Good | Good | Moderate |
| Native PyTorch | Poor | Poor | Excellent | Basic | Excellent | Excellent |
Performance Tiers
Single GPU, static batching, no optimization. Suitable for development and low-volume production.
ITL < 100ms, throughput > 500 tokens/s, cost < $5/1M tokens
Continuous batching, quantization, basic caching. Suitable for medium-volume production.
ITL < 50ms, throughput > 2000 tokens/s, cost < $1/1M tokens
Full optimization stack, multi-GPU, advanced caching. Suitable for high-volume production.
ITL < 30ms, throughput > 5000 tokens/s, cost < $0.50/1M tokens
Custom optimizations, speculative decoding, global deployment. Suitable for large-scale enterprise.
ITL < 20ms, throughput > 10000 tokens/s, cost < $0.25/1M tokens
Frontier optimizations, custom hardware integration, research-level techniques.
ITL < 15ms, throughput > 20000 tokens/s, cost < $0.15/1M tokens
Real World Examples
Real World Examples
Real-World Scenarios
(8 examples)E-commerce Product Description Generation
Large e-commerce platform generating product descriptions for millions of items. Batch processing acceptable, cost is primary concern.
Deployed 7B model with INT4 quantization on spot instances. Used large batch sizes (64+) with continuous batching. Implemented semantic caching for similar products.
Achieved $0.20 per 1000 descriptions, 10x cost reduction from initial deployment. Latency of 5-10 seconds acceptable for batch workflow.
- 💡Batch workloads can achieve much lower costs than real-time
- 💡Semantic caching highly effective for similar inputs
- 💡Spot instances viable for fault-tolerant batch processing
- 💡Quality acceptable with aggressive quantization for this use case
Customer Support Chatbot
Financial services company deploying AI chatbot for customer support. Strict latency requirements (<3s total response), high accuracy required.
Used 70B model with tensor parallelism across 4 A100s. Implemented streaming responses. Deployed prefix caching for common greeting and context patterns.
Achieved P95 TTFT of 400ms, P95 total latency of 2.5s for typical responses. Quality met accuracy requirements. Cost higher but justified by customer satisfaction improvement.
- 💡Streaming essential for acceptable user experience
- 💡Prefix caching very effective for chat with common patterns
- 💡Larger models justified when accuracy is critical
- 💡Tensor parallelism effective for latency-sensitive large models
Code Completion IDE Integration
Developer tools company integrating LLM code completion into IDE. Sub-second latency critical for user experience.
Deployed 7B code-specialized model with speculative decoding using 1B draft model. Aggressive prefix caching for common code patterns. Edge deployment in major regions.
Achieved P95 TTFT of 150ms, P95 completion latency of 500ms. Speculative decoding provided 2.2x speedup for code completion.
- 💡Speculative decoding highly effective for predictable outputs like code
- 💡Specialized smaller models can match larger general models for specific tasks
- 💡Edge deployment critical for global user base
- 💡Sub-second latency achievable with right architecture
Document Analysis Pipeline
Legal tech company processing contracts for clause extraction. Documents up to 100K tokens. Throughput more important than latency.
Used long-context model with Flash Attention. Implemented chunking strategy for very long documents. Optimized for throughput with large batches.
Processed 10,000 documents per hour on 8 A100 cluster. Cost of $0.05 per document acceptable for value provided.
- 💡Long-context models require careful memory management
- 💡Chunking strategy critical for documents exceeding context limits
- 💡Throughput optimization dramatically reduces per-document cost
- 💡Flash Attention essential for long-context efficiency
Real-time Translation Service
Global communication platform providing real-time translation. Ultra-low latency required for conversational flow.
Deployed specialized translation models (3B parameters) in each major region. Used INT8 quantization. Implemented aggressive caching for common phrases.
Achieved P95 latency of 200ms for typical sentences. Regional deployment reduced network latency to <50ms.
- 💡Specialized smaller models can outperform general large models for specific tasks
- 💡Regional deployment essential for real-time applications
- 💡Phrase caching highly effective for translation
- 💡INT8 quantization minimal quality impact for translation
Content Moderation at Scale
Social media platform moderating user-generated content. Millions of items per day. Cost and throughput critical.
Fine-tuned small model (1B) for classification. Deployed on cost-effective GPUs (T4). Massive batching with continuous batching.
Achieved 100,000 items per hour per GPU at $0.001 per item. Accuracy comparable to larger models for this specific task.
- 💡Fine-tuned small models highly effective for classification tasks
- 💡Cost-effective GPUs viable for throughput-focused workloads
- 💡Classification doesn't require generation, enabling much higher throughput
- 💡Task-specific fine-tuning can dramatically reduce model size requirements
Interactive Story Generation
Gaming company generating dynamic narrative content. Creative quality important, some latency acceptable.
Used 70B model for quality. Implemented streaming with dramatic reveal timing. Cached common story elements and character descriptions.
Achieved engaging user experience with 3-5 second generation times. Streaming made wait feel intentional and dramatic.
- 💡Streaming can be designed into user experience positively
- 💡Creative applications benefit from larger models
- 💡Caching effective for repeated narrative elements
- 💡Latency tolerance varies significantly by application context
API Service for Multiple Customers
AI startup providing LLM API to multiple B2B customers. Need to balance cost, quality, and latency across diverse use cases.
Implemented tiered service with different model sizes. Used continuous batching with fair scheduling. Implemented per-customer rate limiting and monitoring.
Served 50+ customers with varying requirements on shared infrastructure. Achieved 70% GPU utilization average.
- 💡Multi-tenant serving requires careful fair scheduling
- 💡Tiered offerings allow customers to choose cost/quality tradeoff
- 💡Shared infrastructure more efficient than dedicated per-customer
- 💡Monitoring per-customer essential for SLA management
Industry Applications
Healthcare
Clinical documentation and medical coding assistance
HIPAA compliance requires dedicated infrastructure. Accuracy critical for medical decisions. Latency less critical for documentation workflows. Must handle medical terminology accurately.
Financial Services
Document analysis, customer service, and compliance checking
Regulatory requirements for audit trails. High accuracy required for financial decisions. May require on-premises deployment. Real-time requirements for customer service.
E-commerce
Product descriptions, customer support, and personalization
High volume requires cost optimization. Seasonal traffic spikes require elastic scaling. Multilingual support often required. Quality directly impacts conversion.
Legal
Contract analysis, legal research, and document drafting
Long documents require large context windows. Accuracy critical for legal implications. Confidentiality requirements may require private deployment. Citation accuracy important.
Education
Tutoring, content generation, and assessment
Age-appropriate content filtering required. Pedagogical accuracy important. Cost sensitivity for educational institutions. Accessibility requirements.
Media and Entertainment
Content creation, localization, and personalization
Creative quality important. Large-scale content generation for personalization. Multilingual requirements. Brand voice consistency.
Manufacturing
Technical documentation, quality analysis, and maintenance support
Domain-specific terminology. Integration with existing systems. May require edge deployment for factory floor. Safety-critical accuracy requirements.
Government
Citizen services, document processing, and analysis
Strict security and compliance requirements. Accessibility requirements. May require on-premises or government cloud deployment. Transparency and explainability important.
Telecommunications
Customer service, network analysis, and documentation
High volume customer interactions. Real-time requirements for support. Integration with existing CRM systems. Multilingual support for global operations.
Insurance
Claims processing, underwriting support, and customer service
Regulatory compliance requirements. Accuracy critical for claims decisions. Document processing for policy analysis. Fraud detection integration.
Frequently Asked Questions
Frequently Asked Questions
Frequently Asked Questions
(20 questions)fundamentals
LLM inference is memory-bound, not compute-bound. The GPU spends most of its time loading model weights from memory rather than performing calculations. A powerful GPU with high FLOPS but limited memory bandwidth won't significantly improve LLM inference speed. The autoregressive nature of generation also means each token must be generated sequentially, creating a fundamental latency floor regardless of compute power.
metrics
optimization
architecture
debugging
hardware
cost
operations
Glossary
Glossary
Glossary
(29 terms)Acceptance Rate
In speculative decoding, the percentage of draft model tokens accepted by the target model.
Context: Higher acceptance rates lead to greater speedup from speculative decoding.
Attention Mechanism
The core operation in transformers that allows each token to attend to all other tokens, computing weighted combinations of value vectors based on query-key similarity.
Context: Source of O(n²) complexity that becomes bottleneck for very long sequences.
Autoregressive Generation
The process of generating text one token at a time, where each token's probability depends on all previously generated tokens. This creates sequential dependency that limits parallelization.
Context: Fundamental to understanding why LLM generation cannot be arbitrarily parallelized.
Batch Size
The number of requests processed simultaneously in a single forward pass through the model.
Context: Larger batches improve throughput but increase individual request latency.
Compute-Bound
A computation where performance is limited by available compute capacity rather than memory bandwidth.
Context: LLM prefill phase is compute-bound, benefiting from GPU parallelism.
Context Window
The maximum number of tokens a model can process in a single inference, including both input and output.
Context: Larger context windows require more memory and may increase latency.
Continuous Batching
Dynamic batching technique that allows requests to join and leave batches during generation, improving GPU utilization compared to static batching.
Context: Essential optimization for production LLM serving with variable workloads.
Decode Phase
The phase where output tokens are generated one at a time, each requiring a forward pass through the model.
Context: Memory-bound phase that dominates total latency for longer outputs.
Draft Model
A smaller, faster model used in speculative decoding to propose tokens for verification by the target model.
Context: Draft model quality directly determines speculative decoding effectiveness.
Feed-Forward Network (FFN)
The dense layers in each transformer block that process each token independently after attention.
Context: Contains most of the model parameters and dominates memory bandwidth usage.
Flash Attention
Optimized attention implementation that reduces memory traffic by fusing operations and using tiling to keep intermediate results in fast SRAM.
Context: Standard optimization for efficient attention computation, especially for long sequences.
HBM (High Bandwidth Memory)
The high-speed memory used in modern GPUs, providing bandwidth of 1-3 TB/s.
Context: HBM bandwidth is the primary constraint for LLM inference speed.
Inter-Token Latency (ITL)
The average time between generating consecutive tokens during the decode phase. Determined by memory bandwidth and model size.
Context: Determines the 'typing speed' of streamed responses and total generation time.
KV Cache
Storage for key and value projections from previous tokens, enabling efficient attention computation without reprocessing the entire sequence for each new token.
Context: Critical for inference efficiency but consumes significant GPU memory.
Memory Bandwidth
The rate at which data can be transferred between GPU memory (HBM) and compute units, measured in GB/s or TB/s.
Context: Primary bottleneck for LLM inference during the decode phase.
Memory-Bound
A computation where performance is limited by memory bandwidth rather than compute capacity.
Context: LLM decode phase is memory-bound, explaining low GPU compute utilization.
Model Parallelism
Distributing a model across multiple devices, either by layers (pipeline parallelism) or within layers (tensor parallelism).
Context: Required for models too large to fit on a single GPU.
NVLink
NVIDIA's high-speed interconnect for GPU-to-GPU communication, providing much higher bandwidth than PCIe.
Context: Required for efficient tensor parallelism across multiple GPUs.
Paged Attention
Memory management technique for KV cache using virtual memory concepts, reducing fragmentation and enabling more efficient memory utilization.
Context: Enables larger effective batch sizes by reducing memory waste.
Prefill Phase
The initial phase of inference where the input prompt is processed in parallel, computing attention over all input tokens and populating the KV cache.
Context: Compute-bound phase that determines TTFT.
Prefix Caching
Caching KV states for common prompt prefixes (like system prompts) to avoid recomputation for requests with shared prefixes.
Context: Effective for reducing TTFT in applications with repeated prompt patterns.
Quantization
Reducing the precision of model weights and/or activations (e.g., FP16 to INT8) to reduce memory requirements and improve inference speed.
Context: Common optimization trading small quality reduction for significant speed improvement.
Semantic Caching
Caching LLM responses and retrieving them for semantically similar queries using embedding similarity.
Context: Can dramatically reduce latency and cost for applications with clustered query patterns.
Speculative Decoding
Optimization technique using a smaller draft model to propose multiple tokens that are verified in parallel by the target model.
Context: Can provide 2-3x speedup when draft model predictions are accurate.
Streaming
Delivering LLM output tokens to the client as they are generated rather than waiting for complete generation.
Context: Improves perceived latency without reducing actual generation time.
Tensor Parallelism
Distributing model layers across multiple GPUs to parallelize computation and memory bandwidth within each layer.
Context: Reduces per-token latency for large models but requires high-speed interconnects.
Throughput
The number of tokens generated per unit time across all requests, typically measured in tokens per second.
Context: Key metric for cost efficiency and capacity planning.
Time to First Token (TTFT)
The latency from receiving a request to generating the first output token. Primarily determined by prompt processing (prefill) time.
Context: Key metric for user-perceived responsiveness, especially with streaming.
Token
The basic unit of text processing in LLMs, typically representing a word, subword, or character depending on the tokenizer.
Context: All LLM latency and cost metrics are fundamentally per-token.
References & Resources
Academic Papers
- • Attention Is All You Need (Vaswani et al., 2017) - Foundational transformer architecture paper
- • FlashAttention: Fast and Memory-Efficient Exact Attention (Dao et al., 2022) - Efficient attention implementation
- • Fast Inference from Transformers via Speculative Decoding (Leviathan et al., 2023) - Speculative decoding technique
- • Efficient Memory Management for Large Language Model Serving with PagedAttention (Kwon et al., 2023) - vLLM and paged attention
- • LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (Dettmers et al., 2022) - Quantization for LLMs
- • GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (Frantar et al., 2023) - Advanced quantization
- • Scaling Laws for Neural Language Models (Kaplan et al., 2020) - Understanding model scaling
- • Training Compute-Optimal Large Language Models (Hoffmann et al., 2022) - Chinchilla scaling laws
Industry Standards
- • MLPerf Inference Benchmark - Industry standard for ML inference performance measurement
- • NVIDIA TensorRT-LLM Documentation - Best practices for optimized LLM inference
- • vLLM Documentation - Production LLM serving framework
- • Hugging Face Text Generation Inference - Open-source inference server
- • OpenAI API Documentation - Industry reference for LLM API design
- • NVIDIA CUDA Best Practices Guide - GPU programming optimization
Resources
- • NVIDIA Developer Blog - LLM Inference Optimization Series
- • Hugging Face Blog - Transformer Optimization Techniques
- • Anyscale Blog - Production LLM Serving
- • Modal Labs Blog - Serverless LLM Deployment
- • Together AI Blog - Inference Infrastructure
- • Databricks Blog - LLM Serving at Scale
- • Google Cloud Architecture Center - ML Inference Patterns
- • AWS Machine Learning Blog - SageMaker LLM Deployment
Continue Learning
Related concepts to deepen your understanding
Last updated: 2026-01-05 • Version: v1.0 • Status: citation-safe-reference
Keywords: LLM performance, inference speed, latency optimization, LLM bottlenecks