Part 2: Scaling Translation Inference: +82% Throughput

​

Author

​

Recap: The Problem

• Queue serialization overhead
• GPU compute contention between worker processes
• Spiky GPU utilization pattern

​

Attempt 2: Static Batching

​

Implementation

# Within worker process
MAX_BATCH_SIZE = 16
BATCH_TIMEOUT = 0.05  # 50ms

while True:
    batch_keys = []
    batch_tasks = []

    # Collect first task (blocking)
    first_key = queue.get()
    batch_keys.append(first_key)
    batch_tasks.append(tasks[first_key])

    # Try to collect more tasks (non-blocking with timeout)
    batch_start = time.time()
    while len(batch_keys) < MAX_BATCH_SIZE:
        time_remaining = BATCH_TIMEOUT - (time.time() - batch_start)
        if time_remaining <= 0:
            break
        try:
            key = queue.get(timeout=time_remaining)
            batch_keys.append(key)
            batch_tasks.append(tasks[key])
        except Empty:
            break

    # Process batch using vLLM
    results = translation_provider.translate_batch(
        texts=[t.text for t in batch_tasks],
        source_langs=[t.source_lang for t in batch_tasks],
        target_langs=[t.target_lang for t in batch_tasks]
    )

• Batch size: 16 requests
• Timeout: 50ms (don’t wait indefinitely for full batch)
• vLLM processes multiple sequences together
• Still uses multiprocessing workers

​

Results

Figure 1: Static batching delivers significant throughput and response time improvements

​

Trade-offs

• Massive throughput gains
• GPU better utilized
• Simple implementation

• Head-of-line blocking: All requests wait for the slowest one
• With variable-length inputs, short translations wait for long ones
• Example: [50 tokens, 50 tokens, 200 tokens] – first two wait for the 200-token translation

​

Attempt 3: Continuous Batching

​

What is Continuous Batching?

• New requests join mid-generation
• Completed requests leave immediately (don’t wait for others)
• Batch composition updates every token
• vLLM’s AsyncLLMEngine handles this automatically

​

Implementation

from vllm import AsyncLLMEngine, EngineArgs

engine_args = EngineArgs(
    model=model_id,
    max_num_seqs=64,  # Initial attempt
    max_num_batched_tokens=16384,
    gpu_memory_utilization=0.3,
    enable_chunked_prefill=True,
)

engine = AsyncLLMEngine.from_engine_args(engine_args)

@app.post("/translate")
async def translate(request: TranslateRequest):
    result_generator = engine.generate(
        request.text,
        sampling_params,
        request_id=generate_id()
    )

    async for output in result_generator:
        final_output = output
    return TranslateResponse(translation=final_output.text)

• AsyncLLMEngine used directly in FastAPI
• vLLM handles batching internally via continuous batching engine
• Pure async/await throughout

​

Testing Reality Check

​

Initial Results (Uniform Inputs)

Figure 2: Continuous batching with uniform inputs showing impressive 15 RPS throughput

​

Variable-Length Inputs (Reality)

• Very heavy load: 1.1 RPS (vs 2.2 RPS with uniform)
• Even baseline performed worse with realistic data

• Very heavy load: 3.5 RPS (with max_num_seqs=16 tuning)
• Same configuration that gave us 15 RPS with uniform inputs

Figure 3: Performance gap between uniform test data and realistic variable-length inputs

​

Configuration Tuning

​

What We Found

# vLLM Prometheus metrics we monitored:
# - vllm:time_to_first_token_seconds (TTFT)
# - vllm:time_per_output_token_seconds (decode time)
# - vllm:gpu_cache_usage_perc (KV cache utilization)
# - vllm:num_requests_running / waiting (queue depth)

• Actual workload: 2-20 concurrent requests per server (production peak ~20 per server)
• Configuration: max_num_seqs=64
• Result: 60+ empty slots creating overhead

• KV cache pre-allocated for 64 sequences
• vLLM scheduler manages 64 slots but only uses 5-10
• Decode time per token increases
• Memory wasted on unused sequence slots
• Scheduler overhead for empty slots

​

Tuning Approach

1. Measure actual concurrent request distribution in production
2. Start with max_num_seqs=1, gradually increase: 2 → 4 → 8 → 16 → 32
3. Monitor decode time and tail latency at each step
4. Stop when performance degrades

​

Final Configuration

from translation_lib.config import AsyncVLLMTranslationProvider

provider = AsyncVLLMTranslationProvider(
    model_name=model_id,
    revision=model_revision,
    gpu_memory_utilization=0.3,  # ~10GB on RTX 5090
    max_num_seqs=16,  # Right-sized to actual workload per server
    huggingface_token=hf_token,
    supported_language_pairs=None,  # Multilingual model
)

await provider.initialize_engine()

​

Configuration Rationale

• Production peak: ~20 concurrent requests per server
• Testing: Validated up to 25 concurrent
• Provides headroom without wasting resources
• Scheduler overhead matched to actual load

• Reduced from default 16384
• Better suited for our average sequence lengths
• Reduces memory pressure

• Allocates ~10GB VRAM for model + KV cache on RTX 5090 (32GB)
• Tracked via vllm:gpu_cache_usage_perc
• Balanced for our configuration

Figure 4: Throughput progression through all optimization attempts

​

Production Results

​

Before vs After

Figure 5: Production deployment results showing 82% throughput improvement

Figure 6: P95 latency improvements across optimization attempts

Figure 7: Response time evolution with variable-length inputs

​

Summary

​

What Worked

• AsyncLLMEngine handles batching automatically
• No manual batch collection overhead
• Direct async/await integration with FastAPI

• max_num_seqs=16 (matched actual workload per server)
• Not 64 (theoretical max that created overhead)
• gpu_memory_utilization=0.3 for 10GB allocation

• Variable-length inputs exposed configuration issues
• Uniform test data gave misleading 15 RPS

• KV cache usage
• Decode time per token
• Queue depth
• Guided configuration decisions

​

Complete Journey