VRAM Squeezing & Concurrency Optimization on a Single RTX 4090

Running a useful language model on one consumer GPU is less about chasing a single benchmark and more about balancing weights, KV cache, context length, and concurrent requests.

Start with a memory budget

Treat VRAM as a budget with four major consumers: model weights, runtime workspace, KV cache, and fragmentation. Leave a small safety margin before increasing concurrency.

budget = total_vram - model_weights - runtime_workspace - safety_margin
max_sequences = budget // kv_cache_per_sequence

Tune in the right order

1. Select a quantization format that your hardware executes efficiently.
2. Establish a realistic maximum context length.
3. Measure one-request latency before raising concurrency.
4. Increase batch tokens gradually and watch tail latency.

The best configuration is the one that remains predictable under your real request distribution—not the one with the highest isolated throughput.

Practical baseline

For a 24 GB card, start conservatively, reserve memory for runtime variance, and record both tokens per second and p95 latency. Re-run the same workload after every change.