Sample article — starter content for NeuralSys.
Introduction
Inference is where AI meets physics: every token costs memory bandwidth, and every millisecond of latency shapes product design. You don't need to be a kernel engineer — but you need the mental model.
Prefill vs Decode
- Prefill (processing the prompt): compute-bound, parallelizable. This sets time to first token.
- Decode (generating tokens one by one): memory-bandwidth-bound. This sets tokens per second.
Long prompts hurt TTFT. Long answers hurt throughput. Design UX around both: stream early, keep answers tight.
The KV-Cache
The key-value cache stores attention state so the model doesn't recompute the whole prefix per token. Consequences:
- Memory grows with sequence length × batch size — the real serving limit.
- Prefix caching (shared system prompts) is nearly free throughput.
- Cache eviction policy is a product decision disguised as infra.
Batching, Quantization, Throughput
| Technique | What it buys | Cost |
|---|---|---|
| Continuous batching | 3–10× throughput | Scheduler complexity |
| INT8/FP8 quantization | ~2× memory headroom | Small quality delta, eval needed |
| Speculative decoding | Lower latency | Extra draft model |
| Prefix caching | Cheap shared prompts | Cache invalidation logic |
# Mental model for capacity planning
memory_per_request_gb = (
model_weights_gb
+ layers * seq_len * hidden * 2 * bytes_per_element / 1e9
)
max_concurrent = (gpu_memory_gb - model_weights_gb) // memory_per_request_gbKey Takeaways
- Optimize TTFT for interactivity, throughput for cost.
- The KV-cache is the scarcest resource — budget sequence length deliberately.
- Fewer, richer model calls beat many tiny ones in agentic systems.