PagedServe is a from-first-principles LLM serving architecture. It owns request state, continuous batching, paged KV memory, prefill and decode scheduling, token streaming, and the boundary between model execution and custom kernels.
Aggregate output tokens produced per second.
Time from request arrival to the first streamed token.
Inter-token latency after decode begins.
Unused capacity inside allocated cache pages.
Active sequences sustained under memory pressure.
A call to model.generate(...) hides the most interesting
serving problems. Real systems must decide which requests run at
each token step, how long prompts share the GPU with active decodes,
and how KV memory is allocated without reserving every sequence’s
maximum possible length.
PagedServe separates policy from mechanics so the scheduler, block allocator, model loop, sampling path, and metrics can each be tested and compared without outsourcing the runtime to another server.
The API creates request state, tokenizes the prompt, and places the sequence in the waiting queue with its generation limits.
Prompt tokens are processed in bounded chunks so one long prompt cannot monopolize an entire scheduling iteration.
Finished sequences leave immediately and newly eligible requests join later decode steps without waiting for a static batch.
Generated tokens stream to the client while completed or preempted sequences return KV blocks to the free pool.
Hugging Face provides model weights and tokenization, while the runtime owns batching, sequence state, KV writes, and sampling.
Fixed-size blocks, per-sequence block tables, reclamation, and pressure handling make memory use observable instead of implicit.
Serial, static batching, continuous batching, and external baselines use the same model, precision, prompts, and outputs.
Throughput is never read alone. A scheduler can produce more tokens by making individual users wait longer, so each experiment records throughput beside first-token latency, inter-token latency, request latency, KV occupancy, and concurrency. vLLM is an external baseline, while the primary comparison remains PagedServe’s own serial and static-batching paths.
Hire me!
You still here?
It’s over