The stateful, high-throughput inference engine for LLMs.
Agents and live feeds stay on the GPU between requests, and new data is read the moment it arrives.
Built for Agents,
Not Single Requests
Sessions are not prefix caching. In vLLM and SGLang, the prefix cache only does something when a request arrives: it looks up what it still holds, and between requests nothing happens for that agent. LayerScale sessions are a novel design, unique to LayerScale. The agent's transcript stays on the GPU and the engine keeps working between requests, so each turn starts ahead and pays only for what it adds.
On every turn
The transcript is held in a session on the GPU. The turn picks up where the last one ended and reads only the new tokens.
vLLM and SGLangEach turn is a new request, matched against a cache and read again wherever the cache no longer has it.
When the GPU is busy
A live session is never pushed out to make room. Hour three of a coding session still starts where it left off.
vLLM and SGLangThe cache is shared and best effort. Under load a long transcript is pushed out and read again in full.
Between turns
In your coding agent or editor, it reads your prompt while you type, checks whether the agent is looping or stuck, and keeps a summary up to date.
vLLM and SGLangNothing happens until the next request.
On the way to the GPU
One program written from scratch: the server, the scheduler and our own GPU kernels, with no framework in between.
vLLM and SGLangPython, PyTorch and several libraries sit between the request and the GPU.
Works with the agent you already use
Claude Code, Codex, opencode and the rest speak the OpenAI or Anthropic API. Change the base URL and they run on LayerScale. See how it works →
- opencode
- Cursor
- GitHub Copilot
- OpenAI Agents SDK
- Vercel AI SDK
- Codex
- Claude Code
- Gemini CLI
Bring Your Own Model
and Your Own GPUs
Serve any open-weight transformer model, from Llama to Qwen, on NVIDIA and AMD GPUs or Apple silicon.
Made for Blackwell
and Vera Rubin
LayerScale is built for the GPUs that run AI in production. Our kernels are written for NVIDIA Blackwell and Vera Rubin, and Hopper cards such as the H100 and H200 run them too. The same image grows from one server to a fleet of them.
- Kernels for each generation. Our own kernels for Hopper and Blackwell, with FP8 on H100 and H200, and NVFP4 weights and KV cache on B200, B300, GB200 and GB300.
- One model across a server. Tensor parallelism splits a model over every GPU in the box, talking over NVLink.
- Prefill and decode apart. Disaggregated serving runs prompts and token generation on separate workers, each scaled on its own, so a long prompt never stalls anyone else's reply.
- A fleet behind one endpoint. Add workers behind NVIDIA Dynamo's KV-aware router and deploy them on Kubernetes with its operator. Each request goes to the worker most likely to hold its prompt already.