Sub-Millisecond Inference for
Continuous Agents
You push data into a session and LayerScale reads it right away, in the background. When you ask, the model has already read it, and any question you registered in advance already has its answer waiting.
An Engine Written From Scratch
We wrote the engine ourselves instead of wrapping a framework, which is what lets it keep your data on the GPU between requests and skip the work it has already done.
Reasons Between Requests
Native sessions runs queries on idle GPU cycles. New data gets added as it arrives, and a question picks up from there.
Same Model, Same Quality
We run the model you load exactly as it was trained. No sparse attention, no state-space stand-in, no shortcut that trades accuracy for speed. The speed comes from not repeating work.
Fewer layers between
your request and the GPU
Most serving stacks put a framework such as PyTorch between your request and the GPU. We wrote LayerScale from scratch, and its server, scheduler and GPU kernels are compiled into one program, so a request goes from our code straight onto the GPU with no framework in between.
The layers each engine walks on every request, from the serving layer at the top to the GPU at the bottom.
Built for Data That
Keeps Arriving
Say you are watching a sensor feed. Each reading you push goes into a session on the GPU and gets read as it arrives. When you ask what happened in the last hour, the model has already read it and only your question is new.
It handles ordinary request-and-response traffic too. The scheduler mixes many requests, streams and sessions on the same GPUs to keep them busy, and a shared prefix, such as a system prompt, is computed once and reused.
What People Use It For
Agents that loop, feeds that never stop, questions asked again after every update. The longer the work runs, the more it pays not to start over.
Coding Agents
A coding agent resends its whole transcript on every step, and most engines take all of it in again each time. LayerScale keeps each conversation in a session instead: the transcript stays on the GPU, each turn pays only for the tokens it adds, and LayerScale starts reading your prompt while you are still typing. Between turns it keeps working: it checks whether the agent is looping or has hit an error, and keeps a running summary current. Every agent and SDK below speaks the OpenAI Chat Completions or Responses API or the Anthropic Messages API to the server.
- opencode
- Cursor
- GitHub Copilot
- OpenAI Agents SDK
- Vercel AI SDK
- Codex
- Claude Code
- Gemini CLI
Agent Loops
An agent calls one tool after another. In a session each step picks up where the last one ended, sub-agents start from a shared prompt that is already loaded, and many agents share the same GPUs.
Streaming Data
A trading desk pushes ticks and candles into a session as they arrive. The model reads each update as it arrives and re-answers the questions you registered in advance (Flash Queries) after every batch. Reading the latest answer takes no model work, because it is already there.
Long-Running Sessions
A support bot in hour three of a conversation still has every message on the GPU, up to the context length you set, so its next reply does not start by re-reading the first two hours. Research and document assistants work the same way.
Flash Queries
Register your questions once. After every data update, LayerScale answers them again in the background and streams the fresh answers to you over SSE, so reading one is just a lookup. Learn more →