The stateful, high-throughput inference engine for LLMs.
Agents and live feeds stay on the GPU between requests, and new data is read the moment it arrives.
Read the data when it arrives,
not when someone asks.
You push data into a session and LayerScale reads it right away, in the background. When you ask, the model has already read it, and any question you registered in advance already has its answer waiting.
An Engine Written From Scratch
We wrote the engine ourselves instead of wrapping a framework, which is what lets it keep your data on the GPU between requests and skip the work it has already done.
Remembers Between Requests
A session keeps a conversation or data feed on the GPU until you delete it. New data gets added as it arrives, and a question picks up from there.
Same Model, Same Quality
We run the model you load exactly as it was trained. No sparse attention, no state-space stand-in, no shortcut that trades accuracy for speed. The speed comes from not repeating work.
Measured against vLLM and SGLang,
on the record
We loaded the identical open-weight checkpoint into LayerScale, vLLM and SGLang on two NVIDIA H100s and ran them at temperature zero. Five models, five stateless workloads and two stateful ones. Every figure comes from a result file you can read yourself.
Fewer layers between
your request and the GPU
Most serving stacks put a framework such as PyTorch between your request and the GPU. We wrote LayerScale from scratch, and its server, scheduler and GPU kernels are compiled into one program, so a request goes from our code straight onto the GPU with no framework in between.
The layers each engine walks on every request, from the serving layer at the top to the GPU at the bottom.
Built for Data That
Keeps Arriving
Say you are watching a sensor feed. Each reading you push goes into a session on the GPU and gets read as it arrives. When you ask what happened in the last hour, the model has already read it and only your question is new.
It handles ordinary request-and-response traffic too. The scheduler mixes many requests, streams and sessions on the same GPUs to keep them busy, and a shared prefix, such as a system prompt, is computed once and reused.
What People Use It For
Agents that loop, feeds that never stop, questions asked again after every update. The longer the work runs, the more it pays not to start over.
Coding Agents
A coding agent resends its whole transcript on every step, and most engines take all of it in again each time. Our OpenCode plugin opens a LayerScale session per conversation instead: the transcript stays on the GPU, each turn pays only for the tokens it adds, and LayerScale starts reading your prompt while you are still typing. In idle GPU time it checks whether the agent is looping or has hit an error, and keeps a running summary current.
Agent Loops
An agent calls one tool after another. In a session each step picks up where the last one ended, sub-agents start from a shared prompt that is already loaded, and many agents share the same GPUs.
Streaming Data
A trading desk pushes ticks and candles into a session as they arrive. The model reads each update as it arrives and re-answers the questions you registered in advance (Flash Queries) after every batch. Reading the latest answer takes a fraction of a millisecond, because it is already there.
Long-Running Sessions
A support bot in hour three of a conversation still has every message on the GPU, up to the context length you set, so its next reply does not start by re-reading the first two hours. Research and document assistants work the same way.
Bring Your Own Model
and Your Own GPUs
Serve any open-weight transformer model, from Llama to Qwen, on NVIDIA CUDA or AMD ROCm GPUs.
Anthropic/OpenAI Compatible API
Keep your OpenAI or Anthropic code and point it at LayerScale, streaming included. Session endpoints let you push data in as it arrives. Python and TypeScript client libraries are available. Full API reference
Endpoints
- POST /v1/chat/completions OpenAI-compatible
- POST /v1/messages Anthropic-compatible
- POST /v1/sessions/init Create session
- POST /v1/sessions/{id}/stream/push Push data
- POST /v1/sessions/{id}/generate Query the session
- GET /v1/sessions/{id}/stream/status Stream stats
You can also stream over WebSockets or Server-Sent Events (SSE), which keep one connection open.
# Initialize a streaming session curl -X POST https://api.layerscale.ai/v1/sessions/init \ -H "Content-Type: application/json" \ -d '{"prompt": "You are a financial analyst..."}' # Push streaming data (non-blocking) curl -X POST https://api.layerscale.ai/v1/sessions/{id}/stream/push \ -H "Content-Type: application/json" \ -d '{"data": [{"o": 150.25, "h": 151.00, "l": 149.80, "c": 150.90, "v": 100000}]}' # Query the standing session curl -X POST https://api.layerscale.ai/v1/sessions/{id}/generate \ -H "Content-Type: application/json" \ -d '{"prompt": "What is the current trend?"}'
Flash Queries
Register your questions once. After every data update, LayerScale answers them again in the background and streams the fresh answers to you over SSE, so reading one is just a lookup. Learn more →
Run It Anywhere
Docker Recommended
docker run --gpus all -p 8080:8080 \ -e HF_TOKEN=$HF_TOKEN \ layerscale/layerscale:latest \ --model meta-llama/Llama-3.1-8B-Instruct \ --license-key $LAYERSCALE_LICENSE_KEY
Don't have a key yet? Get a license key
Cloud API
# Use the hosted API directly curl -X POST https://api.layerscale.ai/v1/sessions/init \ -H "Authorization: Bearer $API_KEY" \ -H "Content-Type: application/json" \ -d '{"prompt": "You are a financial analyst..."}'
NVIDIA Dynamo New
- Kubernetes. Deploy through the Dynamo operator with ready-made manifests, health probes and metrics included.
- Disaggregated serving. Prefill-only and decode-only workers, scaled independently, so long prompts never stall the decoding of other requests.
- Tensor parallelism. Each worker spans the GPUs in its pod with
--tp, whether it serves whole requests or runs as one half of a prefill/decode pair. - KV-aware routing. Every worker reports its prefix cache to the router page by page, so a request lands where its prompt is already cached.
Prefill and Decode, Two GPUs Each
docker run --gpus '"device=0,1"' layerscale/layerscale \ dynamo --model Qwen/Qwen3-8B --tp 2 \ --disaggregation-mode prefill docker run --gpus '"device=2,3"' layerscale/layerscale \ dynamo --model Qwen/Qwen3-8B --tp 2 \ --disaggregation-mode decode # Or on Kubernetes, with the Dynamo operator kubectl apply -f disagg.yaml
GPU Support
NVIDIA H100, H200, B200, B300 and Vera Rubin, AMD Instinct MI300X, MI325X, MI355X and MI455X. CUDA and ROCm.
Model Support
Any open-weight model. The Llama, Mistral, Qwen and Phi families work out of the box.
Platforms
Linux, in Docker, on cloud or on-premise GPU hosts. Kubernetes through NVIDIA Dynamo.