The inference engine built for agents and streams.
Stateful inference for the next generation of AI.
Rethinking inference.
Process the stream, not the request.
LayerScale introduces a new computation model. Data is processed the moment it arrives, not when you query. By the time you ask a question, the answer is already there.
Built From the Ground Up
A new engine, not a serving layer on top of a framework. Fused kernels sit directly on the GPU, with no interpreter and no framework on the request path.
Stateful by Design
A session keeps its context resident on the GPU. New data is folded into that standing state as it arrives, and a query is answered from the state instead of rebuilding it.
Full Attention Preserved
Unlike sparse attention or state-space models that sacrifice model capacity for speed, LayerScale maintains full quadratic self-attention. No quality compromises.
Measured against vLLM and SGLang,
on the record
One run on two NVIDIA H100s, the identical open-weight checkpoint loaded on every engine, temperature zero. Six models, five stateless workloads and two stateful ones, and every figure read from a result file you can re-read yourself.
Fewer layers between
your request and the GPU
LayerScale is a new inference engine written from the ground up, not a serving layer on top of a framework. A request goes from the engine into fused kernels compiled into the engine and onto the GPU. Nothing is interpreted or dispatched through a framework on the way: the server, the scheduler and the kernels are one binary.
The layers each engine walks on every request, from the serving layer at the top to the GPU at the bottom.
Built for Streaming
from the Ground Up
LayerScale is stateful from the first line. A session keeps its context resident on the GPU, new data is folded into that standing state as it arrives, and a query is answered from the state rather than by rebuilding it. Custom fused kernels handle decode, the KV cache and sampling close to the metal, with no framework between the engine and the hardware.
The same engine serves conventional traffic: a continuous-batching scheduler admits many streams and sessions at once and keeps the GPUs busy, and shared prefixes are reused across requests instead of being recomputed.
Where LayerScale Excels
Agents that loop, feeds that never stop and questions that stay open. Work that keeps going is where a stateful engine pulls away.
Coding Agents
A coding agent resends its whole transcript on every step, and a conventional engine pays for all of it every time. Our OpenCode plugin gives each conversation a LayerScale session: the transcript stays resident on the GPU, a turn costs only the tokens it adds, the prompt is prefilled while you type, and a loop detector, an error watch and a running summary stay answered out of idle GPU time.
Agent Loops
Multi-step tool calling runs as stateful inference. Each step resumes where the last one ended, sub-agents start from a shared warm prefix, and continuous batching keeps many agents on the same GPUs.
Streaming Data
Market data never stops, and neither does the session. Ticks and candles are folded into the standing state as they arrive, Flash Queries are re-answered on every update, and reading an answer takes a fraction of a millisecond with no model work at all.
Long-Running Sessions
Support, research and document assistants keep their full context on the GPU for the life of the session. Nothing is re-read, and a sliding window keeps memory bounded however long it runs.
Run Any Model
on Any Hardware
Any open-weight transformer model, optimized and production-ready on NVIDIA CUDA and AMD ROCm.
Anthropic/OpenAI Compatible API
Drop-in replacement for existing workflows. Standard endpoints with streaming support, plus specialized session APIs for continuous data injection. Python and TypeScript client libraries available. Full API reference
Endpoints
- POST /v1/chat/completions OpenAI-compatible
- POST /v1/messages Anthropic-compatible
- POST /v1/sessions/init Create session
- POST /v1/sessions/{id}/stream/push Push data
- POST /v1/sessions/{id}/generate Query the session
- GET /v1/sessions/{id}/stream/status Stream stats
Streaming endpoints also available via WebSockets and Server-Sent Events (SSE) for low-latency persistent connections.
# Initialize a streaming session curl -X POST https://api.layerscale.ai/v1/sessions/init \ -H "Content-Type: application/json" \ -d '{"prompt": "You are a financial analyst..."}' # Push streaming data (non-blocking) curl -X POST https://api.layerscale.ai/v1/sessions/{id}/stream/push \ -H "Content-Type: application/json" \ -d '{"data": [{"o": 150.25, "h": 151.00, "l": 149.80, "c": 150.90, "v": 100000}]}' # Query the standing session curl -X POST https://api.layerscale.ai/v1/sessions/{id}/generate \ -H "Content-Type: application/json" \ -d '{"prompt": "What is the current trend?"}'
Flash Queries
Define your questions up front and get pre-computed answers after every data update, streamed back over SSE with no model work left on the query path. Learn more →
Run Anywhere
Docker Recommended
docker run --gpus all -p 8080:8080 \ -e HF_TOKEN=$HF_TOKEN \ layerscale/layerscale:latest \ --model meta-llama/Llama-3.1-8B-Instruct \ --license-key $LAYERSCALE_LICENSE_KEY
Don't have a key yet? Get a license key
Cloud API
# Use the hosted API directly curl -X POST https://api.layerscale.ai/v1/sessions/init \ -H "Authorization: Bearer $API_KEY" \ -H "Content-Type: application/json" \ -d '{"prompt": "You are a financial analyst..."}'
GPU Support
NVIDIA H100, H200, B200, B300 and Vera Rubin, AMD Instinct MI300X, MI325X, MI355X and MI455X. CUDA and ROCm.
Model Support
Any open-weight model. The Llama, Mistral, Qwen and Phi families work out of the box.
Platforms
Linux, in Docker, on cloud or on-premise GPU hosts.