SGLang
| SGLang | |
|---|---|
 | |
| Developer | LMSYS |
| Release | January 17, 2024 |
| Written in | Python, Rust, CUDA, C++ |
| Type | Large language model inference engine |
| License | Apache License 2.0 |
| Website | sglang |
| Repository | github |
SGLang (short for Structured Generation Language) is an open-source framework for programming and serving large language models and multimodal models. It was introduced by researchers affiliated with LMSYS and other institutions as a system combining a Python-embedded language for structured generation with a runtime for high-throughput inference.
The project is designed for low latency and high-throughput inference workloads, and its documentation describes support for features such as structured outputs, speculative decoding, continuous batching, quantization, and compatibility with OpenAI-style APIs.