162 lines
10 KiB
Markdown
162 lines
10 KiB
Markdown
<p align="center">
|
||
<picture>
|
||
<source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/vllm-project/vllm/main/docs/source/assets/logos/vllm-logo-text-dark.png">
|
||
<img alt="vLLM" src="https://raw.githubusercontent.com/vllm-project/vllm/main/docs/source/assets/logos/vllm-logo-text-light.png" width=55%>
|
||
</picture>
|
||
</p>
|
||
|
||
<h3 align="center">
|
||
Easy, fast, and cheap LLM serving for everyone
|
||
</h3>
|
||
|
||
<p align="center">
|
||
| <a href="https://docs.vllm.ai"><b>Documentation</b></a> | <a href="https://vllm.ai"><b>Blog</b></a> | <a href="https://arxiv.org/abs/2309.06180"><b>Paper</b></a> | <a href="https://x.com/vllm_project"><b>Twitter/X</b></a> | <a href="https://slack.vllm.ai"><b>Developer Slack</b></a> |
|
||
</p>
|
||
|
||
---
|
||
|
||
We’re excited to invite you to the first **vLLM China Meetup** on **March 16** in **Beijing**!
|
||
|
||
Join us to connect with the **vLLM team** and explore how vLLM is leveraged in **post-training, fine-tuning, and deployment**, including [verl](https://github.com/volcengine/verl), [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory), and [vllm-ascend](https://github.com/vllm-project/vllm-ascend).
|
||
|
||
👉 **[Register Now](https://mp.weixin.qq.com/s/n77GibL2corAtQHtVEAzfg)** to be part of the discussion!
|
||
|
||
---
|
||
|
||
*Latest News* 🔥
|
||
|
||
- [2024/02] We hosted [the ninth vLLM meetup](https://lu.ma/h7g3kuj9) with Meta! Please find the meetup slides from vLLM team [here](https://docs.google.com/presentation/d/1jzC_PZVXrVNSFVCW-V4cFXb6pn7zZ2CyP_Flwo05aqg/edit?usp=sharing) and AMD [here](https://drive.google.com/file/d/1Zk5qEJIkTmlQ2eQcXQZlljAx3m9s7nwn/view?usp=sharing). The slides from Meta will not be posted.
|
||
- [2025/01] We are excited to announce the alpha release of vLLM V1: A major architectural upgrade with 1.7x speedup! Clean code, optimized execution loop, zero-overhead prefix caching, enhanced multimodal support, and more. Please check out our blog post [here](https://blog.vllm.ai/2025/01/27/v1-alpha-release.html).
|
||
- [2025/01] We hosted [the eighth vLLM meetup](https://lu.ma/zep56hui) with Google Cloud! Please find the meetup slides from vLLM team [here](https://docs.google.com/presentation/d/1epVkt4Zu8Jz_S5OhEHPc798emsYh2BwYfRuDDVEF7u4/edit?usp=sharing), and Google Cloud team [here](https://drive.google.com/file/d/1h24pHewANyRL11xy5dXUbvRC9F9Kkjix/view?usp=sharing).
|
||
- [2024/12] vLLM joins [pytorch ecosystem](https://pytorch.org/blog/vllm-joins-pytorch)! Easy, Fast, and Cheap LLM Serving for Everyone!
|
||
- [2024/11] We hosted [the seventh vLLM meetup](https://lu.ma/h0qvrajz) with Snowflake! Please find the meetup slides from vLLM team [here](https://docs.google.com/presentation/d/1e3CxQBV3JsfGp30SwyvS3eM_tW-ghOhJ9PAJGK6KR54/edit?usp=sharing), and Snowflake team [here](https://docs.google.com/presentation/d/1qF3RkDAbOULwz9WK5TOltt2fE9t6uIc_hVNLFAaQX6A/edit?usp=sharing).
|
||
- [2024/10] We have just created a developer slack ([slack.vllm.ai](https://slack.vllm.ai)) focusing on coordinating contributions and discussing features. Please feel free to join us there!
|
||
- [2024/10] Ray Summit 2024 held a special track for vLLM! Please find the opening talk slides from the vLLM team [here](https://docs.google.com/presentation/d/1B_KQxpHBTRa_mDF-tR6i8rWdOU5QoTZNcEg2MKZxEHM/edit?usp=sharing). Learn more from the [talks](https://www.youtube.com/playlist?list=PLzTswPQNepXl6AQwifuwUImLPFRVpksjR) from other vLLM contributors and users!
|
||
- [2024/09] We hosted [the sixth vLLM meetup](https://lu.ma/87q3nvnh) with NVIDIA! Please find the meetup slides [here](https://docs.google.com/presentation/d/1wrLGwytQfaOTd5wCGSPNhoaW3nq0E-9wqyP7ny93xRs/edit?usp=sharing).
|
||
- [2024/07] We hosted [the fifth vLLM meetup](https://lu.ma/lp0gyjqr) with AWS! Please find the meetup slides [here](https://docs.google.com/presentation/d/1RgUD8aCfcHocghoP3zmXzck9vX3RCI9yfUAB2Bbcl4Y/edit?usp=sharing).
|
||
- [2024/07] In partnership with Meta, vLLM officially supports Llama 3.1 with FP8 quantization and pipeline parallelism! Please check out our blog post [here](https://blog.vllm.ai/2024/07/23/llama31.html).
|
||
- [2024/06] We hosted [the fourth vLLM meetup](https://lu.ma/agivllm) with Cloudflare and BentoML! Please find the meetup slides [here](https://docs.google.com/presentation/d/1iJ8o7V2bQEi0BFEljLTwc5G1S10_Rhv3beed5oB0NJ4/edit?usp=sharing).
|
||
- [2024/04] We hosted [the third vLLM meetup](https://robloxandvllmmeetup2024.splashthat.com/) with Roblox! Please find the meetup slides [here](https://docs.google.com/presentation/d/1A--47JAK4BJ39t954HyTkvtfwn0fkqtsL8NGFuslReM/edit?usp=sharing).
|
||
- [2024/01] We hosted [the second vLLM meetup](https://lu.ma/ygxbpzhl) with IBM! Please find the meetup slides [here](https://docs.google.com/presentation/d/12mI2sKABnUw5RBWXDYY-HtHth4iMSNcEoQ10jDQbxgA/edit?usp=sharing).
|
||
- [2023/10] We hosted [the first vLLM meetup](https://lu.ma/first-vllm-meetup) with a16z! Please find the meetup slides [here](https://docs.google.com/presentation/d/1QL-XPFXiFpDBh86DbEegFXBXFXjix4v032GhShbKf3s/edit?usp=sharing).
|
||
- [2023/08] We would like to express our sincere gratitude to [Andreessen Horowitz](https://a16z.com/2023/08/30/supporting-the-open-source-ai-community/) (a16z) for providing a generous grant to support the open-source development and research of vLLM.
|
||
- [2023/06] We officially released vLLM! FastChat-vLLM integration has powered [LMSYS Vicuna and Chatbot Arena](https://chat.lmsys.org) since mid-April. Check out our [blog post](https://vllm.ai).
|
||
|
||
---
|
||
|
||
## About
|
||
|
||
vLLM is a fast and easy-to-use library for LLM inference and serving.
|
||
|
||
Originally developed in the [Sky Computing Lab](https://sky.cs.berkeley.edu) at UC Berkeley, vLLM has evolved into a community-driven project with contributions from both academia and industry.
|
||
|
||
vLLM is fast with:
|
||
|
||
- State-of-the-art serving throughput
|
||
- Efficient management of attention key and value memory with [**PagedAttention**](https://blog.vllm.ai/2023/06/20/vllm.html)
|
||
- Continuous batching of incoming requests
|
||
- Fast model execution with CUDA/HIP graph
|
||
- Quantizations: [GPTQ](https://arxiv.org/abs/2210.17323), [AWQ](https://arxiv.org/abs/2306.00978), INT4, INT8, and FP8.
|
||
- Optimized CUDA kernels, including integration with FlashAttention and FlashInfer.
|
||
- Speculative decoding
|
||
- Chunked prefill
|
||
|
||
**Performance benchmark**: We include a performance benchmark at the end of [our blog post](https://blog.vllm.ai/2024/09/05/perf-update.html). It compares the performance of vLLM against other LLM serving engines ([TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM), [SGLang](https://github.com/sgl-project/sglang) and [LMDeploy](https://github.com/InternLM/lmdeploy)). The implementation is under [nightly-benchmarks folder](.buildkite/nightly-benchmarks/) and you can [reproduce](https://github.com/vllm-project/vllm/issues/8176) this benchmark using our one-click runnable script.
|
||
|
||
vLLM is flexible and easy to use with:
|
||
|
||
- Seamless integration with popular Hugging Face models
|
||
- High-throughput serving with various decoding algorithms, including *parallel sampling*, *beam search*, and more
|
||
- Tensor parallelism and pipeline parallelism support for distributed inference
|
||
- Streaming outputs
|
||
- OpenAI-compatible API server
|
||
- Support NVIDIA GPUs, AMD CPUs and GPUs, Intel CPUs and GPUs, PowerPC CPUs, TPU, and AWS Neuron.
|
||
- Prefix caching support
|
||
- Multi-lora support
|
||
|
||
vLLM seamlessly supports most popular open-source models on HuggingFace, including:
|
||
- Transformer-like LLMs (e.g., Llama)
|
||
- Mixture-of-Expert LLMs (e.g., Mixtral, Deepseek-V2 and V3)
|
||
- Embedding Models (e.g. E5-Mistral)
|
||
- Multi-modal LLMs (e.g., LLaVA)
|
||
|
||
Find the full list of supported models [here](https://docs.vllm.ai/en/latest/models/supported_models.html).
|
||
|
||
## Getting Started
|
||
|
||
Install vLLM with `pip` or [from source](https://docs.vllm.ai/en/latest/getting_started/installation/gpu/index.html#build-wheel-from-source):
|
||
|
||
```bash
|
||
pip install vllm
|
||
```
|
||
|
||
Visit our [documentation](https://docs.vllm.ai/en/latest/) to learn more.
|
||
- [Installation](https://docs.vllm.ai/en/latest/getting_started/installation/index.html)
|
||
- [Quickstart](https://docs.vllm.ai/en/latest/getting_started/quickstart.html)
|
||
- [List of Supported Models](https://docs.vllm.ai/en/latest/models/supported_models.html)
|
||
|
||
## Contributing
|
||
|
||
We welcome and value any contributions and collaborations.
|
||
Please check out [CONTRIBUTING.md](./CONTRIBUTING.md) for how to get involved.
|
||
|
||
## Sponsors
|
||
|
||
vLLM is a community project. Our compute resources for development and testing are supported by the following organizations. Thank you for your support!
|
||
|
||
<!-- Note: Please sort them in alphabetical order. -->
|
||
<!-- Note: Please keep these consistent with docs/source/community/sponsors.md -->
|
||
Cash Donations:
|
||
- a16z
|
||
- Dropbox
|
||
- Sequoia Capital
|
||
- Skywork AI
|
||
- ZhenFund
|
||
|
||
Compute Resources:
|
||
- AMD
|
||
- Anyscale
|
||
- AWS
|
||
- Crusoe Cloud
|
||
- Databricks
|
||
- DeepInfra
|
||
- Google Cloud
|
||
- Lambda Lab
|
||
- Nebius
|
||
- Novita AI
|
||
- NVIDIA
|
||
- Replicate
|
||
- Roblox
|
||
- RunPod
|
||
- Trainy
|
||
- UC Berkeley
|
||
- UC San Diego
|
||
|
||
Slack Sponsor: Anyscale
|
||
|
||
We also have an official fundraising venue through [OpenCollective](https://opencollective.com/vllm). We plan to use the fund to support the development, maintenance, and adoption of vLLM.
|
||
|
||
## Citation
|
||
|
||
If you use vLLM for your research, please cite our [paper](https://arxiv.org/abs/2309.06180):
|
||
|
||
```bibtex
|
||
@inproceedings{kwon2023efficient,
|
||
title={Efficient Memory Management for Large Language Model Serving with PagedAttention},
|
||
author={Woosuk Kwon and Zhuohan Li and Siyuan Zhuang and Ying Sheng and Lianmin Zheng and Cody Hao Yu and Joseph E. Gonzalez and Hao Zhang and Ion Stoica},
|
||
booktitle={Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles},
|
||
year={2023}
|
||
}
|
||
```
|
||
|
||
## Contact Us
|
||
|
||
- For technical questions and feature requests, please use Github issues or discussions.
|
||
- For discussing with fellow users and coordinating contributions and development, please use Slack.
|
||
- For security disclosures, please use Github's security advisory feature.
|
||
- For collaborations and partnerships, please contact us at vllm-questions AT lists.berkeley.edu.
|
||
|
||
## Media Kit
|
||
|
||
- If you wish to use vLLM's logo, please refer to [our media kit repo](https://github.com/vllm-project/media-kit).
|