[Doc] Improve contributing and installation documentation (#9132)
Signed-off-by: Rafael Vasquez <rafvasq21@gmail.com>
This commit is contained in:
Rafael Vasquez
GitHub
parent
1874c6a1b0
commit
de24046fcd
@ -1,30 +1,23 @@
|
||||
# Contributing to vLLM
|
||||
|
||||
Thank you for your interest in contributing to vLLM!
|
||||
Our community is open to everyone and welcomes all kinds of contributions, no matter how small or large.
|
||||
There are several ways you can contribute to the project:
|
||||
Thank you for your interest in contributing to vLLM! Our community is open to everyone and welcomes all kinds of contributions, no matter how small or large. There are several ways you can contribute to the project:
|
||||
|
||||
- Identify and report any issues or bugs.
|
||||
- Request or add a new model.
|
||||
- Request or add support for a new model.
|
||||
- Suggest or implement new features.
|
||||
- Improve documentation or contribute a how-to guide.
|
||||
|
||||
However, remember that contributions aren't just about code.
|
||||
We believe in the power of community support; thus, answering queries, assisting others, and enhancing the documentation are highly regarded and beneficial contributions.
|
||||
We also believe in the power of community support; thus, answering queries, offering PR reviews, and assisting others are also highly regarded and beneficial contributions.
|
||||
|
||||
Finally, one of the most impactful ways to support us is by raising awareness about vLLM.
|
||||
Talk about it in your blog posts, highlighting how it's driving your incredible projects.
|
||||
Express your support on Twitter if vLLM aids you, or simply offer your appreciation by starring our repository.
|
||||
Finally, one of the most impactful ways to support us is by raising awareness about vLLM. Talk about it in your blog posts and highlight how it's driving your incredible projects. Express your support on social media if you're using vLLM, or simply offer your appreciation by starring our repository!
|
||||
|
||||
|
||||
## Setup for development
|
||||
## Developing
|
||||
|
||||
### Build from source
|
||||
Depending on the kind of development you'd like to do (e.g. Python, CUDA), you can choose to build vLLM with or without compilation. Check out the [building from source](https://docs.vllm.ai/en/latest/getting_started/installation.html#build-from-source) documentation for details.
|
||||
|
||||
```bash
|
||||
pip install -e . # This may take several minutes.
|
||||
```
|
||||
|
||||
### Testing
|
||||
## Testing
|
||||
|
||||
```bash
|
||||
pip install -r requirements-dev.txt
|
||||
@ -36,15 +29,16 @@ mypy
|
||||
# Unit tests
|
||||
pytest tests/
|
||||
```
|
||||
**Note:** Currently, the repository does not pass the mypy tests.
|
||||
**Note:** Currently, the repository does not pass the ``mypy`` tests.
|
||||
|
||||
## Contribution Guidelines
|
||||
|
||||
## Contributing Guidelines
|
||||
### Issues
|
||||
|
||||
### Issue Reporting
|
||||
If you encounter a bug or have a feature request, please [search existing issues](https://github.com/vllm-project/vllm/issues?q=is%3Aissue) first to see if it has already been reported. If not, please [file a new issue](https://github.com/vllm-project/vllm/issues/new/choose), providing as much relevant information as possible.
|
||||
|
||||
If you encounter a bug or have a feature request, please check our issues page first to see if someone else has already reported it.
|
||||
If not, please file a new issue, providing as much relevant information as possible.
|
||||
> [!IMPORTANT]
|
||||
> If you discover a security vulnerability, please follow the instructions [here](/SECURITY.md#reporting-a-vulnerability).
|
||||
|
||||
### Pull Requests & Code Reviews
|
||||
|
||||
@ -53,4 +47,4 @@ Please check the PR checklist in the [PR template](.github/PULL_REQUEST_TEMPLATE
|
||||
### Thank You
|
||||
|
||||
Finally, thank you for taking the time to read these guidelines and for your interest in contributing to vLLM.
|
||||
Your contributions make vLLM a great tool for everyone!
|
||||
All of your contributions help make vLLM a great tool and community for everyone!
|
||||
|
||||
@ -2,11 +2,10 @@
|
||||
|
||||
## Reporting a Vulnerability
|
||||
|
||||
If you believe you have found a security vulnerability in vLLM, we encourage you to let us know right away.
|
||||
We will investigate all legitimate reports and do our best to quickly fix the problem.
|
||||
If you believe you have found a security vulnerability in vLLM, we encourage you to let us know right away. We will investigate all legitimate reports and do our best to quickly fix the problem.
|
||||
|
||||
Please report security issues using https://github.com/vllm-project/vllm/security/advisories/new
|
||||
Please report security issues privately using [the vulnerability submission form](https://github.com/vllm-project/vllm/security/advisories/new).
|
||||
|
||||
---
|
||||
Please see PyTorch Security for more information how to securely interact with models: https://github.com/pytorch/pytorch/blob/main/SECURITY.md
|
||||
This document mostly references the recommendation from PyTorch, thank you!
|
||||
|
||||
Please see [PyTorch's Security Policy](https://github.com/pytorch/pytorch/blob/main/SECURITY.md) for more information and recommendations on how to securely interact with models.
|
||||
|
||||
@ -1,19 +1,20 @@
|
||||
.. _installation:
|
||||
|
||||
============
|
||||
Installation
|
||||
============
|
||||
|
||||
vLLM is a Python library that also contains pre-compiled C++ and CUDA (12.1) binaries.
|
||||
|
||||
Requirements
|
||||
------------
|
||||
===========================
|
||||
|
||||
* OS: Linux
|
||||
* Python: 3.8 -- 3.12
|
||||
* GPU: compute capability 7.0 or higher (e.g., V100, T4, RTX20xx, A100, L4, H100, etc.)
|
||||
|
||||
Install released versions
|
||||
--------------------------
|
||||
===========================
|
||||
|
||||
You can install vLLM using pip:
|
||||
|
||||
@ -46,8 +47,11 @@ You can install vLLM using pip:
|
||||
|
||||
Therefore, it is recommended to install vLLM with a **fresh new** conda environment. If either you have a different CUDA version or you want to use an existing PyTorch installation, you need to build vLLM from source. See below for instructions.
|
||||
|
||||
|
||||
.. _install-the-latest-code:
|
||||
|
||||
Install the latest code
|
||||
----------------------------
|
||||
=========================
|
||||
|
||||
LLM inference is a fast-evolving field, and the latest code may contain bug fixes, performance improvements, and new features that are not released yet. To allow users to try the latest code without waiting for the next release, vLLM provides wheels for Linux running on x86 platform with cuda 12 for every commit since v0.5.3. You can download and install the latest one with the following command:
|
||||
|
||||
@ -75,18 +79,25 @@ These docker images are used for CI and testing only, and they are not intended
|
||||
|
||||
Latest code can contain bugs and may not be stable. Please use it with caution.
|
||||
|
||||
Build from source (without compilation)
|
||||
---------------------------------------
|
||||
.. _build_from_source:
|
||||
|
||||
If you want to develop vLLM, and you only need to change the Python code, you can build vLLM without compilation.
|
||||
Build from source
|
||||
==================
|
||||
|
||||
The first step is to follow the previous instructions to install the latest vLLM wheel:
|
||||
Python-only build (without compilation)
|
||||
----------------------------------------
|
||||
|
||||
If you only need to change Python code, you can simply build vLLM without compilation.
|
||||
|
||||
The first step is to install the latest vLLM wheel:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
$ pip install https://vllm-wheels.s3.us-west-2.amazonaws.com/nightly/vllm-1.0.0.dev-cp38-abi3-manylinux1_x86_64.whl
|
||||
pip install https://vllm-wheels.s3.us-west-2.amazonaws.com/nightly/vllm-1.0.0.dev-cp38-abi3-manylinux1_x86_64.whl
|
||||
|
||||
After verifying that the installation is successful, we have a script for you to copy and link directories, so that you can edit the Python code directly:
|
||||
You can find more information about vLLM's wheels `above <#install-the-latest-code>`_.
|
||||
|
||||
After verifying that the installation is successful, you can use `the following script <https://github.com/vllm-project/vllm/blob/main/python_only_dev.py>`_:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
@ -94,94 +105,96 @@ After verifying that the installation is successful, we have a script for you to
|
||||
$ cd vllm
|
||||
$ python python_only_dev.py
|
||||
|
||||
It will:
|
||||
The script will:
|
||||
|
||||
- Find the installed vLLM in the current environment.
|
||||
- Copy built files to the current directory.
|
||||
- Rename the installed vLLM
|
||||
- Symbolically link the current directory to the installed vLLM.
|
||||
* Find the installed vLLM package in the current environment.
|
||||
* Copy built files to the current directory.
|
||||
* Rename the installed vLLM package.
|
||||
* Symbolically link the current directory to the installed vLLM package.
|
||||
|
||||
This way, you can edit the Python code in the current directory, and the changes will be reflected in the installed vLLM.
|
||||
Now, you can edit the Python code in the current directory, and the changes will be reflected when you run vLLM.
|
||||
|
||||
.. _build_from_source:
|
||||
|
||||
Build from source (with compilation)
|
||||
------------------------------------
|
||||
Full build (with compilation)
|
||||
---------------------------------
|
||||
|
||||
If you need to touch the C++ or CUDA code, you need to build vLLM from source:
|
||||
If you want to modify C++ or CUDA code, you'll need to build vLLM from source. This can take several minutes:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
$ git clone https://github.com/vllm-project/vllm.git
|
||||
$ cd vllm
|
||||
$ pip install -e . # This can take a long time
|
||||
|
||||
.. note::
|
||||
|
||||
This will uninstall existing PyTorch, and install the version required by vLLM. If you want to use an existing PyTorch installation, there need to be some changes:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
$ git clone https://github.com/vllm-project/vllm.git
|
||||
$ cd vllm
|
||||
$ python use_existing_torch.py
|
||||
$ pip install -r requirements-build.txt
|
||||
$ pip install -e . --no-build-isolation
|
||||
|
||||
The differences are:
|
||||
|
||||
- ``python use_existing_torch.py``: This script will remove all the PyTorch versions in the requirements files, so that the existing PyTorch installation will be used.
|
||||
- ``pip install -r requirements-build.txt``: You need to manually install the requirements for building vLLM.
|
||||
- ``pip install -e . --no-build-isolation``: You need to disable build isolation, so that the build system can use the existing PyTorch installation.
|
||||
|
||||
This is especially useful when the PyTorch dependency cannot be easily installed via pip, e.g.:
|
||||
|
||||
- build vLLM with PyTorch nightly or a custom PyTorch build.
|
||||
- build vLLM with aarch64 and cuda (GH200), where the PyTorch wheels are not available on PyPI. Currently, only PyTorch nightly has wheels for aarch64 with CUDA. You can run ``pip3 install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu124`` to install PyTorch nightly, and then build vLLM on top of it.
|
||||
|
||||
.. note::
|
||||
|
||||
vLLM can fully run only on Linux, but you can still build it on other systems (for example, macOS). This build is only for development purposes, allowing for imports and a more convenient dev environment. The binaries will not be compiled and not work on non-Linux systems. You can create such a build with the following commands:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
$ export VLLM_TARGET_DEVICE=empty
|
||||
$ pip install -e .
|
||||
|
||||
$ pip install -e .
|
||||
|
||||
.. tip::
|
||||
|
||||
Building from source requires quite a lot compilation. If you are building from source for multiple times, it is beneficial to cache the compilation results. For example, you can install `ccache <https://github.com/ccache/ccache>`_ via either ``conda install ccache`` or ``apt install ccache`` . As long as ``which ccache`` command can find the ``ccache`` binary, it will be used automatically by the build system. After the first build, the subsequent builds will be much faster.
|
||||
Building from source requires a lot of compilation. If you are building from source repeatedly, it's more efficient to cache the compilation results.
|
||||
For example, you can install `ccache <https://github.com/ccache/ccache>`_ using ``conda install ccache`` or ``apt install ccache`` .
|
||||
As long as ``which ccache`` command can find the ``ccache`` binary, it will be used automatically by the build system. After the first build, subsequent builds will be much faster.
|
||||
|
||||
.. tip::
|
||||
To avoid your system being overloaded, you can limit the number of compilation jobs
|
||||
to be run simultaneously, via the environment variable ``MAX_JOBS``. For example:
|
||||
|
||||
.. code-block:: console
|
||||
Use an existing PyTorch installation
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
There are scenarios where the PyTorch dependency cannot be easily installed via pip, e.g.:
|
||||
|
||||
$ export MAX_JOBS=6
|
||||
$ pip install -e .
|
||||
* Building vLLM with PyTorch nightly or a custom PyTorch build.
|
||||
* Building vLLM with aarch64 and CUDA (GH200), where the PyTorch wheels are not available on PyPI. Currently, only the PyTorch nightly has wheels for aarch64 with CUDA. You can run ``pip3 install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu124`` to `install PyTorch nightly <https://pytorch.org/get-started/locally/>`_, and then build vLLM on top of it.
|
||||
|
||||
This is especially useful when you are building on less powerful machines. For example, when you use WSL, it only `gives you half of the memory by default <https://learn.microsoft.com/en-us/windows/wsl/wsl-config>`_, and you'd better use ``export MAX_JOBS=1`` to avoid compiling multiple files simultaneously and running out of memory. The side effect is that the build process will be much slower. If you only touch the Python code, slow compilation is okay, as you are building in an editable mode: you can just change the code and run the Python script without any re-compilation or re-installation.
|
||||
To build vLLM using an existing PyTorch installation:
|
||||
|
||||
.. tip::
|
||||
If you have trouble building vLLM, we recommend using the NVIDIA PyTorch Docker image.
|
||||
.. code-block:: console
|
||||
|
||||
.. code-block:: console
|
||||
$ git clone https://github.com/vllm-project/vllm.git
|
||||
$ cd vllm
|
||||
$ python use_existing_torch.py
|
||||
$ pip install -r requirements-build.txt
|
||||
$ pip install -e . --no-build-isolation
|
||||
|
||||
$ # Use `--ipc=host` to make sure the shared memory is large enough.
|
||||
$ docker run --gpus all -it --rm --ipc=host nvcr.io/nvidia/pytorch:23.10-py3
|
||||
|
||||
If you don't want to use docker, it is recommended to have a full installation of CUDA Toolkit. You can download and install it from `the official website <https://developer.nvidia.com/cuda-toolkit-archive>`_. After installation, set the environment variable ``CUDA_HOME`` to the installation path of CUDA Toolkit, and make sure that the ``nvcc`` compiler is in your ``PATH``, e.g.:
|
||||
Troubleshooting
|
||||
~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. code-block:: console
|
||||
To avoid your system being overloaded, you can limit the number of compilation jobs
|
||||
to be run simultaneously, via the environment variable ``MAX_JOBS``. For example:
|
||||
|
||||
$ export CUDA_HOME=/usr/local/cuda
|
||||
$ export PATH="${CUDA_HOME}/bin:$PATH"
|
||||
.. code-block:: console
|
||||
|
||||
Here is a sanity check to verify that the CUDA Toolkit is correctly installed:
|
||||
$ export MAX_JOBS=6
|
||||
$ pip install -e .
|
||||
|
||||
.. code-block:: console
|
||||
This is especially useful when you are building on less powerful machines. For example, when you use WSL it only `assigns 50% of the total memory by default <https://learn.microsoft.com/en-us/windows/wsl/wsl-config#main-wsl-settings>`_, so using ``export MAX_JOBS=1`` can avoid compiling multiple files simultaneously and running out of memory.
|
||||
A side effect is a much slower build process.
|
||||
|
||||
$ nvcc --version # verify that nvcc is in your PATH
|
||||
$ ${CUDA_HOME}/bin/nvcc --version # verify that nvcc is in your CUDA_HOME
|
||||
Additionally, if you have trouble building vLLM, we recommend using the NVIDIA PyTorch Docker image.
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
$ # Use `--ipc=host` to make sure the shared memory is large enough.
|
||||
$ docker run --gpus all -it --rm --ipc=host nvcr.io/nvidia/pytorch:23.10-py3
|
||||
|
||||
If you don't want to use docker, it is recommended to have a full installation of CUDA Toolkit. You can download and install it from `the official website <https://developer.nvidia.com/cuda-toolkit-archive>`_. After installation, set the environment variable ``CUDA_HOME`` to the installation path of CUDA Toolkit, and make sure that the ``nvcc`` compiler is in your ``PATH``, e.g.:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
$ export CUDA_HOME=/usr/local/cuda
|
||||
$ export PATH="${CUDA_HOME}/bin:$PATH"
|
||||
|
||||
Here is a sanity check to verify that the CUDA Toolkit is correctly installed:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
$ nvcc --version # verify that nvcc is in your PATH
|
||||
$ ${CUDA_HOME}/bin/nvcc --version # verify that nvcc is in your CUDA_HOME
|
||||
|
||||
|
||||
Unsupported OS build
|
||||
----------------------
|
||||
|
||||
vLLM can fully run only on Linux but for development purposes, you can still build it on other systems (for example, macOS), allowing for imports and a more convenient development environment. The binaries will not be compiled and won't work on non-Linux systems.
|
||||
|
||||
Simply disable the ``VLLM_TARGET_DEVICE`` environment variable before installing:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
$ export VLLM_TARGET_DEVICE=empty
|
||||
$ pip install -e .
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user