vllm/docs/source/design/multimodal/multimodal_index.md

(multi-modality)=

Multi-Modality

.. currentmodule:: vllm.multimodal

vLLM provides experimental support for multi-modal models through the {mod}vllm.multimodal package.

Multi-modal inputs can be passed alongside text and token prompts to supported models via the multi_modal_data field in {class}vllm.inputs.PromptType.

Currently, vLLM only has built-in support for image data. You can extend vLLM to process additional modalities by following this guide.

Looking to add your own multi-modal model? Please follow the instructions listed here.

Guides

:maxdepth: 1

adding_multimodal_plugin

Module Contents

.. automodule:: vllm.multimodal

Registry

.. autodata:: vllm.multimodal.MULTIMODAL_REGISTRY

.. autoclass:: vllm.multimodal.MultiModalRegistry
    :members:
    :show-inheritance:

Base Classes

.. automodule:: vllm.multimodal.base
    :members:
    :show-inheritance:

Input Classes

.. automodule:: vllm.multimodal.inputs
    :members:
    :show-inheritance:

Audio Classes

.. automodule:: vllm.multimodal.audio
    :members:
    :show-inheritance:

Image Classes

.. automodule:: vllm.multimodal.image
    :members:
    :show-inheritance:

Video Classes

.. automodule:: vllm.multimodal.video
    :members:
    :show-inheritance: