8 lines
514 B
ReStructuredText
8 lines
514 B
ReStructuredText
|
.. _deploying_with_bentoml:
|
||
|
|
|
||
|
|
Deploying with BentoML
|
||
|
|
======================
|
||
|
|
|
||
|
|
`BentoML <https://github.com/bentoml/BentoML>`_ allows you to deploy a large language model (LLM) server with vLLM as the backend, which exposes OpenAI-compatible endpoints. You can serve the model locally or containerize it as an OCI-complicant image and deploy it on Kubernetes.
|
||
|
|
|
||
|
|
For details, see the tutorial `vLLM inference in the BentoML documentation <https://docs.bentoml.com/en/latest/use-cases/large-language-models/vllm.html>`_.
|