20231088/vllm
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

[V1][Log] Add max request concurrency log to V1 (#12569)

Browse Source 
Signed-off-by: mgoin <michael@neuralmagic.com>
This commit is contained in:
Michael Goin 2025-01-30 18:07:19 -05:00 committed by GitHub
parent bd2107e30a
commit 4078052f09
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 4 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
vllm/v1/core/kv_cache_utils.py
View File

@ -393,6 +393,10 @@ def _get_kv_cache_config_uniform_type(vllm_config: VllmConfig,
num_blocks = num_gpu_blocks_override
logger.info("# GPU blocks: %d", num_blocks)
max_concurrency = (num_blocks * vllm_config.cache_config.block_size /
vllm_config.model_config.max_model_len)
logger.info("Maximum concurrency for %s tokens per request: %.2fx",
vllm_config.model_config.max_model_len, max_concurrency)
per_layer_size = page_size * num_blocks