20231088/vllm
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

[Bugfix][TPU] Fix pad slot id (#5977)

Browse Source
This commit is contained in:
Woosuk Kwon 2024-06-28 18:55:17 -07:00 committed by GitHub
parent 54814fd85b
commit 7f83f40dee
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
vllm/worker/tpu_model_runner.py
View File

@ -19,7 +19,7 @@ from vllm.utils import make_tensor_with_pad
logger = init_logger(__name__)
_PAD_SLOT_ID = 0 # FIXME(woosuk)
_PAD_SLOT_ID = -1 # NOTE(woosuk): In PyTorch XLA, index -1 is ignored.
# FIXME(woosuk): Temporarily disabled top-p sampling since it's too slow.
_ENABLE_TOP_P = False
# FIXME(woosuk): A temporary hack to support `n > 1`.