20231088/vllm

History

Gregory Shtrasberg e97f802b2d

[FP8][Kernel] Dynamic kv cache scaling factors computation (#11906 )

Signed-off-by: Gregory Shtrasberg <Gregory.Shtrasberg@amd.com>
Co-authored-by: Micah Williamson <micah.williamson@amd.com>

2025-01-23 18:04:03 +00:00

cutlass_benchmarks

[Kernel]: Cutlass 2:4 Sparsity + FP8/Int8 Quant Support (#10995 )

2024-12-18 09:57:16 -05:00

disagg_benchmarks

[Misc] Update disaggregation benchmark scripts and test logs (#11456 )

2024-12-25 06:58:48 +00:00

fused_kernels

[torch.compile] Dynamic fp8 + rms_norm fusion (#10906 )

2024-12-13 03:19:23 +00:00

kernels

[FP8][Kernel] Dynamic kv cache scaling factors computation (#11906 )

2025-01-23 18:04:03 +00:00

overheads

[Core] Deprecating block manager v1 and make block manager v2 default (#8704 )

2024-10-17 11:38:15 -05:00

structured_schemas

[Benchmark] Benchmark structured output with datasets (#10557 )

2024-12-03 17:21:06 -07:00

backend_request_func.py

[Benchmark] More accurate TPOT calc in benchmark_serving.py (#12288 )

2025-01-22 13:46:14 +08:00

benchmark_guided.py

[Benchmark] Benchmark structured output with datasets (#10557 )

2024-12-03 17:21:06 -07:00

benchmark_latency.py

[Bugfix] fix beam search input errors and latency benchmark script (#11875 )

2025-01-09 17:36:39 +08:00

benchmark_long_document_qa_throughput.py

Doc fix in benchmark_long_document_qa_throughput.py (#11933 )

2025-01-10 23:51:43 +08:00

benchmark_prefix_caching.py

[Misc] Update benchmark_prefix_caching.py fixed example usage (#11920 )

2025-01-10 20:39:22 +00:00

benchmark_prioritization.py

[Misc] Make benchmarks use EngineArgs (#9529 )

2024-10-22 15:40:38 -07:00

benchmark_serving_guided.py

Benchmark serving structured output (#10880 )

2024-12-04 16:28:21 -05:00

benchmark_serving.py

[Benchmark] More accurate TPOT calc in benchmark_serving.py (#12288 )

2025-01-22 13:46:14 +08:00

benchmark_throughput.py

[misc] benchmark_throughput : Add LoRA (#11267 )

2024-12-19 15:43:16 +08:00

launch_tgi_server.sh

[CI/Build] Add shell script linting using shellcheck (#7925 )

2024-11-07 18:17:29 +00:00

README.md

[Feature] Update benchmark_throughput.py to support image input (#9851 )

2024-11-05 19:30:02 +00:00

sonnet.txt

feat(benchmarks): Add Prefix Caching Benchmark to Serving Benchmark (#3277 )

2024-03-27 13:39:26 -07:00

README.md

Benchmarking vLLM

Downloading the ShareGPT dataset

You can download the dataset by running:

wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json

Downloading the ShareGPT4V dataset

The json file refers to several image datasets (coco, llava, etc.). The benchmark scripts will ignore a datapoint if the referred image is missing.

wget https://huggingface.co/datasets/Lin-Chen/ShareGPT4V/resolve/main/sharegpt4v_instruct_gpt4-vision_cap100k.json
mkdir coco -p
wget http://images.cocodataset.org/zips/train2017.zip -O coco/train2017.zip
unzip coco/train2017.zip -d coco/