20231088/vllm
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

[Doc] Update examples to remove SparseAutoModelForCausalLM (#12062)

Browse Source 
Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>
This commit is contained in:
Kyle Sayers 2025-01-15 01:36:01 -05:00 committed by GitHub
parent ad34c0df0f
commit 3f9b7ab9f5
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
2 changed files with 8 additions and 10 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

11
docs/source/features/quantization/fp8.md
View File

@ -54,16 +54,15 @@ The quantization process involves three main steps:
### 1. Loading the Model ### 1. Loading the Model
Use `SparseAutoModelForCausalLM`, which wraps `AutoModelForCausalLM`, for saving and loading quantized models: Load your model and tokenizer using the standard `transformers` AutoModel classes:
```python ```python
from llmcompressor.transformers import SparseAutoModelForCausalLM from transformers import AutoTokenizer, AutoModelForCausalLM
from transformers import AutoTokenizer
MODEL_ID = "meta-llama/Meta-Llama-3-8B-Instruct" MODEL_ID = "meta-llama/Meta-Llama-3-8B-Instruct"
model = AutoModelForCausalLM.from_pretrained(
model = SparseAutoModelForCausalLM.from_pretrained( MODEL_ID, device_map="auto", torch_dtype="auto",
MODEL_ID, device_map="auto", torch_dtype="auto") )
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID) tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
``` ```

7
docs/source/features/quantization/int8.md
View File

@ -30,14 +30,13 @@ The quantization process involves four main steps:
### 1. Loading the Model ### 1. Loading the Model
Use `SparseAutoModelForCausalLM`, which wraps `AutoModelForCausalLM`, for saving and loading quantized models: Load your model and tokenizer using the standard `transformers` AutoModel classes:
```python ```python
from llmcompressor.transformers import SparseAutoModelForCausalLM from transformers import AutoTokenizer, AutoModelForCausalLM
from transformers import AutoTokenizer
MODEL_ID = "meta-llama/Meta-Llama-3-8B-Instruct" MODEL_ID = "meta-llama/Meta-Llama-3-8B-Instruct"
model = SparseAutoModelForCausalLM.from_pretrained( model = AutoModelForCausalLM.from_pretrained(
MODEL_ID, device_map="auto", torch_dtype="auto", MODEL_ID, device_map="auto", torch_dtype="auto",
) )
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID) tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)