[Bugfix] catch AssertionError in MistralTokenizer as ValueError (#16344)

Signed-off-by: Guillaume Calmettes <gcalmettes@scaleway.com>
This commit is contained in:
Guillaume Calmettes 2025-04-09 19:33:24 +02:00 committed by GitHub
parent a25866ac8d
commit c3b5189137
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194

View File

@ -1193,8 +1193,15 @@ def apply_mistral_chat_template(
**kwargs,
)
try:
return tokenizer.apply_chat_template(
messages=messages,
tools=tools,
**kwargs,
)
# mistral-common uses assert statements to stop processing of input
# if input does not comply with the expected format.
# We convert those assertion errors to ValueErrors so they can be
# are properly caught in the preprocessing_input step
except AssertionError as e:
raise ValueError from e