4 Commits

Author	SHA1	Message	Date
Roy	6110c39dc8	[BugFix] Fix tokenizer out of vocab size (#3685)	2024-03-29 08:18:59 -07:00
SangBin Cho	01bfb22b41	[CI] Try introducing isort. (#3495)	2024-03-25 07:59:47 -07:00
Antoni Baum	bfdb1ba5c3	[Core] Improve detokenization performance for prefill (#3469) ... Co-authored-by: MeloYang <meloyang05@gmail.com>	2024-03-22 13:44:12 -07:00
Antoni Baum	fb96c1e98c	Asynchronous tokenization (#2879)	2024-03-15 23:37:01 +00:00