this post was submitted on 08 Nov 2023
1 points (100.0% liked)
LocalLLaMA
4 readers
4 users here now
Community to discuss about Llama, the family of large language models created by Meta AI.
founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Hi, i have the following error while trying to run it from transformers copying the code provided in huggingface
Traceback (most recent call last):
File "/home/XXX/project/translation/translateMADLAD.py", line 10, in
tokenizer = T5Tokenizer.from_pretrained('jbochi/madlad400-3b-mt')
File "/home/lXXX/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 1841, in from_pretrained
return cls._from_pretrained(
File "/home/lXXX/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2060, in _from_pretrained
raise ValueError(
ValueError: Non-consecutive added token '' found. Should have index 256100 but has index 256000 in saved vocabulary.