Running the model locally gives a value error.

#21
by namantjeaswi - opened

I am running flan-t5-large locally and I am encountering the error

  ValueError: The following `model_kwargs` are not used by the model: ['return_full_text']

I am able to comment the code

self._validate_model_kwargs(model_kwargs.copy())

in the utils.py of the transformers library and run the model. Although it works this way but I believe this is not intended.

Hi @namantjeaswi
Thanks for the issue ! can you share a reproducible snippet of the error?

Hello, Thank you for your response

Here is the code and I have also included my system information.

from transformers import pipeline
from transformers import AutoTokenizer, AutoModelForCausalLM, AutoModelForSeq2SeqLM
from langchain.llms import HuggingFacePipeline



#tokenizer = AutoTokenizer.from_pretrained("google/flan-t5-large")
#model = AutoModelForSeq2SeqLM.from_pretrained("google/flan-t5-large") 


#save the model

save_directory = "flant-t5-large"
#tokenizer.save_pretrained(save_directory)    
#model.save_pretrained(save_directory)

#load the model
tokenizer = AutoTokenizer.from_pretrained(save_directory, local_files_only=True)
model = AutoModelForSeq2SeqLM.from_pretrained(save_directory, local_files_only=True)



pipe = pipeline(task="text2text-generation", 
                model=model,
                tokenizer=tokenizer,
                device='cuda'
                )   


local_llm = HuggingFacePipeline(pipeline=pipe)

res = local_llm("tell me about the paper attention is all you need")
print(res)


Error
File "C:\Users\Naman\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\LocalCache\local-packages\Python312\site-packages\transformers\generation\utils.py", line 1121, in _validate_model_kwargs
raise ValueError(
ValueError: The following model_kwargs are not used by the model: ['return_full_text'] (note: typos in the generate arguments will also show up in this list)

Fix I used: #commented line 1325 i.e. self._validate_model_kwargs(model_kwargs.copy()) in utils.py of transformers library to avoid error

System Information

  • transformers version: 4.39.3
  • Platform: Windows-11-10.0.22631-SP0
  • Python version: 3.12.2
  • Huggingface_hub version: 0.21.4
  • Safetensors version: 0.4.2
  • Accelerate version: 0.29.1
  • Accelerate config: not found
  • PyTorch version (GPU?): 2.2.2+cu118 (True)
  • Tensorflow version (GPU?): not installed (NA)
  • Flax version (CPU?/GPU?/TPU?): not installed (NA)
  • Jax version: not installed
  • JaxLib version: not installed
  • Using GPU in script?: Yes
  • Using distributed or parallel set-up in script?: No

Thanks @namantjeaswi !
Hmmm this indicates the issue in on langchain side, I don't think the fix should go on transformers as the pipeline is correctly initialized
Would you mind opening an issue there? You can also ping me so that i can follow the issue and the potential fix

Okay I will open an issue with langchain on github and share it with you.

Thanks

Sign up or log in to comment