tiiuae

/

falcon-40b

like 2.42k

Running the example code in README generate nonsense like this:
the a- \n in. the. and,,..,\n to, a the a, the. and - to.\n, in to and\n,, the a\n\n,\n a the to and - the-., ,,

Tried several prompts, but all gives the same nonsense.
Any idea?
The example code:

from transformers import AutoTokenizer, AutoModelForCausalLM
import transformers
import torch

model = "tiiuae/falcon-40b"

tokenizer = AutoTokenizer.from_pretrained(model)
pipeline = transformers.pipeline("text-generation", model=model, tokenizer=tokenizer, torch_dtype=torch.bfloat16, trust_remote_code=True, device_map="auto",)
sequences = pipeline("Anything here ...", max_length=200, do_sample=True, top_k=10, num_return_sequences=1, eos_token_id=tokenizer.eos_token_id,)
for seq in sequences:
print(f"Result: {seq['generated_text']}")

Running on 4 GPUs (Quadro RTX 8000)
conda env:
python 3.11.3
transformers 4.29.2 py311h06a4308_0
pytorch 2.0.1 py3.11_cuda11.8_cudnn8.7.0_0 pytorch